Andrew Nesbitt:     Welcome to The Manifest, a podcast all about package management. My name's Andrew Nesbitt.

Alex Pounds:        And I'm Alex Pounds.

Andrew Nesbitt:     And together we're talking to folk from around the world of package management. We're exploring the technical details, hearing the stories and the histories of their projects, and learning about the communities around them, too. Today we're joined by Todd Gamblin the creator of Spack, a package manager for supercomputers. Todd, welcome to The Manifest.

Andrew Nesbitt:     So Spack is a package manager for supercomputers in high-performance computing environments. So why is package management different in those environments, and just what is a supercomputer or high-performance computing environment?

Todd Gamblin:       I work at Lawrence Livermore National Lab, and we are primarily a science organization where people do research, and to drive most of that research we're doing simulations. So this could be for climate. It could be for other things like astrophysics. There are lots of things that you can simulate, but in general, it's very computationally intensive and you need a lot of hardware to do that.

Todd Gamblin:       If you wanna think about what high-performance computing is like, it's like a cloud except for it's not serving web applications or doing data-intensive batch jobs necessarily, it's doing a lot of number crunching and computing for physics problems, and so that equates to very large clusters. So our biggest machine at the moment is, I think, 1.5 million cores and people compile very complex codes to run on those machines. When you have a machine like this you really wanna get all the performance you can out of it, and we want to build the software to the hardware. People typically build from source in an HPC environment and there are lots and lots of build options, and they may differ by platform. We care about lots of different architectures, lots of different network fabrics and GPUs, things like that. I think that's the main reason that the software stack becomes so complicated.

Todd Gamblin:       A lot of the codes span multiple languages. Most of our production codes are written in C++ here at Livermore, but [inaudible 00:02:12] library's written in FORTRAN and they may have front-ends written in things like Python. We frequently want to use vendor packages in addition to what we might install with the package manager. So using stuff that's already installed is pretty important to us.

Todd Gamblin:       The reason we care about that is because if you don't use your vendor's network library or if you don't use some of the math libraries that they ship with the machine, you may not be getting the most out of the hardware. So we'll typically use the vendor BLAS implementation or LAPACK, those are two linear algebra libraries that go way back. And then we'll typically use the vendor's implementation of MPI, which is the message passing interface. The processes running on these millions of cores will talk to each other using MPI that's optimized for networks like InfiniBand or Omni-Path, which we run on our clusters. I think that makes for a much more complicated compilation environment then what you might see in say, Ruby, where everything is interpreted, or JavaScript, or something like that.

Alex Pounds:        And with these clusters, are they running an operating system which most people might be familiar with already, like Windows, Linux, or Mac OS? Or are they running something completely custom? Like when we talk about vendor packages are we talking about something from Apple, Microsoft, GNU, or some other provider?

Todd Gamblin:       You're typically talking about something from vendors you may have heard of like Cray or IBM, HP Enterprise, places like that. The OS is not unfamiliar to typical users. Most of these machines run Linux of some sort, but there are a lot of custom libraries and things that people build on them, so it's not the same OS as what you would see with like a standard Ubuntu image. I think the other thing that's very common at the HPC centers is the OSs tend to be older so we will run Red Hat Enterprise 6, and we've recently switched to 7. But the software stack that you get from the OS is maybe not as up to date as what you would get if you ran the latest bleeding edge Ubuntu image.

Todd Gamblin:       Part of that is just because that's what the vendor ships with the machine, and part of that is because the driver support is much more robust in the older kernels, and so we'll tend to stick to those.

Alex Pounds:        And how do these clusters compare to some of the options available in the wider world for high performance computers? Like you can rent spot instances off AWS reasonably cheaply, and you can get a lot of GPUs and a lot of CPU cores. Are they comparable now?

Todd Gamblin:       I think the nodes are by and large comparable, so if you were to grab a C4 Xlarge instance on Amazon it's probably pretty similar to what we would deploy on like our commodity clusters here. It's probably a Linux machine with Intel hardware, and, yeah, you can get GPUs on those things.

Todd Gamblin:       The place where the HPC centers really differentiate themselves at least for now with the cloud is on the network side. We're very sensitive to latency, so if you have physical space chopped up across processors they will communicate with their neighbors very frequently to say, "Hey, this part of the world has been updated, and I want to tell you, my neighbor, about these changes." That may happen many, many times per second, and getting the latency as low as possible can really help the performance of the simulations.

Todd Gamblin:       The other thing is that if you look at some of the bigger HPC centers like here at Livermore or at Argon or Oak Ridge, some of the other national laboratories, the biggest ones, you will see things that don't look a lot like a cloud node. Our new machine that we're putting on the floor right now, it's a Power 9 system, and so it has Power 9 nodes with three Invidia GPUs per node. It's about 4,000 nodes of that. It's a different architecture from what you could get in most clouds, although I think IBM does offer some power nodes. We're also deploying RM nodes, and then there are big Cray machines with Xeon Phi processors on the back end. They're like your standard Intel chips in the sense that they understand X86 instructions, but one Xeon Phi might have 64 cores and each one of those cores is like a little 386, and so you really need to use it in a different way than you would use, say, a C4 node with just Xeons. You need to have something that parallelizes well across all the available cores. They can use AVX 512 vector instructions and things like that, that you don't see quite as often in the cloud.

Andrew Nesbitt:     How did you get into the high-performance computing environment. What was your path into working in that space?

Todd Gamblin:       I think I've always been a bit of a systems person. I graduated from under grad and I was a web developer for a year. I decided I wanted to go back to grad school cause I thought I really wanted to work on compilers, operating systems, low level stuff. I ended up working for about a year on hardware. So I worked on asynchronous chips which are clockless chips. They use them cell phones sometimes. I decided that hardware was maybe too low level, so I sort of bounced back up a little bit and got into high-performance computing through performance tools.

Todd Gamblin:       In grad school I worked on how to measure load imbalance across clusters and that work led me to start working with Lawrence Livermore and once I finished it made sense to come out here and continue working with the people I was collaborating with. I've been out here since about 2008.

Andrew Nesbitt:     What was the process of managing all of these pieces of software and preparing things for the super computers, before you invented Spack?

Todd Gamblin:       Most of the HPC centers have teams of people who work on installing software for the big machines. That usually boils down to doing some sort of massive port every time a new machine is installed. In addition to that the users tend to just install their own stuff. They'll build everything from source in their home directory. That's the way that people get the most performance out of the machine, is to build on the machine and let the build system see what the hardware looks like and optimize for it. I think for the most part people were building by hand. There've been other attempts before Spack, so EasyBuild is actually a predecessor to Spack. That's from the University of Ghent in Belgium. Those guys were using it on the sysadmin side. But I'm not sure that it caught on quite as much with HPC developers as maybe Spack has.

Alex Pounds:        Do the vendors provide any tools to help with package management and managing the OS images?

Todd Gamblin:       They do provide some software with the machines, they'll provide optimized libraries and things like that. I think for a very long time HPC was sort of a universe where the vendors would install some software, the center managers would install software and then most people at most HPC centers would just use that software. Or they were developers and they were savvy enough to build everything themselves.

Todd Gamblin:       Now the software is getting much more complicated and people are starting to use more and more libraries and to build off of each others stuff and that's sort of driven this community over to the package management side. I mean I think that's a lot like other communities.

Todd Gamblin:       But in general the vendors provide an OS, with the OS package manager so our sysadmins will use our PM and then the facility staff would just install into a directory on a shared file system. It's not like a cloud where you would get your own node, and you would say "Hey, I want an instance of a C4 Xlarge," or something like that. You're building on some sort of login node that's shared among all the users, and maybe there's several of them and they get load balanced. The vendor compilers and things are installed there and they're shared file systems that tend to be mounted both on the login nodes and then on the back end compute nodes as well.

Todd Gamblin:       Typically what you do is you just install into the shared file system of somewhere in your home directory which is also shared between the front end and the compute nodes. Then would submit a job through the batch scheduler. Then it runs eventually and it gets in the software the way that you've installed it. Working on an HPC system is like managing your own ecosystem in a home directory.

Andrew Nesbitt:     You actually have global shared packages installed for all of the users of that particular HPC?

Todd Gamblin:       Yeah we do and the other thing that maybe pre-dates proper package management on HPC systems is environment modules. People had not automated the installation process on HPC systems until the last five years or so. With environment modules you do the install yourself, it's manual. But you can type module avail to see what things are installed on this system. You can say "module load tool" and it will put that in your path, it'll add to your LD library path to see path and some of these other environment variables so that when you do a build with say, Make or something like that, that it takes those libraries into account. It's as though they're installed on the system.

Todd Gamblin:       There's not a whole lot of consistency checking and things that you would expect from a package manager in the environment module system. You have to kind of declare conflicts among the modules and often it doesn't cover everything that you would expect from a package manager. It's not the full dependency hierarchy, you're assuming some things about the system there.

Alex Pounds:        I guess once difference that might apply in this ecosystem is the way your code runs, in that it's gonna be much more like a batch job. Normally when we're talking about developing and managing dependencies and pulling in all of the various bits and pieces that your app needs, that's software that we want to run continuously. Or be able to actually go back and rerun, right? Like if I'm working as a web developer then I wanna pull in all of my dependencies and be able to run my entire web app. If I'm working on IOS or Android I wanna be able to produce a build of my software and I wanna make sure my colleagues can produce it.

Alex Pounds:        I'm imagining with things like weather simulations and experiments and things like that, it's more of a case of, you're gonna write your code and then you're gonna run the code and then you get a result and then you're kind of done. Does that have any implications for how code is written and packages are managed?

Todd Gamblin:       I mean that's a good question. Maybe it's not so much that scientists don't need reproducibility. Cause I think they do. That's currently a big topic in the research community, is whether or not people could actually reproduce the work that other people have done, given the complexity of the software that they're using. There's a pretty big push to make that possible now. It probably stems more from the fact that the users are mostly scientists and they're not necessarily trained in DevOps or computer science things, they're people who write numerical codes that implement some physics and they implement it the way that they learned to in grad school. So for the most part, at least at the larger HPC centers, they'd have lots and lots of users doing kinda single person projects.

Todd Gamblin:       The users, the developer, they build their code. It's probably experimental and yeah they do just wanna run the experiement, get the results and maybe write a paper about it. But I think for bigger organizations like Livermore, I think we've just typically relied on the shared file system being there to reproduce runs. We could reproduce a particular simulation run, we might not be able to reproduce the build but people have thought that "Well you don't have to rebuild that often."

Todd Gamblin:       These days the hardware's getting more diverse and the number of machines that people have to get up and running on and the number of cross site collaborations that go on is increasing. I think also just the fact that the cloud is out there and that people don't just run simulations on super computers anyumore. They run them on their laptops, they run them in the cloud, they run them at other centers. It's kinda driven people to actually want to reproduce the full build and not just the artifact that works on this machine.

Todd Gamblin:       There is a lot of factors that contribute to this but I wouldn't say that it's just because web developers have a special need to reproduce their build.

Andrew Nesbitt:     Yeah I wasn't so much thinking about the expertise or though it's only a web developer thing. More about, generally programmers want their software to keep running. The app should keep running and users could have that app on a phone, around for say, five years. If you're a web app, you know maybe a bit less than that because you're always releasing and updating it. But I guess I think of the life span of one of these batch jobs as being a lot shorter and just kind of ... It doesn't have the same audience. It's more like the [inaudible 00:14:00] for the individuals.

Todd Gamblin:       I think that's true. A lot of the reason that people have started automating everything in the cloud is just because the services need to be running all the time. People expect high availability. I don't think that was traditionally such a high expectation for simulation codes. But the job ends, and the job doesn't run for that long. They might run for weeks at the most. But, the code lives for a long time. Package management and automating the build from source for the code and all the dependencies, can really help the process of getting up and running on new machines. One of our codes that we tried to build at Los Alamos, at their super computing center, it used to take the team two weeks to get that thing up and running on new platforms. When we actually used Spack for that, in kind of the early days maybe four years ago, it took them a day and a half to get all their dependencies ported and built, just because of the automation ... Someone wasn't hacking in a terminal to try all the different build variants for the new machine. I wouldn't say its perfect but it's definitely an improvement and I think it's helped a lot.

Andrew Nesbitt:     Just to go back into the reproducibility, when you mentioned hardware, the software that you're gonna be installing potentially actually performs slightly differently based on different HPC setups I guess. Does that have an impact on the reproducibility from the scientific side of things?

Todd Gamblin:       It does, the process of getting a code up and running on a new machine is not just building it but also running the test suite. There's correctness issues and you know build errors and compiler errors and things, but once you get passed that people have to run test problems and make sure that their errors are within tolerance and that may involve a lot more in depth debugging, depending on how the hardware performs.

Todd Gamblin:       The typical source of those kinds of errors ... Running on a GPU is different than running on a CPU. Anything parallel is gonna do floating point operations in different orders, depending on the machine or maybe even the run. Then some of the machines, like our new one, and our blue gene machines historically have done some computation in the network fabric. So actually the routers, for like a distributed sum, if I'm gonna sum up all the numbers on a large number of processors, there may be adders or floating point units that sit on the network somewhere. They may behave slightly differently from what the processor would do. Then vectorization can affect those kind of things. There's all sorts of reasons things may not behave the same on one machine versus another.

Andrew Nesbitt:     Do you see scientists putting the details of the hardware that they ran the simulation, when they publish their papers, to allow someone to be able to kind of dig their way back out of a hole if they find themselves in some strange situation, by knowing exactly the hardware and the software that was running in the previous result?

Todd Gamblin:       Yeah for HPC papers people do tend to put hardware information in there. But I guess I would say that most of the codes are built with pretty vigorous test suites and so I think the assumption is that you will fight with the hardware, get all the tests passing, and there are tolerances there. It's not a total disaster if your error is slightly larger, if it's within whatever they've decided is the accepted tolerance for their domain.

Andrew Nesbitt:     What was the tipping point where you went and decided that Spack needed to be built?

Todd Gamblin:       I think probably two things. I had a graduate student and I was working with her on a project for a while. She was doing work on load balancing. That involved a graph partitioner and some other packages that had a whole lot of dependencies and they had to keep building them repeatedly on our big blue gene machine. That was a fairly painful process, because she would update something or the OS would get updated and then the software wouldn't work anymore and I would have to redo this whole thing. Her job was to do a PHD thesis, so frequently I would rebuild the software stacks. That sort of made me angry, and I would complain about it. Then another one of my colleagues told me "You can't just make some package manager to automate all of this." That was sort of the last straw for me, was being told that I couldn't, so I did. That was sort of the tipping point here.

Alex Pounds:        What are some of the specific challenges for package management in these HPC environments?

Todd Gamblin:       The main one is that we care about performance and it's important to build something that's optimized for the particular hardware that you're running on. I think that's the biggest one from which everything else stems. It's important to use the MPI implementation that the vendor shipped, or at least a fast one that you built yourself. It may require some tweaking and experimentation and trying out different compilers before you can really get your code to perform well on the hardware. Vectorization is a notorious one. It's very hard to get vectorized code to stay vectorized with changes, the compilers can be somewhat finicky about whether or not they think that a particular loop is vectorization depending on how you change it.

Todd Gamblin:       Things like that require people to experiment with different compilers, different dependency libraries, different versions of things. I think just the general experimental nature of the stack requires people to play around more with their builds, then maybe they would in a production web environment where, by and large, the hardware's fast enough to run the web app. Whereas a small change in vectorization or threading or something on one of these machines can cause your simulation to take days instead of hours. People really care about that.

Todd Gamblin:       One of the main motivators for Spack was to build something more flexible. Where people could very easily tryout different versions and combinations of software and to build basically the same thing with three different MPI implementations, maybe four different compilers, things like that.

Alex Pounds:        Was it those reasons, like the performance requirements and other things, that made your colleagues say that it couldn't be done, or did he have something else in mind?

Todd Gamblin:       Yes, I would say that the complexity of building on HPC systems is pretty well known. At least within the community. People tend to stay away from this stuff. People are very scared of build systems and getting down into the details of how things link together. Once they get a piece of software working, unless some major performance issue comes up, they kind of wanna keep it the way that it is. Yeah, you know that's why he said you couldn't do this, because it's just very complicated. But I was Mac Ports user for a while and I had used Homebrew a bit and so the fact that they could do it for MacOS and granted they're assuming one particular OS and one type of hardware and that makes it easier. But the fact that they were able to do that motivated me to try to do it for our HPC machines.

Alex Pounds:        Let's say I'm a developer and I have access to one of these HPC systems. How do I use Spack to help me out? How am I gonna include it in my project, how am I gonna build it, what's Spack gonna do behind the scenes for me? Walk us through that process.

Todd Gamblin:       Okay, I mean the first thing you would do is just clone it from GitHub. Github.com/Spack/Spack. If you clone the repository it's pretty much ready to go out of the box. There's a shell script in there that you can source if you want completion and things like that. Or you can just run the Spack program out of the bin directory of the Repo. Then from there you can say, Spack install, package name and it will go and build that package and all of it's dependencies.

Todd Gamblin:       Currently, I think, Spack looks a lot more like Homebrew or a system package manager than maybe some of the other package managers that you guys have talked about on the show. It doesn't look like Bundler yet, though we're working on adding that. So I'm sorta hoping that eventually we would be able to have a Spack file .YAML where you say what your dependencies are and then you just say Spack install for your project and it goes and builds all your dependencies, allows you to create a virtual environment and spits out a lockfile that says here's how I configured all of these packages. I think as far as experimenting with builds, one of the main distinctions with Spack versus other package managers is that you can specify on the command line different options for your build. You can say Spack install package, you can say Spack install package at version, you can say Spack install package at version percent, compiler name, at compiler version. Then we have a pretty robust parameter system, so packages can expose build options.

Todd Gamblin:       In communities like Homebrew kind of discouraged that because it's a CI nightmare to test all the packages. In the Spack world the packages are more templated. We encourage people to put options on their package. You'll see some of the libraries have two dimensional, three dimensional options for different physics simulations. They'll have options to be built either shared or static. They'll have options for things like how big are the floating point numbers that this simulation uses. There's a lot of optional dependencies as well. Different solver libraries for systems of equations may use or not use other solver libraries or other pre-conditioners and things. That's typically configured at build time.

Todd Gamblin:       I think the other major thing that we allow you to do is we have this virtual dependency system, so that if you depend on something like MPI, which is an interface, but not an ABI compatible interface, you can switch out different implementations of that library to compile your code with. There's two major implementations of MPI. There's open MPI and mPitch. Your code doesn't depend on either one specifically, you depend on the MPI interface, which is versioned. Those packages provide that interface at different versions. So we'll match up the version of MPI that your package requires with your build.

Alex Pounds:        Is there a Spack registry? Do you have some database full of packages and compiler configurations that are commonly used or is Spack completely registryless?

Todd Gamblin:       It's registryless in the sense that there's no server that says "Hey, here's the list of Spack packages." It's not distributed in the that MPM is. Spack has this ability to have package repositories. It has a built in package repository that comes with it, and that just sits in the main Spack repository. There's like 2600 recipes in there. Sites can have their own package repositories and they can keep private packages there if they want to. But at the moment, yeah, updating your package requires a pull request. Into Spack or you make your own repo and develop those packages outside the main line.

Todd Gamblin:       It looks a lot like Homebrew in that sense. I think a repo is a lot like a tap and the package recipes look a lot like Homebrew formulae.

Andrew Nesbitt:     You don't have the code itself hosted, it's more like a script to define "Here's where you find the source code, and here's all of the different options that can be used to build it. Here's the tool that should be used to build that once you've knitted everything together."

Todd Gamblin:       Yeah and I think that's another interesting thing about Spack is that even in Homebrew the package files for different versions are mostly separate. Whereas with Spack there's one file per package but it's templated. It's a package.py file, and it implements a class that has the same name as the package. It says "Hey here's the URL, this is where you can find versions of this package." It may have several of those so that you can find lots of different versions. We'll even extrapolate URLs to try to find new versions of packages so you can say Spack check some package name and it'll go out and look for new versions and download tarballs and generate check sums for you so that you can put more of them in the package file.

Todd Gamblin:       Then you add variants to the packages. There were originally just true, false options. But those have grown so we can have multi valued, or string valued, or number valued, variants on the packages. All of that goes into the build providence when we build something. It's sort of like your instantiating that build but with a bunch of different parameters. Spack will sorta make sure that within one build, when we resolve dependencies, that that stuff is consistent based on how the packages depend on each other.

Todd Gamblin:       With the build options you don't have to just depend on another package at a particular version. You can depend on it with a particular option. You can say "Hey I depend on Boost but I need the IO stream library to be built." Or you can say "I depend on this solver library with these options enabled." Or you can say "I depend on something built with a particular compiler." Things like that.

Todd Gamblin:       When you Spack install "name", we call that a speck for the package, you're really giving an abstract description of the package and we go and sort of fill in all the blanks to make it into a concrete build for a particular platform.

Andrew Nesbitt:     Cool, so you could actually end up with one package installed and kinda configured in a very different way depending on which package declared that as a dependency and how it declared it. Which options it cared about.

Todd Gamblin:       Yeah, that's right. We let you have all the different versions built at once. So, suppose I'm gonna install HDF5, which is an IO library, for scientific data. It can do parallel IO, it depends on MPI. I might build it with three different MPI versions, I might build it with open MPI, MPICH, MVAPICH, those all live in separate directories and they are pathed to find their dependencies.

Todd Gamblin:       For people who aren't familiar with rpath versus LD library path, versus other ways of finding libraries. Basically if you're building a compiled library, it has to have a way to find its dependencies. One way is to use the ld.so conf in SE, that's the system library search path. The other way that a lot of people are probably familiar with is by setting LD library path, which I argue is bad. We can talk about that later.

Todd Gamblin:       The other way is by setting rpaths. Rpaths are embedded in the actual binary so if we build a library in Spack we actually put rpath's into the libraries that we build so that they know where to find their dependencies when they run. This is targeted at one of the major sources of frustration for people who try to experiment with lots of different versions of things. If they build a code with one version of MPI and then they run it but they happen to have another version of MPI in their LD library path, things can explode because they're not ABI compatible. The function calls in one binary are not compatible with the ones the library they have loaded expects.

Todd Gamblin:       With Spack we figure that you knew what you were doing, or the package manager knew what it was doing when it built the library. At run time you probably don't want to think about "Hey, what compiler did I build that with, what MPI did I use, what version of what dependency did I use." You just want to run. So we build the binary so that they'll know where all that stuff is and you don't have to necessarily set up environment variables to make it all work.

Andrew Nesbitt:     I guess that avoids Spack becoming not just the package manager but the build tool and the environment orchestration, as you see in a number of other package managers where things like cargo become the whole rust to chain stems from the configuration of the commands that run from that. Or with Bundler where you say, bundle exec, to actually run the program because it knows about everything at runtime to kind of define the environment. You're basically saying Spack is gonna always produce a binary that will do the right thing so then it can kind of get out the way and doesn't need to be involved once it's finished building everything.

Todd Gamblin:       Yeah that's the idea. You still need some environment settings for things like Python or languages that depend strongly on environment variables. We are adding this virtual environment capability but if you build just a regular compiled binary with Spack it should just work if you run it. We don't want to require any sort of environment management system. We do generate environment modules, like I was mentioning before because Spack's used not only by developers in their home directory to build things but also by facility staff to deploy large amounts of software. Modules is kind of the interface that people are used to. We'll generate module files for all the different builds so that users can just module load their favorite package.

Alex Pounds:        So when did you release Spack and what has the adoption been like?

Todd Gamblin:       I guess we released it as Open Source in, I'd say, 2014. Adaption was fairly slow at first. We wrote a paper about Spack and we presented it at the super computing conference in 2015 and I think that's when the adoption really started taking off. We started getting contributors, we started getting lots of pull requests for different builds of packages and we even started getting pull requests for core features. We've had major contributions from folks at EPFL in Switzerland and from Fermilab, they contributed binary packaging to Spack. It's been quite good.

Todd Gamblin:       We went from, I think, 170 packages in 2015, to over 2600 now. It's been pretty rapid. We've had contributors really start to step up and merge pull requests and work on managing the backlog of issues that we've had. That's enabled us to focus on core features as well.

Andrew Nesbitt:     Sorry I went down a rabbit hole of looking at the contributors on the project there. It seems like there's quite a good number of people are actively involved in the project.

Todd Gamblin:       Yeah we merged 30 to 50 pull requests a week, which is really good considering that it's a niche area. HPC doesn't have quite the community that JavaScript does. So I'm pretty impressed that we've been able to get as many contributors as we have. I think we're up to something like 260 now. People who have committed something or another to the Spack repository. Packages or otherwise.

Andrew Nesbitt:     Default packages are defined in the same repository as the actual application code for the package manager itself.

Todd Gamblin:       Yeah that's right. We could move that out to a separate repository, kind of like Homebrew has recently. The main reason it's in there right now is because we're not completely stable on the package API. So we wanna keep the core packages updated with the package manager, at least until we go 1.0. At which point I think we'd have to have some more stability than that.

Andrew Nesbitt:     Yeah at the moment I guess you have a fairly nice reproducibility and backwards compatibility that just comes from everything being in the Git repo, right. You can just roll back to a previous commit to get exactly what you had done before. It doesn't matter if the code has changed in master because the packages worked at the time that they were committed, I guess.

Todd Gamblin:       That's true. Currently our concretizor, which is what we call dependency resolution in Spack, it only looks at the package descriptions. So it's kind of like Yarn in that sense. It's, I guess what you might call a deterministic algorithm. You'll get the same thing for the same speck as long as you're using the same version of Spack.

Todd Gamblin:       We have had people complain about that and there's been debates for how we could modify the dependency resolver to more aggressively reuse what people have already installed on their systems. Which sort of gets into this MPM versus Yarn kind of debate about what should you use for production deployment, what should you use to get something installed quickly on your local machine and how reproduceable are those algorithms for doing dependency resolutions. We're gonna offer both, we'll continue to offer this version where if you went back to an older commit of Spack you could say, concretize without looking at anything that's installed, so that I know that I'm getting the same thing as another person. We'd also like to do reproducibility through the lockfile model. Where basically we say "Okay, we concretized this package, here's the concrete speck for it," you could go and load that on another version of Spack and rebuild the same thing as long as you're on the same machine.

Andrew Nesbitt:     Oh, I can imagine that lockfile having some really interesting issues around that, especially when the compiler is defined by the parent dependency of something?

Todd Gamblin:       That's a fairly infrequent case. I think People tend to build with relatively consistent compiler stacks. I think the trickier thing is like, suppose that the guy who tries to reproduce it doesn't have the same compiler as the person who tried to install it. That's become kind of an issue for newer versions of C++ and compiler capabilities like OpenMP where the support level for OpenMP depends on the version of the compiler and the support level for C++ also depends on the version of the compiler. That gets into what kind of runtime libraries do you have, how much you get them. I think we'll have to do some pretty interesting stuff there where, I'd like it to be where if you take lockfile that someone built on another machine. Even another architecture like Power and you try to build it on say, an ARM system. We could look at the package descriptions and try to re-resolve the dependencies in the build configuration to be as close as possible to the original and still legal and not violating any constraints declared by the packages.

Andrew Nesbitt:     Is there any kind of standard conventions for the way that tests are run on the kind of programs that would be installed or used, Spack?

Todd Gamblin:       Not really. A lot of HPC packages will come with their own fairly extensive test suites. They can get to be pretty gigantic. For example, PETSc, which is a solver library, builds multiple gigabytes of test cases outside the actual installation directory. There tests are actually bigger than their install. I think one of the problems in HPC is that submitting jobs to the compute nodes is fairly inconsistent from site to site. Writing a script that could automate that across the different sites can be difficult cause you have to detect what's the resource manager being used, how do I submit a job, how do I do it consistently so that I'm running N processes per node, or so that I'm running threaded versus with multiple processes. Things like that. There's lots of ways to run a code.

Todd Gamblin:       I think for the most part, the people who are developing stuff that runs at scale, they'll have some kind of test setup that works on their favorite machines, maybe at several sites. But as far as reproducing that I think that's another challenge. It's a building system on top of Spack that could reproduce all the different test cases. With predictable builds underneath.

Todd Gamblin:       I think there's both a build reproducibility problem and then a test running reproducibility problem that we have to face.

Alex Pounds:        Do the configuration files include any of that details? I know they'll include things like we built this with this package and these options but does it say "And then we ran it on this particular job system?"

Todd Gamblin:       No because Spack just builds and then running is a process. Most of these systems have some kind of batch scheduler that's sort of managing the hardware, so we don't really store the provenance for what the resource manager was on the system. You have to run with whatever the facility provides cause that's how you get permission to access the nodes. I'd like to figure that out. I think you would need something that's kind of portable for that. These things tend to be tied to the system. Like Slurm defines infinitive differently on the command line then say, PBS or LFS. Or some of the other resource management systems.

Todd Gamblin:       The projects tend to have runscripts in their directory for maybe some example test codes. There may be a bunch of them depending on which facilities they wanna show you how to run on. So they'll say "Here's one for Mira at Argon, here's one for Sequoia at Livermore, here's one for Titan at Oakridge, and here's one for a generic Linux cluster." Then you sort of have to tweak those to actually run the thing. It's not as automated as Kubernetes or some sort of AWS API at the moment.

Alex Pounds:        I guess the upside of that is that you can work on developing Spack without having access to an HPC environment. As long as you can build your software locally, like you could on pretty much any Linux machine or MacOS or probably windows as well if you have the right tool chains installed. Then the actual task running, that's kind of outside of Spack's purview, so you can work on the package management without needing access to that environment.

Todd Gamblin:       Yeah that's true. I would say that a lot of the development gets done on peoples laptops and then there's an effort to sort of port that to the big machines. Maybe when they need to do a big run or maybe frequently, but people do use their laptops for a tighter development loop. We would like Spack to work in all of those places so that people can use the same tooling.

Andrew Nesbitt:     And Docker doesn't get used in HPC environments, I guess, because you just pave over all of the uniqueness of the machines that you are installing it onto?

Todd Gamblin:       That's a really big topic in HPC right now. How to use containers. I would say that Docker doesn't get used necessarily on the HPC machines, but there are a lot of people who wanna run services and things in front of their HPC applications. There are science work flows where people wanna have some kind of little server sitting there, managing ensembles of jobs. Maybe users can log into that, play around with what jobs are running right now, how much of this ensemble is done, stuff like that. Then those would actually launch jobs through the patch system. For those environments people are looking at using Docker.

Todd Gamblin:       On the actual compute nodes, it's kind of the wild west out there at the moment. There's something called singularity that's getting a lot of traction. It's nice because it's multi-tenant safe. Most of the HPC machines are effectively shared. We use OS security. One user can't access another users information and you can't be root on the machines. Singularity is nice in the container only ever runs as your user. Whereas Docker, if you have access to the Docker command, you're essentially giving your users roots. There's singularity in other container runtime's that people are kind of exploring. The big issue there is, how do I get performance. If I use an Ubuntu image and the binaries for Ubuntu are built generic for generic X86, what am I leaving on the floor as far as actually utilizing the hardware?

Alex Pounds:        Do those security issues come up? Do we have the problem of one user trying to get access to another users data?

Todd Gamblin:       You certainly do. At the very least the facilities care about it. I suspect there are users out there who would very much like to have access to other users data cause on some of the HPC centers people are running export controlled stuff or otherwise sensitive software. We have a classified network here at Lawrence Livermore that people run on and need to know is a big deal there. Yeah, people are justifiably paranoid about that stuff.

Andrew Nesbitt:     Do you ever have anyone doing any crypto mining on these massive machines?

Todd Gamblin:       I think people have been fired for that. Not here but at other labs of course. You're not supposed to use government resources for that. For crypto stuff though, it's not gonna be worth it, at least for the center to mine bitcoins on the machine. The folks who are doing the best at that are working with custom A6 and really low power stuff, whereas if you paid for this whole super computer and tried to mine bitcoins, I don't think it would actually pay for the super computer. I guess if you're a user on the machine and you didn't have to pay for the super computer, then yeah you could get a lot out of it, which is why they don't want people doing that.

Alex Pounds:        Years ago I worked for Sun Microsystems as an intern and one of my colleagues was working in the department where they had access to all of the various hardware. Cause they would spin up customer environments and help them troubleshoot. One of the big machines, I think it was called something like a Sunfire E25K. It was a full height server rack just stuffed full of nodes. It wasn't quite at the same scale as your operating at, but it was a pretty tasty machine. I remember he told me once that he tried running SETI@home on it mostly to see what would happen. I think there might have been a little slim façade of "Well I just wanted to exercise all the CPUs, make sure it worked okay at maximum power draw." He said that the room got noticeably louder as all the fans spun up and the temperature got noticeably warmer and somebody pretty quickly came in and told him to knock that off and don't do that again.

Todd Gamblin:       Our biggest machine, when we'll run the benchmark on it, it's noticeable to the power company that we've done that. The machine has a peak power of nine megawatts and idol it's something much lower. Maybe three or four. So when you run this benchmark that uses 80% of the CPU the power company notices. So we have to tell them when we're gonna do that.

Andrew Nesbitt:     You're actually gonna dim the lights in the room.

Alex Pounds:        You're gonna dim the lights in the neighborhood at nine megawatts right? That's a chunky bit of power.

Alex Pounds:        Do you have any other fun war stories or anecdotes that you could share, from the HPC environment?

Todd Gamblin:       The biggest war story that I'm engaged with at the moment is trying to get tensor flow packaged in Spack cause a lot of people wanna do machine learning on these machines. One of the issues there is that Bazel is kinda designed specifically for what Google does, which is they check out the mono repo ... they control the whole project and I think packaging things that are built with Bazel is very hard. The build system assumes that the compiler is user bin GCC. If you went to use another compiler, we have to patch it.

Todd Gamblin:       The other nasty thing that they tend to do in there is they assume that they own all their dependencies. Rather than building the dependencies and then installing them and using the installed version, they'll build the dependencies in side their tree. They'll copy development headers and internal headers and object files into the install directory for tensor flow and then they will build the rest on top of that. It just sort of violates all the boundaries that you would expect within a package manager between packages.

Todd Gamblin:       We're trying to work around stuff like that and we're trying to work with the Google folks to make it so that Bazel can use external packages so that it might be packageable by not only just Spack but also the Debian's and CentOS's of the world but it's pretty nasty.

Todd Gamblin:       Another thing that comes to mind is building things with BLAS and LAPACK. In some of the lower level numerical libraries that have been around for really long time, I think that's one of the things that's really hung people up in the HPC environment for getting things installed because like MPI, and we talked about how MPI is a source interface, but the implementations are not necessarily binary compatible. LAPACK and BLAS are both source interfaces to some low level linear algebra routines. But from a build system perspective they look very different because every implementation of them ... there are no headers necessarily, there are FORTRAN symbols that you use to link these things and the different implementations that the vendors provide may have different library names in order to support the same interface. A lot of the solver libraries and a lot of the HPC software depends on BLAS and LAPACK. If you wanna be able to experiment with different implementations, the build system has to be able to abstract not only the name of the package that you depend on but also the symbols from the libraries that it includes and what they happen to be called in that particular implementation. That can get to be really nasty.

Todd Gamblin:       We actually put an interface into Spack for this so that you can call some standard methods on your virtual dependencies. If you're client code you can say "Hey BLAS implementation, whatever you are, give me the libraries that I need to link with you." It works kinda like package config, but I think the interesting this is that we made it so that one package may provide both BLAS and LAPACK or they may come from two entirely separate packages or maybe, depending on the configuration, one may provide both or one may provide one and another may provide the other.

Todd Gamblin:       So we've sort of abstracted that with this virtual dependency layer and made it so that the same code in a client package can link against those libraries without all the headaches that people have faced in the past. It doesn't look anything like the nicely factored Senver interfaces that people are shipping in the JavaScript and Go worlds these days.

Andrew Nesbitt:     There was a brilliant talk about that kind of topic at Fostem titled "How to Make Package Managers Cry," where there was a lot of jabs at Tensor flow and how painful it was to install by Kenneth Hoste from the EasyBuild project where I think he'd run into many of the same problems.

Todd Gamblin:       He's very angry about that. I was at the talk, I talk to Kenneth a lot cause we're both doing HPC build tools and yeah, I thought that was a great talk.

Alex Pounds:        Do you two collaborate a lot on directions? Do you have any talks about interoperability?

Todd Gamblin:       We've talked about interoperability a bit. Originally we talked a lot about how might we merge these things, how might they come together. The easy build guys, there packaging philosophy is a little different in that they fix versions. So there packages are really "I built this thing, with these versions of its dependencies, they don't do dependency resolution or version ranges," whereas Spack does. So that's sort of a fundamental architectural difference between the two. We both allow swapping of compilers and things like that but I would say that they're sort of more geared toward facility deployment. We're trying to support that, but also function more like some of the package managers that people love from other ecosystems like from Rust or Ruby or JavaScript. I'd say we talk a lot and we trade jabs at each other. But we don't necessarily share any code.

Andrew Nesbitt:     If there was one feature from another package manager that you could steal and add to Spack, what would that be?

Todd Gamblin:       I think the one that I like the most lately that I've seen is the dependency resolution stuff that the Pub folks are doing. It's Natalie Weizenbaum, right, with Pub Grub. They recently added this really cool tweak to the solver in their dependency resolver where they check all the reasons that conflicts have arisen and they give really good error messages to users. I would like to steal something like that. We're currently working on rebuilding the dependency resolver in Spack to handle a lot of things like cross compiles and different compilers, we'd have to add support for all of that and handle error messages with respect to more than just packages and versions. But yeah I would like to steal that.

Alex Pounds:        When we first started talking about this computing environment, we talked about some of the tools which were made available by the original vendors. Have you had any contact from people working for the vendors, have they suggested maybe even including it in the standard distribution.

Todd Gamblin:       So we have had talks with vendors about that kind of thing. We haven't gotten vendors to say "Hey we'll use Spack as the package manager on our hardware," although we're sort of pushing to try to get some of them to do that. But we have gotten vendors to contribute packages. In our latest procurement of this big Power9 system, Sierra, that we're deploying right now. IBM has done a lot of Spack packaging for the math libraries so that they'll work with the XL compilers that ship with the IBM systems. So we're starting to get vendor buy in.

Todd Gamblin:       ARM also has done a lot of packaging with Spack and a lot of the labs have done Spack packaging for ARM clusters as they try to get there software stacks up and running on ARM systems, which are looking pretty interesting for HPC.

Alex Pounds:        So you say some of them have contributed code. Have any of them contributed money? Is that even possible in this world? Cause, normally when we talk about supporting Open Source, it's from companies which use it. But in the academia/governmental world, maybe it's not even feasible for some big player like IBM or HP to say "We want to sponsor the development of this."

Todd Gamblin:       At least at Livermore and probably at most of the HPC sites, people are really the scarce resource. Even if someone were to pay us more money to work on Spack, we wouldn't necessarily be able to scratch up the developers here to work on it. I think the fact that they've started contributing time from their own engineers and that they've gotten their own people familiar with Spack and you know savvy with packaging software for these platforms, I think that's more valuable than maybe the money that they would contribute.

Todd Gamblin:       At the same time we're working on increasing funding for Spack both here at Livermore but also under the U.S. Exascale project which is a big source of funding in the department of energy. As far as increasing development effort or as far as contributing, I would much rather have time then money at this point.

Alex Pounds:        You said before that some of the new features in Space include things like the ability to package binaries. What else is coming down the pipe. What are the new features that you are working on or planning to work on, real soon now.

Todd Gamblin:       Okay so the main things that we're working on right now are infrastructure for doing builds of the binaries. So we'd really like to have Spack packages for some of the more common OS's, just built and ready to install. Along with that we're working on better supporting compiler run times for the binaries. If you build with a newer version of C++ then say your OS supports, then we need to rpath in the lib standard C++ from your compiler. So we're adding in the ability to do relocation's for things like that and to actually install the run time if you need it. Beyond just the build automation, we're looking at adding environments so that we can do this kind of manifest lockfile model that's getting to be so popular. We're working on rebuilding the dependency resolver, or the concretizor as we like to call it in Spack.

Todd Gamblin:       So those are the three immediate term things. We've had some really good contributions from the external community. So I think one of the more interesting ones was from Fermilab. They contributed the ability to take multiple instances of Spack and chain them together. So that if you had a bunch of packages installed by the system administrators, as frequently you do on the HPC machines, or by maybe an application team. You could point your Spack in your home directory at that and reuse the installed packages from there. So that's sort of bundled up with the new concretizor. Cause we need to make it so that it can more aggressively reuse already installed things. That's kinda the near term.

Todd Gamblin:       I think the longer term things that I would really to do would be to better model compiler dependencies. So I'd like to have packages able to say "I require C++ 17," and then to be able to swap compilers in and out based on what features of C++ they actually support. Then if I really wanted to go far out, I think I would want to build everything down through LIBC and make it so that you could actually swap out LIBC implementations from within Spack so that you could build the same software but with GLIBC or MUSCLE or EGLIBC or something like that. We'd have to rework the linker model a little bit and also add to the dependency model to do that but I don't think its infeasible.

Alex Pounds:        So I think the last thing that we have to talk about is the name of Spack itself, given that both Andrew and I are British, and the word Spack is an ableist slur in the UK? Has this come up, how has this impacted the project.

Todd Gamblin:       It's come up a few times, we have British collaborators. So we have folks at universities in the UK and also at AWE who are using Spack and you know they've mentioned it but they haven't actually asked us to rename the tool. I think some of them may call it by a different name internally. So I think at AWE they call it S pack or Soft pack. But aside from that we haven't had any really strong push back on the name.

Todd Gamblin:       I guess, on our side the name comes from, it's just short for, super computing package manager and it was Googleable. That was why we chose it. It's somewhat regrettable that it's actually a slur in the UK.

Alex Pounds:        And I'm presuming the project does have a code of conduct so hopefully that kind of balances things out the other way to help make it a more inclusive environment.

Todd Gamblin:       We do and I would say that we haven't had that many issues where people have just been rude or demeaning towards people on GitHub thankfully. I'm hoping that we don't, so we put the code of conduct in to sort of head that off. But, you know, fingers crossed.

Andrew Nesbitt:     If someone wanted to try out some of these HPC things, say, on their laptop. Is there ways that they could simulate or experiment with that without having access to a real HPC environment?

Todd Gamblin:       You can install most things from Spack on a Linux desktop or on a Mac laptop. We don't currently have Windows support but we'd like to do that at some point as well. You don't need a vendor MPI, Spack will build you an MPI if you don't have one. So all the different parallel codes you can just build on your laptop.

Todd Gamblin:       Most of the compilers that we would use on an HPC node, you can get some equivalent of that for your Mac. So, OpenMP is supported in GCC on a Mac. You can build that. You can even install Intel compilers on a Mac or on a Linux desktop and play around with them. I don't think there's anything that really prevents you from building or running the codes that you could build with Spack, you just won't be able to run them on as many processors.

Andrew Nesbitt:     Do people actually run Windows HPC setups?

Todd Gamblin:       They have in the past and I think for the most part that's peoples desktop environment. So they're doing there development with an HPC system in mind on a Windows desktop. In some cases, we have some customers maybe in the DOD and other places who run Windows on their desktops and they wanna be able to run simulations there without the HPC resources. But small scale stuff.

Andrew Nesbitt:     If people wanted to get involved with the development of Spack and Spack packages, where can they go to find that.

Todd Gamblin:       Just go to Github.com/Spack and we have a contributor guide, you'll find a link to our documentation on Read the Docs and there's a whole page on, how to contribute to Spack, how to contribute your first pull request. We try to be pretty receptive although we do have a bit of a backlog but we're open to PR's and we'd love to have more contributors.

Andrew Nesbitt:     If people want to find out more about you and what you do, where can they find out about that?

Todd Gamblin:       If you Google Todd Gamblin, I think that my profile at Livermore will come up, so that's one thing. The other thing you could do is, you could ... I had a Changelog podcast a while ago, just about Moore's law and HPC in general. That's a good one to look at if you're interested in this area in general and kinda the stuff that I do.

Andrew Nesbitt:     Great, we'll get links to those in the show notes as well.

Alex Pounds:        Well thank you so much for coming on and telling us all about Spack and package management and also teaching both Andrew and I about the HPC environment, I feel like you've had to go over a lot of the basic ground as we're both web developers and we've both been trying to map our prior experience onto this world and trying to figure out exactly how it works and what it looks like and what the differences are.

Todd Gamblin:       Yeah it's been great. Thanks.

Alex Pounds:        We'll be back with another episode soon.