Episode 1 - Homebrew with Mike McQuaid - https://manifest.fm/1
--------------------------------------------------------------

Alex Pounds:        Welcome to The Manifest, a podcast all about package management. I'm Alex Pounds.

Andrew Nesbitt:     I'm Andrew Nesbitt.

Alex Pounds:        Together, we're going to explore the technical details of package management, the stories of people behind projects, and the communities around them, too. Today, we're joined by Mike McQuaid, lead maintainer of Homebrew. Mike, welcome to The Manifest.

Mike McQuaid:       Thanks very much for having me.

Alex Pounds:        So, to get started, what is Homebrew?

Mike McQuaid:       So, Homebrew is a package manager for Macs primarily. We might talk more about other things later on. It's around for, I guess, coming up to eight or nine years now. It's the main way of ... for many people, many developers, at least, installing developer tools, like server domains, various other command line primarily applications on their Macs.

Alex Pounds:        And do you want to tell us a bit about your background Mike?

Mike McQuaid:       I always feel a bit boring talking about my background, considering it's a kind of fairly typical ... as a child I liked computers, then I did computer science at university. Since then, I've worked on computering. That's sort of my story, and went to Edinburgh University, graduated 10 years ago, have had a variety of jobs, and Homebrew sort of changed my career path a little bit, because before that I was primarily kind of writing C and C++, started writing Ruby for Homebrew, and kind of fell in love with it as a language, and have been kind of writing that professionally mostly since ... and nowadays I'm working at GitHub on their open source team.

Alex Pounds:        Oh, wow. So you actually ended up teaching yourself Ruby to kind of be able to contribute to Homebrew?

Mike McQuaid:       Yeah. Pretty much. It's funny, because in some ways this is why I always say to people with open source stuff. I was a maintainer of Homebrew of probably maybe a year before I would say I even started teaching myself Ruby, and before that it was more or less I would change stuff until it seemed ... well, I mean, I guess like all developers are familiar when you are familiar with one language, and you were learning another one. The very, very early days you're like, "Okay, I understand how programming works, but I don't really understand how this language works, but I can sort of try and fiddle with things, and try and figure out through mostly almost trial and error, and a mental model of how things probably work." So, yeah, that was the way I kind of first wrote Ruby was to do that for Homebrew, 'cause Homebrew is written in Ruby, and I hadn't really ever touched it before.

Alex Pounds:        And you mentioned that you fell in love with Ruby, what was it about Ruby that you fell in love with?

Mike McQuaid:       I mean, obviously stuff like the performance is horrible in Ruby compared to C and C++ that I was working with previously. I think it's just kind of the readability of the language, the way it feels like you can write very elegant solutions to a lot of problems in a way that's also very readable. Just I guess that sort of expressiveness, but then at the same level all the kind of method programming that you can do as well, which is when, obviously, you start to veer a little bit away from readability, but you can end up avoiding writing enormous quantities of boiler plate occurred through some of the kind of runtime kind of introspection that Ruby gives. Which, with my previous experience mostly with C and C++, even with Qt, which is the kind of toolkit I mainly worked with, which provides some sort of runtime introspection stuff. You can probably do a 10th of the stuff you were able to do with Ruby, and that always kind of excited me, and still excites me today.

Alex Pounds:        Would you say there was a lot of match programming inside of Homebrew?

Mike McQuaid:       You could argue that one of Homebrew's most successful features is the formula, which is effectively the description of how you can install a piece of software, and if you read the formula for something like wget, it's pretty obvious what commands Homebrew is running to build this software, where it's downloading things from, where it's putting things, stuff like that. I think that DSL is something that ... Ruby made that process much easier, because at the same time as being able to write the code in Ruby, and interpret this DSL with Ruby, you can do effectively everything you can do with Ruby within them. That gives you a lot more power than building some new language from scratch that you use just for your description files.

Alex Pounds:        Before we get too far into the details of things like the formulas, and exactly what gives Homebrew it's strength, maybe we should take a step back, and have you tell us a little bit about why would somebody use Homebrew? And also what made Homebrew be created in the first place, because MacPorts was a similar bit of software.

Mike McQuaid:       Yeah. Good question. So I've been trying for years to get my sort of pithy, what is Homebrew, and why should I care down. The kind of almost one I have for less technical people is Homebrew is effectively a package manager for your terminal. That relies on people knowing what the terminal is, but essentially anything that you might run in there, anything that you might interact with primarily in the terminal be that a command like that application, say, then you wanted a new version than the version Macs come with. Say it's MySQL for some kind of ware application you're running, or whatever, or a newer version of Ruby. Those are things that you would all install with Homebrew. You type "brew install" and then whatever the command is, most of the time it's in there, and it will install and work, and be on your machine in a fairly easy fashion. Without a package manager, of course, you can manually download the source card for such things, and build them yourselves, and manually download the source card for any dependencies, but this is pain and redundancy.

Mike McQuaid:       So most people do not want to do things that way. In terms of MacPorts, anecdotally, Homebrew appears to be used by more people nowadays than any other Mac command line package manager. Previously that crown was held by MacPorts. MacPorts was something that I ended up using a fair amount personally. I know Max Howell, the original creator of Homebrew, also did as well. Basically, I ended up about a year after he had created Homebrew, reaching the same sort of frustrations with MacPorts, and wanting to kind of work on something differently. So my initial dabbling with MacPorts interestingly, where I tried to effectively kind of build something a bit like Homebrew on top of MacPorts. So something, which annoyed with me with MacPorts, is that they duplicated stuff, which is already provided with the system. So on Macs, you get libxml, for example. Apple has their own version of that. MacPorts would build its own version, 'cause they didn't want to use APple's version.

Mike McQuaid:       This seemed redundant to me. My solution was I would just pack around with MacPorts so it thought libxml was installed when it actually wasn't, and that tricked it into using the system version. But this approach wasn't terribly sustainable long-term, so eventually then I stumbled upon Homebrew, read a little bit about it, and it appeared like it was doing things in the way that I thought things should be done.

Alex Pounds:        So what was the advantage of using the system provided libraries over creating their own beyond the redundancy?

Mike McQuaid:       Yeah. I mean, mainly it's just that's provided for you. Why build stuff and install stuff that you don't need to? Particularly back then, when both MacPorts and Homebrew were primarily from source package managers, you weren't building things once on some server, and distributing it to everyone. Everyone was building that individually on their machines. Something like Git, for example, had no dependencies on Homebrew, but at the time four or five in MacPorts, and that takes significantly longer to install the same software with MacPorts than it did to Homebrew. MacPorts was just installing more stuff that it didn't need to install technically. They would argue that they keep better update with bug fix releases, security releases, etc. We argue, if you're using a Mac anyway, if you don't trust Apple's security and bug triage in process, then you're gonna have bigger problems at that point.

Alex Pounds:        I remember a lot of people switching from MacPorts to Homebrew driven a lot by ImageMagick at least in the Ruby and Rail's world, just waiting hours for MacPorts to install our magic and ImageMagick was incredibly painful. When Homebrew came along and kind of tried to reduce the amount of different things that get built every time you needed to update it was like a breath of fresh air.

Mike McQuaid:       Yeah. I mean, I think that's what stuff like this tends to come down to for end users, is it's just this has saved me time, and therefore word spreads that, "Okay, well, this operation that you have to do once a day, once a week, once a month, whatever, is three or four times faster with this software," and then people move.

Alex Pounds:        Do you think it is that speed, which was the main driver of people adopting Homebrew?

Mike McQuaid:       I think it was a combination of two things. I think partly the speed and in general the feature set, and various sort of things being perhaps slightly easier, or improved. Again, I'm not up to date on MacPorts to know how many of these things still apply today. I think the biggest thing Homebrew had going for it both then and now was, it was a fairly early GitHub project, and Max very smartly built it such that GitHub kind of contribution workflow was how you made all updates to Homebrew effectively as a contributor. So right from the outset, Max kind of decided that he was not trying single handedly keep every single package up to date. And instead, kind of rely on a community of people to kind of keep things up to date, and make bug fixes and stuff like that.

Mike McQuaid:       I mean, obviously, that happens with bigger, more established package managers, but the way they tend to handle things is, say, in Debian and you have a piece of software, where that has a maintainer. That maintainer is the one who's responsible for keeping that software up to date, for merging bug fixes for kind of deciding whether or not, and changes should be made to that software or not, if some other people submit them. But in general, that's kind of that process responsibility, whereas in Homebrew, there was the kind of fairly radical shift taken there, where it's no one's responsibility, and therefore it's everyone's responsibility.

Mike McQuaid:       So if you're using a piece of software and it doesn't ever get updated in Homebrew, well, you can submit a full request to update it, and instead of almost trying to kind of find people, who are willing to go through the arduous process of doing that. We've tried and still continue to try to make it easier and easier and easier for people to kind of submit updates, submit fixes, create new packages, and basically just improve that contribution workflow. I think with the right of Homebrew and the right of GitHub being sort of somewhat side by side, I think it's kind of helped that ... if you know how to submit a poll request to GitHub, you know how to update a package on Homebrew.

Alex Pounds:        I think Homebrew is probably the first of the package managers to use GitHub like that as a database essentially. So if you try and take off your GitHub employee hat for a moment, do you think that potentially creates a point of weakness for Homebrew in the long-term?

Mike McQuaid:       I don't think so, because I think ... I guess the thing that's more interesting than the GitHub side of things is the Git side of things. So if there was some sort of future, where everyone had to move from GitHub to whatever, then I would imagine a Homebrew's contribution workflow would still look very similar. The way most of these platforms tend to kind of handle user contributions now, it's quite similar. I think the beauty of Git being a decentralized virtual control system is that it would actually relatively easy for us to kind of move Homebrew to something else that kind of allowed this sort of contribution workflow that would be possible for us to do that. But I guess almost the Homebrew ethos of why build your own libxml if your Mac provides it. It's kind of similar with this stuff as well. It's almost like why build your own network protocol for doing updates, when you can sort of abuse Git to do the same thing, particularly when your contributors are going to have to use Git to contribute to the project anyway.

Alex Pounds:        So the way that people can add any package and add any package doesn't necessarily need any kind of involvement from the maintain of the actual source code that, that package is installing, do you get many people kind of resistant to the fact that anyone can add an update, someone else's packages?

Mike McQuaid:       Some people certainly struggle with the mental model. You do get people, who, I guess, submit packages the first time, and they're sometimes ... you know, submitting a package that they maybe help maintain what we call the upstream in package manager world, i.e., they write the original software, and then they're being like, "Well, I'll also help write Homebrew's formula for that to install that." There's the assumption that they kind of get a certain amount of veto power, and they get to kind of dictate, "Well, actually no, I don't think that be the case." Or, "I think they should have this feature," or whatever. Quite often they're quite surprised to learn that isn't the case, particularly when if someone else submits your software, and then you turn around in a few years later and say, "I don't think this software should be in Homebrew. Please remove it."

Mike McQuaid:       Well, then we tend to look then at our analytics, and if we've got 1000 people installing that software a day, well, we're not gonna pull the rug from under them. So we try and resolve the issues they have, but we're certainly not willing to just pull things, because other people ask to do so. I think that's fairly commonplace in package managers to be honest. But I think that sometimes presents some confusion, I guess.

Alex Pounds:        One thing I've been curious about with people getting essentially annoyed that you have packaged their software is that there is some software out there, which is either ... it's available commercially, but is also open source, and what you're paying for essentially is a prebuilt binary. I know Crossover Office used to have that on Linux. I know there are other things, which are a little bit donation ware. There's a thing called Cyberduck, which you can compile it yourself for free, or you can get a binary for pay. So I always presumed I guess, that software like that might get a bit annoyed if you made it so ... you type one command and it installs.

Mike McQuaid:       I mean, we've not really run into stuff like that. We do generally try to take those types of things on board, but again, arguably, it's not our responsibility to enforce your preferences. With open source licensing, we have lots of ways of in a legal form that you can say, "Well, actually, no, you can't build my software from source or whatever." I think that's the thing that's tricky with this stuff, is that you get more and more people, who almost want all of the benefits of open source, but then to avoid some of the downsides like, "Well, actually, I want to have this project be open source, but then I want to be able to run a business on top of it without other people almost ... to use this software without my help." While I feel for those people from the sustainability perspective, at the same time I don't think it's Homebrew's responsibility in those cases to say to our users, who want to do that.

Mike McQuaid:       We're not gonna allow you to do that. And particularly, because the way we've set up Homebrew nowadays, we have what we call Taps, which are effectively third-party Homebrew repositories. So anyone can just make a Homebrew repository and put stuff in there if we refuse to package it, or we remove it, or they don't like the way we package it. Ultimately, even if we refuse to package some of this stuff, then other people would probably do so anyway.

Alex Pounds:        So apart from things like that commercial sensitivity, what kind of things would cause the Homebrew project to refuse to package something?

Mike McQuaid:       We tend to be a little bit more conservative about some software we package in terms of ... we basically hate patches, right? So a fairly common package manager thing is the upstream software does something for whatever a particular reason, and then the downstream package manager says, "Well, I don't like the way that's done, so I'm gonna write a small patch to modify that software so it works better with our package manager." Now, this in general, I think is not too much of a problem, and it's not that big a deal. In a small number of cases, it becomes a massive problem, and a massive deal. So the example ... and I hate to bring it up, 'cause again, it's a mistake I could have easily made myself, but in The Debian Project they had an issue with this, where OpenSSL was patched in such a way to basically avoid a valgrind warning about reading from uninitialized memory.

Mike McQuaid:       But that reading from uninitialized memory turns out to be a source of entropy. So this is something, where 99.9% of the time, any software doing this is doing the wrong thing. In OpenSSL's case, it was doing the right thing. As the result of patching up the source of entropy and effectively they reduced the cryptograph effectiveness of OpenSSL on any Debian or Debian derivative system. So that includes Ubuntu. So this was a while ago now, but that was something I remember reading at the time, and thinking, "Well, firstly, it sucks to be that open source maintainer. I would really feel bad if I was them right now. But then on the flip side, this is why patches are generally not a great idea." Because what you're doing is you're saying intentionally or not, "I know better than the Upstream software writer how their software should behave. So I'm going to write a patch, which modifies their software that they've spent hundreds, thousands, whatever of hours writing to make it behave in a slightly different way.

Mike McQuaid:       In general, I think they are dangerous. So, that's an example of something Homebrew might not patch. It's something, which needs to require extensive patching to work inside of Homebrew, on Macs, things like that. We're not really interested in having those types of projects that can't be made really to work with Homebrew without substantial modifications, because ultimately, those modifications should be going back to the original software. If the original software is saying, "Nope. I have no interest in supporting Macs." Or whatever, then there's no point almost fighting a losing battle there.

Alex Pounds:        So I notice there's a Linux fork of Homebrew. Have you had any involvement with that? Or has that been a hands-off kind of thing?

Mike McQuaid:       It's been mostly a hands-off kind of thing. I have tried to make things a little bit easier for them on occasion, and I've done some bits and pieces on occasion to try and do that porting, but I've not been involved much with ... there's almost two sides to it. One side interests me, the other side doesn't. So it's called Linuxbrew, and it's basically a way for people to be able to use Homebrew on Linux. That feels like a weird thing, and most people are like, "Why would you want to use Homebrew on a system," where you have a package manager say, "If you're on Debian apt-get, which in my humble opinion is a vastly superior package manager to Homebrew, why would you want to use Homebrew on Linux?"

Mike McQuaid:       The argument for that is one of the neat side effects of Homebrew is that you can actually run Homebrew as a non-route improper user, and you can install Homebrew in any location. So this is useful for people, who have access to Linux machines, but they do not have access to the package manager. So I believe scientific computing labs, this is relatively common, where you are a scientist, and you have access to these Linux clusters, but you don't have access to install software on them without Git. So if you want to have a newer version of them on that machine, or whatever, then that's when using Linuxbrew could be kind of useful to you.

Mike McQuaid:       As for the stuff that interests me versus the stuff that doesn't, so the actual day-to-day running of that package manager, and trying to make the Homebrew formula work on Linux, and stuff like that. I find not interesting, 'cause I don't use Linux. It's a great server offering system, but I don't use it on any of my desktops, for example. Whereas the actual making the Homebrew package manager code itself, like cross platform, that's the bit I find a bit more interesting. So I kind of worked in the past to try and add a generic kind of OS compatibility layer, and abstract away some of the kind of Mac specific stuff, and keep our test suite running on Linux, and stuff like that.

Mike McQuaid:       That ends up being kind of useful for us as well, because Mac CI like Travis CI is great, it's really good that they provide free Mac CI, but it's significantly slower than the Linux CI. So having those Linux CI builds is useful for getting rapid feedback on our test suite, and stuff like that.

Alex Pounds:        Is there formulas that are shared between the two package managers, or it's just the core code?

Mike McQuaid:       This is when ... I guess to a slightly ideological point of intention in that all of the formulas are shared, but if was dictator of the universe, I would forbid such heresy, because I think that the kind of abstraction layer stuff I added to Homebrew like Homebrew/brew, which is our kind of repository that's theatrical package manager. It's basically was a layer to use kind of OO instead of having if Linux, if Mac this, if Linux this, if Mac this, all over the place. The problem is in formula, that problem hasn't really been solved, so if you want to have a formula work on Linux and Mac you just end up having if Linux, if Mac, if Linux, if Mac all over the shop.

Mike McQuaid:       My humble opinion would be that Linux Group should be maintaining their own separate formula, rather than kind of forking from Homebrew's, but I believe currently they do fork from Homebrew, which means that some of those formula work on Linux and some of them don't.

Alex Pounds:        If you were dictator of the universe, would you have any other kind of hallowed principles?

Mike McQuaid:       No. My principles would be purely about package management and nothing else. Freedom for all men, women, and animals, except for package management, in which there would be a dictatorial regime, I guess.

Alex Pounds:        Right. So what would the package management, dictatorial regime, what would be the principles?

Mike McQuaid:       So I guess, yeah, like the avoid patching upstream software when possible. A big dream of mine would be seeing all package managers sharing some sort of core dependency resolution logic and stuff like that. Some sort of underlying library to basically abstract the way the stuff, which every package manager re-implements form scratch. In Homebrew's case, for example, stuff like dependency resolution is ... the code there is hideous. I hate working on it. It makes me really sad, and it makes me even sadder knowing that apt-get's code lies just a hair's breadth away, and does all this stuff perfectly, and has for 25 years.

Alex Pounds:        Yeah. You mentioned before actually that you have a lot of admiration for apt-get, and you consider it to be a better system than Homebrew overall, what is it about apt-get that you feel is so fantastic?

Mike McQuaid:       So I used to be a desktop Linux guy back in the day. I kind of started using it stereotypically when I was studying CS at uni, and then kind of for a while afterwards, until the demands of my time became too great. That's not to say that Linux on the desktop is not good, but if you're kind of a natural tinkerer, fiddler type like me, then the amount of knobs you can twiddle, which just means that my system was continuously broken. But I think apt-get is just ... it's just a really robust package manager, it's used by many different Linux distributors at this point. The UI is the only part of it I have less than perfect experiences with, like it's really fast, the dependency resolution is really smart, it stores a lot of metadata. It can do an awful lot of things, and it just carves everything up in beautiful ways.

Mike McQuaid:       I'm yet to really see anything in Homebrew that I don't feel could almost be like apt-get behind the scenes. I kind of joke with some of the Homebrew maintainers that if I had infinite time, what I would do is more or less get Homebrew's command-line interface and the kind of formula kind of package DSL, and then just make that a wrapper on top of apt-get, and use apt-get for everything else.

Alex Pounds:        Would that be feasible? Would it be easy, or I guess, not easy, would it be feasible to graph apt-get on top of a Mac OS system?

Mike McQuaid:       So there was the package manager Fink ... I think it's still on their active development, but I don't know if it's used extensively nowadays. So that is using apt-get on Macs to do package management. So it's possible apt-get ... I think you can actually install apt-get within Homebrew bizarrely in a sort of exceptionally weird way. Again, it comes back to the almost two sides of package management, where you have the actual implementation of the package manager itself, then you have the kind of getting all the packages, keeping them up to date, dependency resolution, where do you download them from? Where do you upload them to? What does your CI look like? And all that type of stuff. That's the side I just feel apt-get handles really nicely. If I had six months to year, I feel like I could probably re-implement Homebrew in that way. I'm kind of almost do myself out of the job in a lot of ways.

Mike McQuaid:       But sadly, I do not have that time just yet.

Alex Pounds:        It seems to be a case that Homebrew, especially, actually keeps that local Git repository around, and uses it for thinking as opposed to it being just a standalone piece of software you're essentially just plopping the whole kind of development environment into a hidden folder somewhere, and exposing that via adding it to the path, right?

Mike McQuaid:       Yeah. So if you install Homebrew in the default location, /usr/local/Homebrew, the contents of that folder is exactly what you would get if you do git clone github.com/Homebrew/brew and then there's within the tap folder, which is again, a sub folder of that Homebrew folder you get. Like the homebrew-core repository, which is all our kind of package description files.

Alex Pounds:        So you work at GitHub. Is GitHub paying you to work on Homebrew full-time? Is that your full-time job?

Mike McQuaid:       That is not my full-time job. My full-time job at GitHub is working on the kind of broad mission of making GitHub better for open source software, trying to make things better for maintainers in various ways. I am given some time by GitHub in my current role to work on Homebrew. So that's kind of like a sort of balance that I have set, but no, it's not something they pay me to work on full-time.

Alex Pounds:        But I guess you get a pretty good perspective on things being both a maintainer of a popular open source project, and packaging many other popular open source projects to know what kind of things the various maintainers would find useful.

Mike McQuaid:       Yeah. I think so. I think that's a nice thing is that as more and more open source kind of moved on to GitHub, and Homebrew being so tied into GitHub in so many different ways. Yeah. You do get a real sense of kind of how all these different projects are working, and what their workflow looks like, and what their contribution patterns look like, and how their maintainers interact with upstream projects and downstream projects, and use libraries, and all these types of things. So, yeah, I feel like it does help you to kind of learn how to make GitHub better for open source, but also how to make Homebrew better. On a day-to-day basis, you're seeing how this ecosystem is developing by itself.

Alex Pounds:        So you are Homebrew's lead maintainer, but you have a lot of help from other people right?

Mike McQuaid:       Oh, yeah. Yeah. Loads of help. I mean, I could definitely couldn't run Homebrew by myself. So one of the main people recently ... So we announced we have kind of a lead maintainer now effectively for all our formula repository as well, homebrew-core. So that person is doing in terms of hours in the day way more than I am. Their username on GitHub is ilovezfs, and they work unsurprisingly on the ZFS on OSX project as well. I kind of joke often that I'm not entirely sure that they're not a bot, that's just got very good at natural language processing, because the sheer amount they contribute to Homebrew is kind of staggering.

Alex Pounds:        So in terms of your day-to-day on Homebrew, what kind of takes up your time? How do you spend your time?

Mike McQuaid:       So I'm mainly focusing now on the package manager itself. So I keep an eye on what's happening with the kind of formula updates, and stuff like that. I mostly kind of skim through that stuff, whereas what I'm trying to do kind of day-to-day is triaging new issues in PR as they come in on Homebrew the package manager itself, and then working through a backlog I have sort of maintained in my email client, and such things that are currently broken, things that need improves, things that I find annoying, or I've been at conferences and heard multiple people kind of complain about Homebrew working a certain way. So I try and just bow into that sort of triage bug fixing, review, and feature development. It tends to go in that order as well, where feature development is almost like the treat I get when my Homebrew folder, and my email client is empty, and all the PRs have been reviewed, and stuff like that.

Alex Pounds:        So in that day-to-day what could be a nice surprise, and what would be a horrible surprise?

Mike McQuaid:       I mean, to me, still any PR that comes from someone, and is like high quality off the bat is always a nice surprise. So when I wake up and I open my inbox, and either when it's fixing a bug, or to be honest when it's a new feature that is something that we want, when someone that has just done the research, read the docs, submit the PR, the PR's green, they followed all the starred guidelines like they ... all our tests pass and stuff like that. And then if there's any tiny little niche that need addressed, they are quick to respond to feedback, and they kind of ... it sounds a little self-entitled, but they defer to the authority of the maintainers. By that, it's not that I think the maintainers know more than they do about program, or anything like that, it's just that we probably do know more about Homebrew than they do.

Mike McQuaid:       So it's arguments about why Homebrew is a certain way, or a little bit of a waste of time. So I guess that flips onto the other side, where the stuff that is not pleasant to deal with is just people demanding endless rehashes of decisions that have been made, and well-documented in the past. People resurrecting old issues, saying, "I have this issue," when they don't have the issue. People submitting issues and stuff like that, but they don't read the issue template, and they don't provide the information we need to help them, and then we ask them for the information, and they never reply. These are the types of things that kind of tend to grind you a little bit, and I guess, at the most extreme end, when you have released something in Homebrew, which is particularly contentious, and you start getting hate mail, and people calling you a Nazi, and all these type of things.

Alex Pounds:        I guess Homebrew is kind of in a position, where you might get slightly more people, who are I guess, not developers, less technically inclined, because it's not like a programming package manager, it is for that user software. So as well as the developers, who use the day-to-day, you get a whole bunch of pieces.

Mike McQuaid:       Yeah. I thinks so. I think that's definitely something that's changed over time is that Homebrew used to be very, very developer centric, whereas now, I still feel like the majority of our users are probably people who are either currently working as, or aspiring to work as developers at some point. But we do have more and more users, who have ... they go to some site, and they want to install some piece of software, and then the software says, "Well, step one, install Homebrew. Step two, run "brew install blah". It's like this is literally the first time they've ever run the terminal on their Mac, and there's confusion around stuff like that.

Mike McQuaid:       A really basic example that keeps coming up again, and again, and again, which is annoyingly not something we could even particularly work around. You know how any password prompt on your Mac, when you type in characters, it makes those round blobs for each character you type in, but if you do a pseudo prompt in your terminal, and it asks you for your password when you type in characters, nothing. So it looks like it's just not reading from your keyboard at all if you've not used the terminal before. So this is a really common thing that our installer requires people to enter their password, and they say, "Oh, it won't accept my password," and what they mean by that is, "I type my password and nothing happens." Because they aren't familiar with terminal text entry and stuff like that.

Mike McQuaid:       While we could add a little thing, saying, "We're about to prompt you for things, this will not show you your password." At that point, it feels a little bit of a slippery slope, where it's how much hand holding do we provide for people, who if this is literally the first time they've ever used the terminal, then the number of guardrails we have to work on there, it starts to outweigh the usefulness of them.

Alex Pounds:        You alluded to the whole analytics debacle that you ran into, is that last year?

Mike McQuaid:       Yeah. Yeah. I think last year.

Alex Pounds:        Did you wanna touch on that slightly?

Mike McQuaid:       Yeah. So there's this repeated pattern in Homebrew of laziness that I think I continued, and propagated, which is, if someone else has done something for you then why reduplicate their work if it's not necessary?

Alex Pounds:        So laziness is a virtue in this case, right?

Mike McQuaid:       Yes. Well, hopefully. People may agree or disagree on that. I'll leave that to the audience to decide. So one of the things we do is if you download wget, for example, and if you were building that from source, then that will download it from the main wget website. So as a consequence of that, we don't have any way of knowing how many people have downloaded wget. Now, somebody would say, "Why do you even care about that? Mind your own business." Whatever. It's like, "Well, that's an argument, and that's valid, but the problem is, is when I'm staring at 100 open issues, or I need to kind of refactor some stuff, and I have 1000 packages to choose between, which do I prioritize highest? Which do I jump on and fix first?"

Mike McQuaid:       So for a while we kind of thought about what's the best way of doing that, and the more I've kind of worked in my professional career, these problems just don't really exist in the kind of professional software space, because almost every piece of software has a reasonable analytics nowadays, where you kind of get some insights into what your users are doing. Not because, as some people might say, "You want to spy on your users, or sell their information," or things like that, but literally because so you can focus on making the stuff better that people use, and either remove or just ignore the stuff that people aren't using.

Mike McQuaid:       So for us this culminated in implementing a way of doing this, and using Google analytics. So when you install something on Homebrew, by default it will send an anonymous notification to the analytics basically saying, "I have requested an install of whatever, wget." And then that has allowed us to prioritize and figure out what are people using, what software is working and broken for most people? What software is really, really popular? What software is never installed by anyone ever and can be removed if it breaks? So the problem was, when we released this, there was a fair amount of backlash from people. The backlash appeared to revolve around a few different things.

Mike McQuaid:       So one of them was that the analytics were opt out, rather than opt in. My viewpoint on that was that from a pseudo ... Well, it's semi-scientific perspective. If you're going to opt-in, then you're not really looking at the analytics of people, who use Homebrew. You're looking at the analytics of the type of people, who use Homebrew, and would opt-in to analytics, which is A, very small subset, and B, that small subset may well be biased towards certain ways of using Homebrew. So I thought an opt-out was a better way of gathering better data from people. And the second I think main issue was that this was going to Google analytics, rather than some other platform. That's something that I take on board. I mean, I try to not put all my eggs in one basket as far as tech companies go. So as a result, I did seriously spend a decent amount of time investigating other alternatives, but the problem is, no one provides anywhere near the same sort of three tier the Google analytics does.

Mike McQuaid:       No one provides the same sort of level of performance, and tooling, and stuff like that. It's just not doable for us to just set up our own server to kind of manage the amount of traffic we get through things like this. I mean, if you include bots, Homebrew's getting something around I think a million monthly active users. So at that point, it's pretty hard to just say, "Oh, we're going to spend up new infrastructure to manage all this for the analytics." Who pays for that? How does that get done? And I guess the third category is people who just don't like analytics full stop, and think all the big software makers are spying on them, and now Homebrew's spying on them too. We're gonna sell all their information for money. I mean, they're entitled to think that, but I personally never made a penny's worth of money directly from Homebrew.

Mike McQuaid:       So while that might be a nice idea, it's certainly what I've ever managed to come up with a good way of doing. As a result of this, it kind of blew up on multiple occasions. As a result, you get a lot of issues, and comments, and personal emails saying, that I'm "working with the CIA", or effectively a Nazi, or Hitler, or whatever, and stuff like that. And it's kind of interesting with that stuff, because I can actually laugh about most of that stuff, because more or less anyone who would say those things to me is by my definition at least a blethering moron. I just don't even entertain the viewpoints of people who jump immediately to kind of calling me names. But the disturbing thing about the whole instant is I had a really good conversation actually when this all kicked off, which helpfully occurred while I was mid-Atlantic on a flight from the UK to the U.S.

Mike McQuaid:       I was talking to some of my coworkers, in particularly, in fact my women coworkers said, "See, this is why I never wanna contribute to open source", because this type of both entitlement and massively negative behavior is considered almost like part of the ecosystem, and it sucks, and we try and do what we can to get rid of it. I don't think anyone has any expectations that we're ever going to eliminate this type of behavior, like completely eliminate it such that if you were to repeat this incident, you would not receive a single poisonous email, or comment, or whatever. Yeah. I think it's a much wider question, but I think it's a big thing that we as an open source community need to figure out. How are we gonna deal with the fact that there's a bunch of people, who don't want to contribute to open source, because they just don't want to get shouted at for stuff they work on their free time.

Alex Pounds:        I guess even if it doesn't affect you directly, and you're lucky enough to have that kind of strong constitution, where the abuse doesn't get under your skin. It's still not behavior that you really want to see or encourage in the broader Homebrew community.

Mike McQuaid:       Yeah. Yeah. Exactly. And this is why we had relatively early on a code of conduct with stuff like this. And again, I have no interest in allowing people freedom of speech on Homebrew's issue tracker. The internet is full of ways, if you have a particular thing you want to say about a particular topic, including Homebrew, the Internet's full of ways of publishing your viewpoint. But I don't feel I as a Homebrew maintainer should not have to wake up and read message after message of bile. So as a result, I'm fairly quick with stuff like this to say, "Right, if we get anymore comments in this thread on this topic, you're just gonna be banned immediately. And you're gonna get blocked from Homebrew, and then you can never submit an issue here ever again."

Mike McQuaid:       That tends to have a relatively kind of positive effect. And again, it's kind of useful nowadays with GitHub's ... you can see from people's comments and things, whether people have contributed to your project before, for example, or not, and perhaps unsurprising fact is that the people who are the most vitriol and abusive are the people who have almost no experience in open source, have never contributed to your project, have often never contributed to any open source project before. These are not people you're gonna convert to being great contributors and great maintainers. So I just think it's often easier in many cases to just block these people and move on.

Alex Pounds:        I've heard of people, who have had success using bots to manage some of those kind of aspects of community, especially with anonymous users, potentially even kind of getting better reactions, where a bot tells you your issue is not valid for whatever reason. Have you tried experimenting kind of automating any parts of community interaction?

Mike McQuaid:       Yeah. So we try and do as much as possible with kind of bots, and CI, and scripts, and stuff like that. Again, it's one of those ones where if I had infinite time, the level of bots driven automation on the project, I would go to would be disgusting quite frankly. But one of the main insights I had fairly on with kind of experiment with bots is people tend to respond much better to positivity from a human and negativity from a bot. By negativity, I don't mean name calling, whatever, I mean if your CI build fails, or your bot says, "Okay, RuboCop says that you've used three spaces here, and you should have actually used two." If a human says those things, you tend to think, "Oh, what a pattern. Can you not just merge it and ignore this one little change," whereas most people seem to get if the CI build goes red, and if RuboCop says what to do, then that makes the process much better for everyone.

Mike McQuaid:       Everyone knows what the expectations are there, and in the community management side of things, we've sort of been experimenting with this thing called probot-stale, which my coworker Brandon Keepers wrote. They basically just does sort of similar things with issues, where it basically says, right, if an issue's not got an in progress PR attached to it, if no one's commented on it for a while, I will assume eventually that it's stale, and just close it, because with some of that stuff at least the squeaky wheels get the grease, and ultimately, if one person has complained about a problem that no one can reproduce, we would love to be able to kind of fix it.

Mike McQuaid:       Again, if we've not fixed that within a couple of weeks, then chances are we're probably never gonna fix that at all like on purpose at least. So it's better that, that issue just gets closed. While bots have been starting to close those issues, people seem to be a little bit more understanding that again, I'm not closing this issue, because I hate you, or whatever. I'm closing this issue, just this issue is being closed by a bot, because it meets this particular pattern, and stuff like that.

Mike McQuaid:       A final thing we've been experimenting with a bit more recently is Row, which I'm increasingly excited by, is using RuboCop to do more of our kind of style checks. As a result of that, allowing more Homebrew specific style checks to be ... if you have an editor with RuboCop integration, then it will pop up in your editor saying, "Okay, here's the kind of almost violations in the Homebrew file you're working on right now." And I think that for me is even better in terms of feedback, 'cause then you're not having a command that people need to manually run. You're not waiting for them to see a CI builder, whatever, but they can get that feedback immediately in editor.

Mike McQuaid:       So while they're developing stuff, they can see, "Here are what the problems are, and here's what I need to fix before I even commit a file, rather than pushing it along." I think that's something that would be cool to see more of the open source community experiment with as well, is this idea of how do you push those feedback loops to be as tight as possible, and as close to the user as possible.

Alex Pounds:        So you mentioned reproducibility there, not in the same context that I'm about to bring up, but one of my bug beds of a lot of different package mangers is they don't have the ability to kind of reproduce the state that someone else's dependency tree exists on, say, your coworker's computer. Homebrew has been ignorant of this problem in the way that most other system level package managers have. They won't generate a lock file, or some kind of snapshot of how you successfully manage to install that dependency tree. Is there any plans to add the features to make that available in the future?

Mike McQuaid:       Well, so again, talking to [Debbie 00:44:54], and they've got ... I think they're sort of pioneering the kind of reproducible builds, where their end goal is for reproducible builds of not just dependency trees, but like binary packages as well. Such that I can rebuild a binary package on my machine, and I will get a bit for bit identical result. So I can basically verify that, that package hasn't been tampered with. This comes down to one of those things, where if I had infinite time I would put a lot of it into stuff like this. I think the closest thing we get for reproducibility right now is because our formula repo is ... all the versions are penned at any given Git revision. So you could check out a particular Git revision, and then guarantee that all the dependencies are going to be exactly the same version there.

Mike McQuaid:       In terms of stuff like block files, we don't have that. That's something that feels like it lives sort of a level above Homebrew, and I guess that's sort of segways nicely into ... In fact, something that Andrew made, which was a project called Brewdler originally, we now call it "brew bundle", which is basically a sort of bundle for Homebrew, where you have a Brewfile, which can specify various Homebrew packages that should be installed, and it's basically just a way of in a more decorative fashion saying, "Okay, rather than running brew install this, brew install this, brew install this," you specify these are the piece of software that I want to be installed, and get brew bundle now to install that software, and basically verify that everything is installed that you need to be installed for a given project.

Mike McQuaid:       Again, with that, that also doesn't have log files. The log file stuff for that comes down to, I guess, a Homebrew constraint in terms of the time and resources we have with Homebrew. The problem is when you start to make dependencies more flexible, so if you say, "This software, it needs Ruby 2.1 and up," then if you get an issue filed, where someone says, "Okay. This doesn't work on Ruby 2.16," and someone else says, "Well, it works on 2.17," and someone says, "Well, it works on 2.15." You need to go digging to figure out what it doesn't work on 2.16. In Homebrew's case, our solution to this problem is we say, "Well, we only ever support you in installing our software with the latest version of the entire dependency tree."

Mike McQuaid:       So if you want to have an older version that you want to keep around for a long time, then you make it a dedicated formula that manages that. So say, MySQL is like a MySQL at 5.6 formula, which lets you stick around on that older version of MySQL, and that's a way around requiring the latest versions for everything when that may not match whatever your server might be using.

Alex Pounds:        I get the feeling that a lot of people kind of turning to using Docker on their development machines as kind of a bandaid over the top of the situation of system level dependencies, where they basically can wrap it all up and hide it in a Docker image, which then is system diagnostic, because Docker runs on Windows, Mac, and Linux, but it's actually just you're running Linux everywhere.

Mike McQuaid:       Yeah. I'm definitely seeing a movement in that direction as well, where more and more people are kind of wanting to do things this way. I mean, I haven't done a huge amount myself, so I'm speaking mostly from a position of ignorance, but my experience when I worked on projects that have done this or something similar in the past has been that the development experience is never quite as nice as having a MySQL server that's just running on my actual machine, and I can query the ports, and run brew upgrade to get a newer version, and things like that. This kind of comes down to as well another thing, which makes a literally better tetchy in our industry, which is reproducibility is good, and it's very important. The problem is there's a bunch of tools, which I assume that reproducibility is the most important thing, and as a result assume that using log files in the same version every time are very important.

Mike McQuaid:       What I've seen the result of this in practice tends to be a lot of people install some new gem, and then whatever the tooling they use, locks it all to the same version. And then what happens is in the nice future or present reality, where everyone's doing what they should be doing, then they will monitor that gem, and then when the security update's released, they will then update the gem, and their gem file, and make sure their security updates are done and stuff like that. Whereas what tends to happen most of the time is that gem never gets updated ever again, unless another gem requires it to be updated. So they can have security vulnerabilities, and no one ever notices, and no one ever updates, and no one ever cares.

Mike McQuaid:       This is one of the reasons why I've resisted this stuff in Homebrew, particularly ... I mean, it's bad with a random Ruby gem, it's incredibly bad if you're dealing with something like MySQL, and you get people who say, "I wanna be able to pin to version 5.6.21," or whatever, and the latest version is 5.6.27. Okay. There's bugs that have been fixed between those differences, but a lot of the time there's also security vulnerabilities that were fixed. If you're gonna run a server, which is gonna be accessible over the network, pinning intentionally to a version, which has no insecurity vulnerabilities is a very, very bad idea in my opinion. As a package manager runner, I will rage against the dying of the light in trying to stop you from doing that.

Mike McQuaid:       I appreciate that with production software that idealism gets trotted on in the favor of kind of pragmatism and actually being able to test on the same version, and things like that. But I do generally feel that with the Docker world, you're seeing similar things like that happening, where the people who are doing it properly are making sure that all their system dependencies are kept up to date, whereas you do see a lot of Docker images being throw around, where people are not paying any attention to the security vulnerabilities that are being filed and fixed in their images sometimes. And then as a result, they are pleasantly being happy with the reproducibility of it all while ignoring the security consequences that come from that.

Alex Pounds:        Yeah. I agree. The reproducibility aspect of Docker itself tends to actually be more like a freeze dry. "We'll just put this in the freezer, and we can always take it out and reproduce it." The problem is that as soon as you take it out of the freezer two months later, you've now picked up a load of potential security vulnerabilities that like if you regenerate the Docker image from the Docker file you get a completely different set of dependencies inside of that system. So you haven't really got reproducibility at the kind of the top level. You've just generated an image that will continue to run anywhere at least in the short-term.

Mike McQuaid:       Yeah. I've not really seen ... I think you need much smarter minds than I to figure out how you balance that kind of reproducibility versus security aspects. It's not really clear. I've seen various solutions and some of them just rely on I guess, pragmatism, where I've seen people who have when you deploy a server, say, it effectively does a security release update of all of the software on that system, and then it starts blowing up, then you know that you need to fix that. But again, if you get to even kind of get hub scale level and larger, it doesn't really work to just deploy things, and have them explode. So I don't know.

Mike McQuaid:       Because at the end of the day, it's up to users what software they run and don't run. So it's as long as you're providing as much information as you can on "here be dragons", then it's up to them to not get eaten.

Alex Pounds:        So in terms of community and governance, how can somebody get involved with Homebrew?

Mike McQuaid:       I think generally the best way ... I guess the automatic pedantry and style checks that I have described earlier on, if you push everything to maximum pedantry, there's basically issues all over the place with loads of our formula. So that's generally a really good place to start is that you can run brew audit --strict, which basically says, "Maximum pedantry" effectively, and then make a modification, fix one of those literally warnings, and then submit that. And then basically, as you submit more and more stuff, we will notice you more and more, and set the bar a little bit higher and higher each time for that. And then eventually you get asked to be a maintainer, if you've made a significant number of nontrivial contributions over a reasonable period of time.

Mike McQuaid:       Or in general, if you are saving us more work than you're causing us.

Alex Pounds:        Where's the best place online for someone to find out more about Homebrew and getting involved in it?

Mike McQuaid:       I think our two main places would be our website, which is https://brew.sh, and GitHub Repository, which is linked from the website as well.

Alex Pounds:        Great. Well, thank you so much for coming on and joining us for this chat. It's been really interesting to learn more about Homebrew and the people who are behind it, and some of the issues, and the things you try to balance over the years. If somebody wants to learn more about you personally, where should they go for that?

Mike McQuaid:       Probably my website, https://mikemcquaid.com or my Twitter, which is @MikeMcQuaid. You won't be able to spell it, so I'm sure it'll be written down somewhere.

Alex Pounds:        Yeah. We'll drop that in the show notes. Thank you very much again to Mike, and that's everything for this episode from me and from Andrew.

Andrew Nesbitt:     Very nice.

Alex Pounds:        Thank you for listening. Goodbye.