JOHN: This episode is sponsored by Pantheon. Pantheon is the platform of choice for more enterprise Drupal and WordPress sites than any other platform. A platform with superpowers needs to be run behind the scenes by superhumans of all diversities and backgrounds. Pantheon actively supports non-profit initiatives throughout the Bay Area and beyond such as Techtonica, the Tech Equity Collaborative, and Lesbians Who Tech. Learn more about career opportunities at pantheon.io/GreaterThanCode. JESSICA: Good morning. Welcome to Greater Than Code. I'm Jessica Kerr, and I am happy to be here today with Rein Henrichs. REIN: Hi everyone. It is too early for me to summon that level of excitement, but I will try. I am here with John Sawers. JOHN: Thanks, Rein. And I'm here with our guest, Amitai Schleier. AMITAI: Yeah, and I don't have any time of day when I can ever summon that kind of energy. JOHN: Amitai joins us as an independent software development coach, legacy code wrestler, non-award-winning musician and award-winning bad poet. He publishes fixed-length micropodcasts at Agile in 3 Minutes, writes a variable-length articles at schmonz.com, and contributes code and direction to notable open-source projects such as NetBSD, pkgsrc, ikiwiki, and qmail. Welcome to the show, Amitai. AMITAI: Thank you very much. It's nice to be here. JESSICA: Oh, yeah, qmail. Because you did a talk the other day about why you should find your own mail server. AMITAI: I didn't do that. I wouldn't say should. [Laughter] AMITAI: But check it out. It's this thing you could do and it might be rewarding. It's not as hard as you think. But maybe not should. I guess that's eventually my goal with that project is that everybody, it would be so easy and so reliable, so safe that everyone should, but that's not quite true. REIN: It's hard to make email easy in any capacity. AMITAI: True. JESSICA: Oh yeah, email is one of those things that sounds way easier than it is. JOHN: Oh yeah. It's been a long time since I wrestled Sendmail, but I never want to go back. AMITAI: I'm 100% with you on that. In fact, that's how I got into qmail. I was a college dropout. I worked for like an ISP except that they would accept any kind of business from anybody and so they became one of everything. And one of the things they did was email hosting and this was 1997, 1998. So, qmail just came into existence. For those of you listening to this show who may not be intimately familiar with the history of mail servers on the internet. First of all, good. [Laughter] AMITAI: But that was a really exciting time because up until then Sendmail that John mentioned had been incredible headache for sysadmins because the configuration language, first of all, a lot of sysadmins didn't realize that there was a source language and a generated language. And they were trying to write the generated language. And the generated language look less comprehensible than stereotypes of Perl. It was just a total zoo. But you did have the option to write in an input language and then run the m4 macro processor [inaudible] more comprehensible. That part was okay. Then you run Sendmail that has run time bugs and security holes and gets you in trouble all the time. And so, it was just like there was no good way to run email on the internet. There wasn't like a full time job. And then qmail came out. JESSICA: Now, with Sendmail open source, was that a thing back then? AMITAI: It was. Open source was less of a defined thing back then. It was just sort of a habit that people on the internet were in of making stuff and putting it out there. I wasn't so formalized. It was around that time, I would say in those few years that Richard Stallman got very serious about free software and Eric Raymond and others got very serious about open source software and why they're not the same and why we're actually enemies, and oh my God. [Laughter] JESSICA: Your worst enemy is also your nearest neighbor. AMITAI: Yeah. It was getting so much bigger so faster. And [inaudible] also written in a time when the constraints were different and then they changed out from under it. And with legacy C code in a world of maybe the internet is going to have maybe a few dozen hosts maximum and everybody running an instance on a computer will be a trustworthy person. And so the software itself doesn't have to be so defensive because there won't be so many people running it that we would have to like not know who one of them is. It's never going to happen. That happened. It happened a lot. And qmail came into a world where we got to run mailing lists, we got to run email servers on the internet. We want to not update them all the time. We want to not get broken into because of our email servers and it was just a totally different, more like it was a Unix-y design and architecture for components that you could compose yourself into an email server that did what you wanted. And Sendmail was like one big honking thing. Qmail was a bunch of little things that fit together according to defined interfaces. And it's 20 years later, it's still a beautiful piece of software. It's got rough edges, but it's still beautiful. JESSICA: Awesome. It usually takes a bad one before anybody can write a good one or a couple of bad ones to learn from. AMITAI: Yeah, for sure. And there wasn't only Sendmail, there were others. Another thing that came out at that time, we now know as Postfix, which is the de facto standard today, just like Sendmail was the de facto standard then. It's a much, much, much, much, much better de facto standard. [Inaudible] That never existed, I'd be a happy Postfix user. But qmail came to me as a 19-year old, a really formative time in my life and it had this really weird tension between an excellent application of Unix design philosophy, more so than Unix itself. Therefore, like why are we writing code in every daemon process to disassociate from the [inaudible]. And close our open file descriptors, [inaudible] and have some kind of a logging framework so that it goes to syslog and detaches itself and runs in the background. Why does every process have to do that and some of them get it wrong when you could have one process that knows how to do that for all the other ones and all the other ones can be regular processes? Read standard in, write standard out, log a standard error and not worry about it. And this was cool. It's still cool. In fact now there's like half a dozen derivatives of the qmail authors daemon tools, one of which is like the default thing that docker does to manage processes that started life as inspired by daemon tools [inaudible] piece of software. JESSICA: Okay, okay. Hold on. Daemon tools. Is that the thing that lets you in Unix just write standard in standard out little programs? AMITAI: Yeah, it's a little handful of programs so that you can turn any program that read standard in, write standard out and logs the standard error into a daemon by running it under daemon tools. And then only one thing has to know how to disassociate and run in the background. JESSICA: It sounds like Istio. [Laughs] AMITAI: Is it? JESSICA: Much, much, much later. AMITAI: What is Istio? JESSICA: Istio helps you run a service mesh and it's a little sidecar that you put next to your services in Docker while in pods in Kubernetes and it's supposed to take care of the back pressure and exponential backup and generally make services be nice to each other without actually knowing how to be nice. AMITAI: Okay. Yeah. So, it's the same kind of layer abstractions. JESSICA: It's an abstraction of the network. AMITAI: Yeah. So that was an example of something that people either loved or hated. Qmail had a lot of things that people either loved or hated. You could love that because Unix says have each program do one thing well and specialize. And yet for daemon purposes, it hadn't. And this is a better design. Or you could hate it because you're running an operating system and it has its own startup procedure and it ain't daemon tools. So for all of these design decisions that qmail made, it was like an extreme Unix up to 11. And if you wanted that and you were prepared to live with the attention of that, like the the host system not being as Unix-y as the thing you're running on it. JESSICA: [Laughs] REIN: As fun as this conversation is, we forgot to ask... JOHN: I was just going to say it. JESSICA: I know, I know. I can clearly talk forever about mail service and I am like weirdly fascinated by all of this history, really. You know, the Unix [inaudible] open source and how hard email is. But let's do the thing. REIN: I saw a tweet that was something like what you refer to was email is actually SMTP mime MTA Postfix Dcam. [Laughter] REIN: So anyway, now the question that we always ask right at the beginning of the show before we do anything else... JESSICA: [Laughs] Sorry. REIN: Every time, Amitai, is what is your superpower and how did you acquire it? AMITAI: I had an answer in mind. I'm going to give another answer because of what just happened. JESSICA: I am distracted and derailed. REIN: It's specifically [inaudible]. JESSICA: It's all contextual and we have altered the context. AMITAI: Squirrel power is one of my superpower. [Laughs] AMITAI: The more serious answer is, and this is super useful given my line of work and in fact it led me to this line of work, is that I'm really good at orienting myself in a new problem space. And then because of the reason that I'm good at it, I'm also good at helping other people orient themselves in their own problem spaces. And as a consultant and software development coach, that's a regular thing. I land in a new context and what's going on here and there's eight different contending ideas for what's most important to do next. Let's get oriented together quickly. I don't get to ask 74 questions just because I have 74 questions. I got to pick the right ones to ask at the right times where people can give the right answers because they're comfortable and they believe they can tell it to me. And so, that's my superpower is I can help other people orient themselves quickly in their problems. REIN: How do you do that? What tools do you use that enable you to do that? I'm going to highlight more of the constraints because I think that's what makes it possible. I'm not inherently good at this, which is why I'm able to help other people with it. I'm good at orienting in my own problem space because the default thing that happens to me with my time is that I squirrel myself. So, it wasn't entirely a joke answer. I can easily not have a goal or drag myself away from the goal to whatever looks more fun right now. Or I could set a goal but then rabbit hole myself as well as any of us plus fun. So, like the default thing that happened when I was a new programmer as a young person was I started out with the best of inventions and make a nice big design that seems like it's good enough to get going. And then as soon as the complexity reaches a certain level, I'm in trouble because I don't have a way to tell whether all the things I thought of before are still being thought of now. And so I no longer trust that I have solved as much of the problem as I think I have. It doesn't seem like it's cumulative anymore. I might be playing whack-a-mole instead of knocking something down and losing something else. And so I got of necessity, like by default I'm not effective in a business context and I had to learn how. And so these constraints about how my brain works and how I rabbit hole and how I squirrel and how I just don't have the focus or discipline led me to XP. I got lucky. So for example, clean code, the idea of clean code never sang to me. I never needed reminding to clean my code or what that would look like. I needed help to stop. And for me, everything Kent Beck said sounded like somebody who had the problems that I had and then figured out way more what to do about them. And in particular, TDD, test driven development, for me is a thing that not only changed my ability to cope like getting my programming work done at work, but it changed the way that I related to reality in a way that took me away from my totally ineffective perfectionism. And it took me toward a bounded perfectionism where this system works exactly as well as all of the questions that I've asked it to do so far. And if I want it to be more perfect, I can ask it one more question and I can make it give the right answer. And I can find out instantly if the other answers are still right. And this helped me across the board in life to not have to solve so much at the beginning and be prepared to solve every little thing all the time as a matter of course. And make habits where I'm comfortable with complexity, but also comfortable keeping an eye on it as I go. And because by default I'm not an effective person in a business context, I had to pick up these skills that I can then ask other people the same questions I ask myself. I spot them making the kinds of mistakes that I have mastered because I really recognize them really early. And I think that's basically it. REIN: One of the things I say a lot is that the reason I write simple code is because I'm not good enough to write complex code. I can't keep the things in my head that would enable me to write code that's that complex. I can't even keep a conditional with more than two variables in my head at the same time. So I just literally can't write complex code because I very quickly exceed my ability to comprehend the thing I just wrote. AMITAI: I was just saying that to somebody earlier this week. When I run into code that I don't understand at a glance and then I put in some investment to understand, I'm only willing to do that once for any given block of code because that is all the patience that I have. I will do it once. I do not want to come back here and have to do this again. I will lose that work day. I'll just flip the table and walk away. I don't have patience for that. So I have to leave it in a way that I can glance at it later or else I'll never get any work done. JESSICA: Are you saying you refactor it? AMITAI: Yeah. I leave it simpler than I found it and with his higher degree of assurance as I can get that it behaves the same, just that I understand it now. JESSICA: You understand it, but what about the person who wrote it before and could understand it at a glance but can't now? AMITAI: I'm happy to talk to them if they run into that problem next time and maybe we can negotiate. What I would really rather do is hopefully we're working as a group and then we [crosstalk]. But if it was just me doing solo open source stuff or whatever, then I'm not going to be able to move past. If I had to wrestle with it, I'm not going to keep walking until next time I'm not going to wrestle with it. JESSICA: Yeah. The hard part is knowing. I mean, I'm tempted to like refactor stuff, but then I get it to where it makes sense to me. But that doesn't mean it makes sense to other people. There is no universal simpler. There is only familiar. AMITAI: In general with refactoring, I think we don't put enough context on what we mean when we say refactoring in general that's why it means so many things to so many people. So we say it means changing the code without changing the behavior. Behavior to who? To me as I'm reading it? To the caller as it calls it? To someone else as they're reading it? To someone else as they're changing it? To the performance of the system as a whole? A pretty common way to say refactoring among people who think we know what it means is that all the colors are the same. It's just the internals that look different and otherwise, it's not different. And what we're saying is we also assume that everything else that could be observable about this doesn't matter. REIN: Or we're assuming that what we understand about the behavior of missing is actually everything that there is to understand. JESSICA: Yeah, or that we care about. That we can ignore performance. REIN: Or that there are side effects that we just haven't found yet. AMITAI: Exactly. And so I think a really useful frame for thinking about refactoring is doesn't change behavior we have decided not to care about. And that includes behavior we don't know. That helps. JESSICA: Sorry, this sounds backwards to me. You said refactoring doesn't change behavior that we have decided not to care about. AMITAI: I'll have it rephrasing, maybe it helps. If we had an absolutest view that refactoring means like the inside of the function is different than the outside of the function isn't in [inaudible] nothing else is, there is no expression of a risk there. JESSICA: Okay. AMITAI: There's risk. And if we state refactoring in terms that says we think we have mitigated all the risks we care about with this change, we're going to maybe be wrong about that. Maybe there are aspects that are affected that we didn't think of and we're saying fine. So, I might be nuts because in general on teams that I'm on, XP for me is a risk management set of techniques. JESSICA: I agree with your clarification. AMITAI: I tend to be [inaudible] about risk than other people on my teams. JESSICA: Is it we or are you explicit? AMITAI: I think with risk, that's the whole game. Like if you're not explicit about it then it's not being addressed. JESSICA: Right. But addressing risk doesn't necessarily, you don't have to do stuff about it. You don't have to write performance tests for everything that you refactor. You just acknowledge, yeah, we're assuming that it's not going to change the performance in a way that we would care about. It might change it in a way that we find negligible and that's okay. REIN: There are different ways to mitigate risk and just underwriting the risk and not changing it is a valid one as long as you have identified the tradeoffs well enough that you're making an informed decision, right? AMITAI: Yeah, and often in a case like this, it's just a matter of, "One thing I'm slightly worried about and I don't think we should change what we're doing, I'm just going to say it out loud is this." And then the other person says, "Yeah, I don't think we should do anything about it." Or maybe we'll say, "Let's talk about it." Usually, it's nothing and usually I say it that way, but I'd rather us mention it just in case. REIN: So I get to name drop Virginia Satir which checks off that box for today. JESSICA: [Laughs] REIN: One of the rules that I live by is her 'make hidden things visible, make abstract things concrete and make implicit things explicit'. AMITAI: That is awfully good. I haven't heard that. JOHN: Yeah. AMITAI: Things to live by. REIN: So one of the things I was going to mention earlier when you were talking about how refactoring safely managing risks and while at the same time dealing with the amount of complexity your brain can understand and cope with that has been super effective for me is really aggressively black boxing things. AMITAI: Say more about that. REIN: So if I'm doing a refactoring of something, I will very intentionally draw a circle around some other part of the system that it touches and say, "I will not touch this thing inside this circle. These are the bounds of my refactoring." And I try to make them as small as possible because that contains the risk. And also because if I go, "Oh, this is connected to that and that could be improved," then I will start branching off into improving every different thing in the code. And if I improve 20 things that are all interdependent and I have a 1% chance to make a mistake any place, there's like a 50% chance that I screwed up. AMITAI: Listening, I was doing a lot of vigorous nodding for all of that. JOHN: Yeah. REIN: So I really try to put very aggressive bounds around the thing that I'm refactoring, even to the point where if I need to change the interface to some other object, I will make a new standard object that will mediate that interference change and I won't touch the other objects. AMITAI: Yeah. I think we're very aligned on this. Another thing I'm worried about is not just the risks of the mistakes that I'll make but also the risk that now I'm being a perfectionist in a bounded way and I'm just fixing the whole world and I forgot why I got here and I love refactoring and this is great. What was paying me for what? That's a real risk for me. JESSICA: Bounded perfectionist. JOHN: It's great, yeah. AMITAI: That's why I'm alive. I don't think I'm overstating the case. I was a perfectionist with like no good adaptive behaviors in the world. And then TDD was my first. JESSICA: First good adaptive behavior? AMITAI: As a perfectionist, yeah. It was a big deal. It changed my whole outlook. JOHN: Yeah. The way you're talking about that is really still sinking into my brain and I suspect it will take a while before it's all the way in there. But I really like the idea of starting with that outside bound, like the tests as the bounds of what you're working on and using that to define perfection rather than some sort of abstract ideal. And that way you can code to the test, but you can always also keep it circumscribed so that you just don't run as far as you can run till you run out of energy. [Chuckles] AMITAI: Well, and the flip side of unbounded perfectionism is another problem that I used to face in the old world without TDD is how to take a first step in the Greenfield. And it used to be, I will stay in my thoughts for a very long time and I may forget again when I'm supposed to start and take a step. And so the way I get over that now is this exaggerated other thing where I will write a deliberately wrong, stupid, funny test. So that I'm moving, and it's obviously not right. It's like Cunningham's law, but for my own code. Cunningham's law for the internet says the way to get a right answer is to give the wrong answer. And this gets me started. Like I put something that clearly aim it, but now I'm moving and I can [inaudible] from here. JESSICA: Or in physics, there's static friction and dynamics friction. And static friction is a lot stronger. JOHN: Reminds me of I think it was probably apocryphal, but the story is that Michelangelo, when he was done sculpting for the day, he would take a chisel and just whack anywhere on the statue before he left for the day. So he'd know exactly where to start in the morning. AMITAI: Ooh, I like that. I have something that's sort of like that. I haven't gotten to use it a lot lately because I'm working remotely. But when I'm in an office and people are tapping me on the shoulder all the time, I'm living in my test suite, so it's a very easy thing. Someone comes over and says, "I need you, " I just run a little red test with a fail and a string. And the string is "Yo, the thing you were thinking about was [inaudible]. And I can type that [inaudible]. The first thing that I do anyway is run test and it tells me, "Yo, the thing that we're thinking about is [inaudible]." So, I'm no Michelangelo, but to cope, I needed something like that. REIN: Yes. JESSICA: Was it on Twitter the other day? I think it was [inaudible] was going on about how -- stop complaining about being interrupted and ask like developer time is sacred like no one else's time and freaking develop a system that lets you deal with it. AMITAI: I agree with him. I also think that the message that we're hearing when people say developer time is sacred is depending mostly from people who don't give a shit about developers' time and it's an overreaction. But until it's recognized, people are going to keep overreacting. REIN: I think my solution to that is to expand the definition of developers' time to include activities that aren't just programming. And then it's a question of how you manage that time. Yeah, your developer time does include talking to other people. It does include reviewing other people's code. It does include looking at tickets and issues and stories and grooming things. That's all part of your time and it's your job to manage it effectively. JESSICA: If you don't do any of those things, then you won't have an understanding of why and what you're coding. REIN: I mean, as a senior developer, there are days when I don't write any code at all and sometimes those are days that I feel are some of my most productive. JESSICA: Or even better, delete code. AMITAI: You know, this makes me think of this Coderetreat. Who here has done a Coderetreat? I've done a Coderetreat. Coderetreat is a thing, there's one every year on Global Day of Coderetreat but it's just a one day format that you can do any time. I've administered a couple. The other who thinks that Coderetreat sounds like a good idea, so that filters a bunch of people out. Often it's a Saturday and there is a problem that's pretty well understood or will be pretty soon, and that's usually Conway's Game of Life. And the whole game is you pair with five different people over the course of the day with initially no constraint on the problem other than you only have 45 minutes so you're probably not going to finish. And the other constraint is that once 45 minutes are up, you delete the code. And so, Rein, when you were saying one of those days with no code could be one of your most productive days, that makes me think of a Coderetreat where you walk out of that room at the end of the day with no code, but your brain is different. You've learned some tricks like one of the constraints could be baby steps where if the timer goes off after two minutes and [inaudible]. There's another one where you're not allowed to talk to each other. So the tests are the communication with the intent. There's another one where you're not allowed to talk to each other and it's also called evil pair. So you try to subvert the intention will pass the test, but not the way you wanted me to. And so this push on all the extremes of what's involved in getting code done that does what you mean and conveys what you mean and is understood by your teammates, with different teammates even, just pushes on all the extremes, turns all the knobs. And at the end of the day, you got nothing to show for it except you which is a lot. REIN: Yeah, it's really interesting to explore the boundaries that define sort of your comfort zone. So for example, I used to commit pretty infrequently and then I decided that I was going to change that. And the way I did it was by writing a VIM command that would commit to get every time I saved my buffer. And because I habitually [inaudible] W, what I found is that I was committing about every 30 seconds. And after a couple of weeks of that, what I realized is that was actually way too much and I could back it off. But the place that I backed it off too was an order of magnitude more frequent commits than I started with. AMITAI: [Inaudible] that one? REIN: Pretty much, yeah. AMITAI: So that was a really effective experiment in one shot. How long did you do the commit on [inaudible]? REIN: I did it for two weeks. AMITAI: That's long enough. Cool. Interesting. So what would you say is your average duration? REIN: Probably 15 to 30 minutes. AMITAI: Cool. And do you keep those as is? Do you squash them? REIN: I squashed habitually. Now that git has the fix-up system, it's much easier. The pain I originally felt in squashing has been virtually eliminated now. JOHN: Yeah. I love fixing up. AMITAI: So I bet that relates to our challenge that [inaudible] threw down, which is how can we change our work habits and work systems so that the things we do are smaller and the cost of interruption is smaller and maybe there's just more value than cost. REIN: It's really interesting because many things have transaction costs and often those transaction costs are hidden and it can be difficult to find the inflection point if you can't visualize all of the costs. AMITAI: And by committing more often, do you encounter them more often? REIN: Yeah, there's a fixed time it takes for git commit to run, and then there's the overhead of all the squashing and there was a point at which that dominates the returns you get from committing faster. And I think the same is true for like -- there was a reason we don't do one day sprints. You know what I mean? And I think a big part of that reason is that there is a significant transaction cost to the sprint. AMITAI: You know, that makes me think of, I think this was John Cutler who I first heard the idea from which is that we got to try one day sprint. If it's hard to get done in two weeks, you're sprint should be one week. If it's hard to get done in one week, it should be shorter. What if we could just start a day with the theme and finish the theme at the end of the day and then get some feedback about it. I've also heard if you don't have time to meditate for an hour, then you should meditate for two hours. If you don't have time for a retrospective, you should do a long retrospective, stuff like this. There clearly is a point where the returns diminish. But I can say as a consultant, I've never run into iteration to short feedback excessive cut it back. I've never seen that. Someday I will, but I haven't seen it. REIN: I facilitated a retro for a company, a small company. It was about 20 people in the retro and it was the first company-wide retro they've ever done. And so, I allocated three hours. They were like, "We won't use three hours. That's too much time." And I said, "Well, let's see." And we ended up using the entire day and we still weren't done. AMITAI: What did they do after that? Did they schedule some more? REIN: They started doing it every year. Yeah. AMITAI: I feel like everything we've talked about connects everything else and I'm trying to [inaudible]. I don't know which one. REIN: For me the threat is feedback systems. AMITAI: Yeah. REIN: That's always the threat that I experience. JESSICA: It's a feedback system that tells you when you're allowed to expand your perfectionism a little bit. AMITAI: Nice. I like it. JESSICA: My sister just texted me and she's learning programming and, this is cute. She was like, " I know how to compare two numbers to see if they're equal, but now I need to say which one is farther away than the other." And I was like, "Less than?". She was like, "Oh yeah! I've forgotten that concept." It's adorable that she had not had to deal with math at all for so long and this stuff that is like, we don't even think it, we only type it. REIN: Wait, wait, this is interesting. Farther away from what? JESSICA: Oh, one number is farther away. She has like three numbers - A, B, and C and she needs to say which of A and B is closer to C. And she doesn't have in her head the less than operator yet. REIN: But it's not always less than. What if they're below the number? JESSICA: Yeah. Absolute values. I told her to check absolute values too but those are concepts that we don't even think them. We just breathe them. And it's so hard. I mean, I'm proud of her for asking the question because that's like, "I know this is stupid, but what is the thing?" REIN: It's not at all stupid. It's 'which one of these things is farther away' is the basis of an entire field of mathematics called metrics basis. It's not a stupid question. JESSICA: But she's okay with just absolute value and the less than operator. REIN: Yeah, which is the metric for that space. JESSICA: I am not going to say that to her. Actually, last time we paired, I couldn't resist bringing in entirely too many types and it set her back by weeks because she switched from programming to embroidery for a while because I pushed it too far. REIN: What if she had said, "Which one of these words is farther away?" AMITAI: I use hamming distance in the commit [inaudible]. REIN: There's a metric for that space too. JESSICA: That's fine. There's a thing for that and then you look it up. But when all you need is less than, don't go into metric spaces. REIN: But they're so cool. [Laughs] JESSICA: It's not about, Rein. It's not about you. REIN: No, it's about metric spaces and how cool they are. AMITAI: [Laughs] What this makes me think of is a daily occurrence in a mob programming scene, which is that the person who is navigating and the person [inaudible] everybody else in the room has to negotiate what it is that we're talking about when we're talking about this right now, and how are we going to talk about it in a way that we can all talk. I'm not ready to talk about metric spaces. I like math, but I'm going to need a lot of practice on that. I am willing to talk about absolute value, that's regular for me. And it might not be unusual if someone on my team hasn't seen less than yet. And today's when I learned about it. JESSICA: Yeah, which is cool. I'm tickled to get to share such a tiny thing with her that adds a lot of power without adding any mental burden because she actually does know less than. REIN: It's the use of less than that is interesting. It's using less than to say which of these things is further away is cool. AMITAI: So when she comes to you later and says, "I'm wondering if there's some kind of a generalization of less than." JESSICA: If she ever says that... AMITAI: Then you'll know. [Laughs] REIN: You know, which of these things is farther away. JESSICA: Yeah. Or if she does need to measure the distance between strings or sentences. It's really interesting in maps too, like in geospatial coordinates. What is the distance between here and Costco? Well, that depends where the roads go. AMITAI: Or if you're confined to them. JESSICA: Yeah. And when there's construction. Yeah, our communications are limited at an absolute level by the speed of light. But really they're limited by so much more. AMITAI: Yeah. We would be doing well if we're actually limited by the speed of light. JESSICA: Yeah. But network latency is fascinating because it's not constant. So, okay, my latest abstraction to think about is just the concept of distance, which is probably similar. Maybe I should look at metric spaces and not inflict it on my sister. [Laughs] JOHN: Kind of reminds me of the case of the 500 mile email. JESSICA: Oh, I love that one. So we were talking about email servers earlier, do you want to tell that story, Amitai? AMITAI: It's all connected. I don't think I remember all the details. I just remember that it's like a shaggy dog story and eventually it's just a string of surprises. John, do you remember more? JOHN: I've read it recently, so I'll try and get all the details right. It was a site was reporting that they can only send emails to certain places. And through a whole bunch of debugging, they figured out that the emails could be sent literally 500 miles and then any other place farther away, would not get delivered. And when you look at a ticket that comes in that says, "We can send email literally 500 miles but no farther." JESSICA: [Laughs] But you don't understand what email is. JOHN: Yeah, like what is happening with [crosstalk]. AMITAI: Flows rejected. JOHN: You know, through all sorts of debugging and interesting things, they discovered that there was some quirk of the network configuration that was causing delivery failures because of timeouts. But anything closer than 500 miles would return the response quickly enough because of the speed of light was literally in the amount of milliseconds of timeout that were in the network stack. JESSICA: The timeout was so low that it was limited by the speed of light. And it was physically impossible to get an acknowledgement from a computer [crosstalk]. AMITAI: Awesome. That makes me think of, it was like decades ago around the time that I discovered qmail probably. Some paper or papers theoretically how you could store information in motion networks. I don't remember any details other than like you send it and you keep sending it and then there's a way to look at it when you want to look at it. But the data is stored by being in motion on a network that was the [inaudible]. JESSICA: So like oral history. AMITAI: Yeah, possibly less lossy. JESSICA: Maybe, that depends on the motives of the actors on the network. AMITAI: Also true. And the speed of light. Yeah, I wish I remember details of that. JESSICA: Yeah. One reason that I loved programming when I started was because you could be perfectionist with it. You could predict the results I thought then. But then as soon as you get bigger than an individual process, you really have to give up hope of full predictability. AMITAI: Yeah, it's true. And it seems like unnecessary coping strategy, at least for me is in the small. You have to keep any particular interaction in the small predictable. Once they're fitting together, we're going to find out. But at least the small ones have to be. That's their only chance. JESSICA: Right. Because like Rein said earlier, if you have 20 of them and they each have a 1% chance of failure, you're doomed. So we have to get the multitude reliable. And then the other day I was talking to Martin Thompson who pointed out that you want your database to guarantee consistency through two-phase commit or whatever. But sorry, not a thing. Exactly [inaudible]. Not actually a thing. And that if you accept that, it's quite easy to add item [inaudible] in the application level where you have more context about like what message has invalidated what other message, for instance. And then you can do the ordering at that level and just drop old messages [inaudible] it's really physically impossible to get corrected in the network level. And the harder you try, the more overhead you're adding and you're not really working with reality when you try to enforce certain guarantees. REIN: Yeah. One of the universal challenges for humans is how do you cope with systems that are more complex than a single person can understand. And this shows up over and over everywhere. Every company at some point reaches a boundary or they stop existing. But if they don't stop existing, then I reach a boundary where the company is too complex for any one person to understand at all. JESSICA: And then you have to talk to each other. And what good is that? They'll interrupt you all day and you'll never get anything done. REIN: Many teams then reach this boundary in successful companies. And so the question is how do you remain viable as a system when you're too big for one person to understand. JESSICA: Yeah, Amitai, how do you scale agile? AMITAI: Oh, that's easy. [Laughter] AMITAI: Just detach from worldly outcomes and [inaudible]. [Laughter] JESSICA: [Crosstalk] your perfectionism to following the process. AMITAI: Sure. REIN: As Gerry Weinberg once said, if it doesn't have to work, you can meet any other requirement. AMITAI: And I think [inaudible] version is if it doesn't have to work, it's done now. [Laughter] REIN: The other one is, if it's not worth doing, it's not worth doing right. JOHN: Oh, that's a tough one. JESSICA: Except for the other product of doing it, which has changed you. REIN: Yeah. JESSICA: Which is why the Coderetreats, you're solving a problem that's not worth doing for the next version of you that comes out of that process. JOHN: There is such retreat coming up in September called WeCamp that I'm hoping to go to this year. JESSICA: Cool. Where is it? JOHN: It's in Northern Virginia. we-camp.us. AMITAI: Yeah. The Global Day of Coderetreat that I talked about is usually third or fourth quarter and it's just held in dozens or even hundreds of places around the world. And so, there's often one near you. We can put that in the show notes too. JOHN: Nice. I like the idea of the switching up the pairing and the code being irrelevant to it and learning, like using that to stretch the boundaries of different types of pairing and learning, using it to learn those, because I feel like it's so easy to fall into a single style of pairings especially on a team where all the sort of norms point into a certain direction and you might not get a chance to try out other ways of doing it or learn more effective ways because you've had a local maxima. AMITAI: You get a lot of compare and contrast in the course of one day. Jess, when you were giving an answer involving metric spaces to a less than question, that would've been too much feedback. We're talking about feedback loops. That would have been too much. That would've been the case. That would have been one case or when you drove her to embroidery for a few weeks instead of programming. JESSICA: Yeah, I gave her too much information. REIN: So this is the other hard problem in society, which is how do you deliver the right information at the right time. AMITAI: It's like refactoring. It depends for whom. JESSICA: Absolutely. Communication is not message passing. REIN: And knowledge isn't shared. It's constructed. Amitai, you were talking earlier about refactoring and how do you negotiate the different values people have around the software and what it does. And I was reminded of a thing that Gerry Weinberg said, which is that quality is value to some person, which means that it's contextual, it's inner subjective, it's socially constructed. And one of the questions is how do you know whether you are considering the right people when you decide what quality is. AMITAI: One interpretation of Gerry said, I guess maybe a straight forward one is there isn't one quality. There are many, many, many qualities. REIN: But you have to have some way to manage that complexity. AMITAI: Yeah. JESSICA: Like in refactoring, you choose what you care about. AMITAI: But I think Rein, you're asking sort of the meta question is how do you know that you are caring about enough and not too much? REIN: For example, one way that you can manage that complexity is with personas, right? That's the typical agile-ish sort of thing. You divide your customer base into personas that describe some way of using the software. But that's reductive. That's taking this large complex space and reducing it to something that can't possibly capture all of its variety. AMITAI: Yeah. No matter what we do, it will be reductive. One idea of Joel Spolsky's that I do still like is the hallway usability testing idea. So say we have like an internet service with a million users, we're not going to go get a million opinions. That's too much feedback. We can't process that usefully. Or we give them a really narrow straw to vote one in four options and then it's manageable feedback but we lost information. Somewhere in between there is useful. And Joel's idea from a couple of decades ago is if you get five or six people, that's about enough. You could get more, you might get a few more outliers, but five or six people's opinions, if it's sort of sufficient variety of people will fill up a lot of the space. It's like your 80/20 solution. You can learn a lot pretty quickly without working too hard, without bothering too many people and make a big step in a better direction. I feel like that happens on a daily basis with my programming. When we have a team of about that size, not usually a lot bigger. It's not usually a lot smaller but as we're constructing the thing, the opinions about how we should make it and why we are making it and who this is for, maybe some of those people are even there. It's not infinitely many people that we couldn't handle the feedback and we never get anything done. It's not only two of us or only one of us where we're guessing a little bit. We're going to have some surprise inputs and we're going to have to figure out how to incorporate that into each small batch of work. And at the end of it, we've negotiated a little more firmly the ground that we're standing on for the next batch of work. And so I feel like practically speaking, it's not a solution, but it's a really good heuristic to work in a group about that size. JOHN: Yeah, I think that's great. I feel like I could do more of that. Like anytime I'm stuck with a design problem, even if it's a smallish one, I can come up with a design myself and code it up and then put it out for PR and say, "Hey, what do you think of this design?" By then the code is already written and people are probably just going to go along with it unless there's some massive objection or something I've missed. But I feel like it would've been a better design if we'd gotten all the people together in the first place and just spent an hour hashing the thing out top to bottom than me just doing it on my own and getting feedback later. JESSICA: The people, but yeah, there's a limit to how many you need in all. JOHN: Certainly. JESSICA: And yeah, five or six is reasonable. But then you need some sort of randomness in the five, six, like some dispersal of interests. AMITAI: Right. If they're all five or six really similar and they're kind of clustered around the same point, then the value is much lower. You also want to go outside that system such as like, "We considered everything we think we should consider, let's go get feedback from people who aren't in this group. Maybe five or six people." It's really a handy number. In fact I noticed now that I'm talking about it, I'm actually using this technique right now without having explicitly thought of it. I started a new little open source project derived from qmail over the last weekend that I'm hoping we'll get a community-based effort off the ground, like say individual developers with forks and I'm trying to get us to collaborate on GitHub and produce authoritative community releases that everybody can update to, which is a new dynamic if it works. And so far, it's working. And one of the things I noticed is we have about five or six people and for each idea that we're debating should be on the first release or not. We also have a handful of ideas about how to implement it. We've discussed what the versioning scheme should be, what the first release should and should not contain. I had my own ideas about all this stuff. I started the project. But there are better ideas now, a couple of days later. And it's this five or six people that did it again. And I didn't even realize I was doing it. I guess that's right. It's clearly correct. And that's how you scale agile. JESSICA: [Laughs] JOHN: Nice. JESSICA: Way back at the superpower day, you said that your superpower was orienting in problem space. What does that mean? AMITAI: I think what's happening now with this new open source project is a really good example of that. The handful of us that are starting to get involved in the last few days, we all have a shallow agreement that the state of things with the last release of qmail is worth investing in, but not okay the way that it is. We have pretty shallow agreement there. We also have a sense that there are many, many, many things about this codebase and its adaptiveness to the current internet environment that are in need of improvement. And then there are many, many, many ones that we can't leave the way they are. We are not in agreement but we are rapidly learning because there are many orderings through those improvements that are going to be very suboptimal. For instance, there is a culture of everybody maintaining their own patch set and patches have to be able to continue to apply for a while until we have enough features that people don't have to do that anymore. And if we make a wrong move now including [inaudible] instead of using some custom stuff that there was no author use, that's invasive that will break patches applying and then we're in trouble. That's a good idea. It shouldn't be this way but later is the right time. And we're negotiating this and I think as we negotiate this, like what should the releases have? How should features be designed? I wrote a Wiki page before the project even started because I was thinking this should provoke a reaction from the people I invite either this is something they're nearly 100% aligned with where we can argue about the details or that doesn't sound one of the projects they want to be part of. But I need to provide something that is Cunningham's law again, either this is wrong and I'm going to argue with you because it's usefully right, or forget about it, I'm going through a different Wikipedia. JESSICA: Usefully wrong? AMITAI: Yeah. Like my vision for the project is right enough to be worth arguing with as opposed to [crosstalk]. REIN: Is this like the 'all models are wrong, some models are useful' thing? AMITAI: Exactly. Like let me spitball something. I'm going to work hard on it because this is actually my vision, but it may or may not be right because we're not a group project yet. And in order for it to be right, we have to wrestle with it as a group, change it so that we all mostly agree with it. And then we are oriented in the same problem space because the things that look like solutions [inaudible]. REIN: And also if all models are wrong and some were useful then we shouldn't be attached to the rightness of our model. What we should be attached to was our ability to learn from it. AMITAI: Yeah. One thing I've noticed that I'm really happy with in the collaboration in this very new project is, I sent some PRs with patches of mine that solve problems that I think we all agree we should have. And my first reaction to seeing them critiqued was, "Oh God, here we go." But then immediately after that, I started smiling. I was like, "No, no, no, that's exactly what I wanted. Please, I hope you have a better patch. Let's take yours." And they're not wrong. They have good comments. JESSICA: When I propose something to a group, I don't want them to agree with everything. That is not optimal. They need to bring their own pieces and attach that, and we need to mold it together. Maybe I've provided a useful structure. But if I get it all right, then it's not teamwork and they won't be on board with it and it won't be theirs. It's not going anywhere. AMITAI: So something I'm explicitly trying to avoid is being the leader of this project. I made sure to invite someone who had a patch that was so unobjectionable that it could be the first commit. And then I made sure that someone else was the person who reviews it first. It's not me, it's them. And so I'm very happy to instigate these discussions and have opinions that are based on something. And then if someday people cannot figure out how this project started. JOHN: Yeah. Like you Jess, I always feel like when I'm having a new design proposal, it feels less good to have less input on it. Like did you not think about it? I can't have gotten this perfect the first time. Let's make this better first before we all agree that this is what we're doing. And yeah, I feel like maybe the pull request is too late in the game for that to happen, and I need to work it earlier into the process in order for that feedback to flow. I'm not sure. AMITAI: What makes me think of is what we're doing as a group with this project is like a slow, inefficient lossy version of mob programming. If we could pick time zones that we agreed on and get on the same call together. JESSICA: Maybe if you all received your paychecks from the same company. AMITAI: For example, yeah, instead of our copious free time. And so we're trying to approximate that in a little [inaudible]. JESSICA: Yay. Thank you. That just made something abstract into a concrete example that we can talk about. AMITAI: And that's what my reflection is. JESSICA: Oh, sweet. Okay, okay. My reflection is about -- Amitai, from the start, as soon as we started talking about your superpower, you talked a lot about questions. Your superpower was to be able to pick the right questions out of all the questions that you may have. And it wasn't tools but constraints that make this possible. And later on, we were talking about TDD and bounded perfectionism and you referred to the test as asking questions of your code. And that when you help people become better at being effective in a business context, you ask them the same questions you asked yourself. So there's this focus on the path to helping and the path to being effective isn't by having the answers, it's by having useful questions. AMITAI: Cool. The way you reflected that back to me is an insight for me. Unfortunately, I already have another one. JESSICA: [Laughs] Sweet. JOHN: Yeah. The bounded perfectionism concept and the test as questions were both really interesting and I'm definitely going to keep thinking about these because like you were saying, you use that as a general life tool like the TDD mindset of let's put a boundary around this and then we can work on what's inside the boundary rather than worrying about the field of infinite possibility. And I find that really fascinating. And as you clarify Jess, about the code asking questions, the test asking questions of the code, that's also a really fascinating way for me to think about it because I still struggle with writing tests. I don't write as many as I would have liked to have written. And I'm wondering if thinking about them as just questions is going to actually change the way I relate to them in the way that they fit into my coding process. I'm really curious to see how switching up that thinking actually affects the way I write them in the future. So, I'm looking forward to that, seeing how that plays out. There's a lot here for me to chew on. Also, Rein's quote about 'knowledge isn't shared, it's constructed' is also going to sit with me for awhile because I really like that as well. REIN: I was thinking about bounded perfectionism and I was thinking about progress. And one of the things that I think a lot about is what are the minimum possible criteria for progress? What are the things that you have to have? What are the minimum necessary and sufficient criteria for progress? And to paraphrase Karl Popper, I think that it's (a) the possibility of success. JESSICA: Bingo! [Laughs] REIN: It is. And (b) the ability to recover from failure. And I would also add (c) from Stafford Beer - the system has to be viable at the time. JESSICA: Double bingo! REIN: So you have to be able to continue to exist. But if you have those three things, you can make progress and everything else is a bonus or an optimization. But if you don't have those things, you fundamentally can't make progress. So before you worry about other things, you need to think about whether your system meets those criteria. AMITAI: I already have an insight. For me, Rein, you had a bunch of jackpot quotes for me today. The one that really, really got me in the fields and the words was the Virginia Satir quote. Could you say that one again? REIN: Yeah. It's make hidden things visible, make abstract things concrete and make implicit things explicit. AMITAI: That's my new motto. REIN: It is very good. Almost every problem can be solved just by understanding the problem. AMITAI: There was another [inaudible] recently was that extremely often you can get a better solution to a problem by figuring out a better framing of the problem. REIN: Russell Ackoff might say that we need less problem solvers and more problem formers. JESSICA: Asking questions. AMITAI: As soon as that happened, I thought like I got to burn that into my skin somewhere. That's amazing. I'm going to have to listen over also because there were a lot of things in here for me. But that one, as soon as I heard that, I'm like, "Okay, that brings a lot that I'm supposed to have at the end of this whole [inaudible]. [Laughter] JESSICA: Awesome. AMITAI: I could go on for a long time about what I'm doing with qmail which I already did, what I'm doing with pkgsrc, which is this cross-platform Unix package manager that is super weird and is super useful, ikiwiki, which is one of the earliest static site generators and it's not only static, it's a really powerful system. I have really weird choices of open source projects and it's just like, I don't say people should run an email server. I don't say people should look at these projects, but they do have a lot to recommend them and I'm happy to talk about them any time. And if people want to invest in learning them, there will be a huge payoff. It's just that the startup cost is high. JESSICA: Okay, awesome. So how should people find more of your work? AMITAI: On Twitter, where I am fairly active. My handle is @schmonz. That is also my personal website - schmonz.com. I have a three-minute podcast, as just mentioned. And you will find an episode [inaudible] at agilein3minut.es. REIN: I have to ask, how do you cue something interesting and useful in three minutes? AMITAI: Big design upfront. [Laughter] AMITAI: Actually, the writing process for me at least is really involved and that's why I haven't been making new episodes lately. It was really typically four to six hours to write the script and that's where all the editing happens. Now I do one take or occasionally two takes and off it goes. But it's all upfront. So that's Agile in 3 Minutes. The government of Spain so far has continued to allow me to own this domain, and my consulting side is latentagility.com. JESSICA: Thank you so much for joining us today. AMITAI: Thank you very much for having me. It was a pleasure. JOHN: Yeah, this has been great. JESSICA: And if you want more of the rest of us, you could join us on the Greater Than Code Slack. We have the Patreon. This is a listener supported podcast almost entirely, and if you donate any amount patreon.com/GreaterThanCode, then you will get an invitation to join our Slack, which is low volume and very friendly. Thank you for joining us and see you next week.