ARTY: Hi everyone. Welcome to Episode 142 of Greater Than Code. I am Arty Starr and I'm here with my friend, Rein Henrichs.

REIN: Hi, Arty. And I'm here with my friend, Jessica Kerr.

JESSICA: Thanks, Rein. And I am proud to introduce our guest today, which is Will Larson. Will Larson has been an Engineering Manager at Digg, at Uber, and at Stripe. And I worked for him at Stripe for a little while so I can vouch that he is a good one. Recently, Will has published a book, An Elegant Puzzle. Will, welcome to the show.

WILL: Thank you so much. I am really, really excited to get to chat with all of you.

JESSICA: We always start with the exciting question. What is your superpower and how did you acquire it?

WILL: When I think about superpowers, I used to think about strengths and weaknesses and the things I'm good at and things I'm bad at. As I've gotten deeper in my career, I've realized there are more superpowers, which are in the best sense, something you're really wonderful at. And in the worst sense, your kind of secret weakness, your kind of arch nemesis exploits to your kind of a plotline demise. And so, I think my two kind of superpowers that I think about are first, I think the ability to take something complicated and define simple ways to think about it that work most of the time but don't always work. And then second, a rigorous love of structure, which almost always works well for org design. It almost always works well for solving problems until it doesn't. And then you cling to it too long in a pretty catastrophic way. So I think those are kind of my two superpowers which are both extremely, extremely important and then also kind of devastating in their kind of misapplication. How did I get them? I think whenever I talk to folks, the number one book I recommend is Thinking in Systems: A Primer by Donella Meadows. Systems thinking has been core to kind of my work for some time. I got to go to a training when I was like 16 or something, which I didn't really understand and I was surrounded by college professors talking about how to get more grant funding, which I really did not understand at all either. But for me, that was really kind of a foundational thinking system that's grown more and more over time, kind of embody the entirety of my approach.

ARTY: It was a class on systems thinking?

WILL: My father who retired a couple years ago was professor of economics and he was starting to investigate systems thinking. And what better place to bring a jaded disaffected 16-year old than a training course on systems thinking in Boston. And in retrospect, it was actually phenomenal and I'm really glad he did. Although I think I was slightly skeptical on the onset whether this was going to be kind of the best use of my weekend.

ARTY: [Laughs] What we want to do and what will make us happy are so different.

WILL: Yeah. But I have no idea what I did for the surrounding 200 weekends around that weekend. But that weekend I actually remember. So in retrospect, I'm quite glad it happened.

REIN: So what did the 16-year old learn about systems theory?

WILL: The first thing I learned, which is now something I'm learning from the opposite direction, is just how complicated kind of UIs are. A lot of the systems thinking software is fairly unintuitive. And most of this two-day course was really focused on teaching folks how to just navigate the basics of the UI. And I think at that point for me, it was very easy to get in and just start using it and learning it. It was very intuitive. These were the days when you're futzing with your Windows 95 INI registry files or something. And you just have this kind of innate sense of wonder about computers working at all. To me, the first thing I learned is just how easy it is I think when you are a "digital native" I guess to kind of figure out some of these UIs, but also how hard it was for folks to actually get to the systems thinking. We have these really powerful ideas in our software that often are interfaces, the usability means we can't even get to. And really to me that was like the thing that I remember the most. But also this idea of just the basics of system thinking, stocks, flows, modeling the changes and not being able to just talk about the outcome you want, but actually being forced to manipulate the system to get to the outcome you want, which is incredible. Much harder way to think, but quite important and useful for me ever since.

REIN: So what kind of software did they use for systems theory? Was it like flow diagram kind of things?

WILL: There were two large companies doing this and Stella is I think the one that has maybe absorbed the other at this point. But yeah, there's this Stella software. If you are in kind of educational system, it's a little bit cheaper, but if you wanted to go out and buy Stella today, it's I think roughly $3,000 to buy a single seat license. So it's quite expensive. And also it feels a lot like it did 15 or 20 years ago. My secret dream, not that secret I suppose, is to actually kind of make this software usable. There's so few practitioners of systems thinking that I don't quite know how you make a business around this that is actually like a life sustaining business.

REIN: Does it cost $3,000 a head because there are so few practitioners? Or are there so few practitioners because it costs $3,000 a head?

WILL: I did an experiment, it's up on GitHub. It's called systems. It's a Python library where you can actually write kind of a DSL, just a text-based format, a little bit like the dot file format if for like this and can actually like run and compile the programs and see the outputs and get the data from it. And so that was an experiment I did. What if you could actually write this and there's no UI at all. Instead of using an old confusing interface, you can just use different old confusing interfaces like vi or Emacs or something to write it out if you so choose, or a normal text editor I suppose. But I do think the costs are definitely part of it where it's hard to get in. But actually there's another, I forget the name of it, but there is a free online systems modeling software that you can just use for free. No software to install. There are solutions out there and it does seem like there's not a whole lot of excitement around it right now. One of my friends, Bobby Powers actually got a Master's Degree in systems thinking, but he ended up leaving the US to get that Master's Degree. There are very few degree programs left kind of in this specialty. So it does seem like it's an idea that has not fully caught on even though I think the tools are remarkably impactful for the folks who do use them.

JESSICA: Are these like abstract systems like little tanks and little boxes and pipes? I'm trying to picture the software and what you would use it for.

WILL: These are indeed the most abstract of clouds and sinks and pipes and boxes, the typical way that you would use this. As a 16 year old, what did I try to use this for? My goal is to model kind of the progression of worms on the internet kind of spreading across systems and kind of an infection model. And then the interesting thing about infection is that there is immunities that because you have an initial kind of spread, you develop a patch, the patch get applies to a certain amount and that starts to slow the spread. But does it actually work enough, et cetera. Just some examples of where I've kind of used this approach in the last year, hiring funnels. I think often you're trying to grow quickly but how much can you actually hire and what is your actual constraint particularly when you start looking at the amount of interviewer load and are you actually able to get anything done or are you too busy interviewing? And then how does improving your training programs drive your ability to hire faster? And so kind of understanding that problem. Typically, the output of when you run one of these models is actually like a line graph that shows kind of the different values over time. And then you can experiment tuning the different flows and see kind of the different results. A little bit low tech, surprisingly insightful if you play around with them for awhile. And it's really the changing the model and learning where the constraints are is like really the value, not the line charts specifically that it's a composition of lies you've told to simplify reality that it's not true, but it is like a useful model that helps kind of refine your thinking.

REIN: This is the 'all models are wrong, some models are useful' theory.

WILL: Yeah, absolutely. At least these models are not in an Excel spreadsheet or you can actually verify them more easily than you can in Excel, which can be so complex that it's actually hard to know what the model is underneath it sometimes.

REIN: How do you know where to stop? Because when I have done sort of causal diagrams or flow diagrams, you can pick anything and you can keep going because complex systems are full of interactions. Let's say it's developer productivity. You want to understand what the constraints are on your developers' productivity. You can get all the way into the weeds, you can get all the way into like how the sales funnel works because 10 steps away, that impacts something that impacts something. How do you know when to stop modeling?

WILL: I think when you run out of improvements to make. When I think about modeling or I think about architecture, any sort of improvement and program, you can pretty easily outpace your ability to absorb change. And so, I think the goal of any good model or any project you're running at all is to get as much planning, as much learning done until you can no longer absorb more and then focus on executing that kind of implementation. So, the real answer is it's never enough, you're never done. But you might pause at a given moment to kind of focus on integrating the learnings, and then that will go back into improving the model again. But the models themselves are not the value. It's the conversation between the model, the recommendations of the model and the reality of using those recommendations, that's the value. The model itself is just like a tool to support the process.

REIN: You mentioned the rate at which a system can absorb change. That's the sort of meta level metric for the system that I've noticed a lot of teams don't really think about. They just try to find the right changes and then execute as many of them as they can. And I've noticed that this isn't always successful. So, how do you figure out what your team's rate of change is, how quickly it can absorb changes?

WILL: There are a few different types of changes you can think about. And roughly, I put that as organizational changes, there are process changes, and there are changes to the software systems that you're managing. I think with software and systems, this one's probably the easiest. What you can do here is look is it Kanban, but some mechanism to limit work in progress and to hold yourself honest. I think a lot of times you get to 90% and you're like, "Oh, we completed it," and you kind of pull it off the Kanban board because it's really annoying to leave it on and to finish. But really being honest with yourself about kind of the completion of work and not adding more work until you finish. I think this is the area I think any engineering leader has the most experience is managing kind of rate of change in their software and their systems. For process, I think adoption is the the best one. I worked with an engineering leader, I don't know, like eight or nine years ago. His theory was only do one change at a time until it sticks. I think it's a pretty good rule of thumb. It's not perfect, but who hasn't seen a company roll out agile and like, "We're done. We're agile now." And then you kind of say that you've completed it and you move on, but you actually are still doing exactly what you were before. I think the desire to believe you've been successful can outweigh any desire to measure impact or success for a lot of process change. So at a minimum, being I think slow and thoughtful and measuring adoption, somehow I think is a step above the majority. Then for organizational change, I'm pretty conservative here. I think you'll get different folks who think very differently about this. To me, like the most powerful thing in the world is gel team that knows how to work with each other well, is affected with each other, that complementary strengths, a lot of respect, they can communicate quickly and well. And to me, disrupting those is so damaging to your productivity in kind of an invisible way that I really try to minimize change at the team level. Once you think about how you kind of have managers of managers kind of connected into other managers of managers of managers, I think that's less disruptive, to be honest. Typically the folks are so experienced and those teams tend to do less work directly together. So I think change at that level in an organization is typically a lot easier to absorb. You can do like a little bit more of it.

REIN: Would you all be mad at me if I name drop Virginia Satir right now?

JESSICA: It wouldn't be Greater Than Code if you didn't.

REIN: Cool. Because I just did it. Virginia Satir has a change model that I think you might find really interesting if you're not familiar with it, especially because she was a systems theorist. Her change model is that you start with the current status quo and then there's some foreign element that introduces the need for a change. This could be the things that were working for you but aren't working anymore or new customer requirements or a change to the team, like someone leaves the team and so on. And there's something you have to do to change to meet this challenge. And so the change happens and then after the change happens, there's a period of chaos. And what she means by chaos is that there's a distortion of cause and effect relationships that you're used to. So your expectations that you had for how things happen on the team, the rules that you use to work together start to apply less and you start to be confused about what's happening, how to act in this new system. And so that can make it difficult for the change to stick. But as you get through this, there's a period of integration. And after this period of integration, there's a new status quo that's hopefully better than the previous status quo. And when she talks about what the optimal rate for change is, it's not during the period of chaos and it's not during the period of integration but it's before the status quo ante has a real chance to sort of solidify and calcify and be resistant to future change.

WILL: The interesting thing about that model is that it means that if you're too conservative and continuing to make changes that you actually get significantly worse at it, which I think is a known [crosstalk] but kind of surprising to think about.

REIN: Yeah, there's a Goldilocks zone and you can do worse on either side of it.

JESSICA: Oh yeah, because you want to keep emotion, you want dynamic friction not static friction.

REIN: Yeah, exactly. You want the feeling that you're changing and that each change is helpful and beneficial and then you can look forward to the next change. You don't want the feeling that change is this alien thing that you have to fight against.

JESSICA: And yet, there's a recognition that adoption is never perfect. There is no done.

WILL: Something I think about occasionally is it's like tug and pull between kind of product thinking and kind of infrastructure thinking or kind of like an infrastructure thinking, I think sometimes the work is done, the system is correct, it is meeting as always under certain growth like those escalates will change again. But it is interesting where I always kind of think there actually is done, there is completeness I think on the infrastructure side or really any side where you're maintaining long lived systems versus kind of adding functionality or components distinct from the kind of long term ownership of the system.

JESSICA: An infrastructure, I feel like it could always be better. It could always be smoother. We can always have better visibility. And then there's always the next new infrastructure because the platforms that we have access to change. And it does feel like your infrastructure could always be better or newer or cheaper or something.

ARTY: I feel like we're talking about a system of constraints but along two different dimensions that have somewhat different characteristics. So if I think about infrastructure, I've got a system of capabilities that at any given moment in time, if I freeze time, I've got a set of capabilities. And the set of capabilities even in that moment of time has certain constraints of what it can't support and what it can. It has certain service level agreements to the users with that system. And so there's a set of dynamics that exist outside of time. And then there's a set of dynamics and constraints that exist in resistance to time. So if I think about we've got some new product, some reason for poles, something we're trying to build, an intention we're trying to move toward and our system of capabilities is insufficient to meet the demands of our insatiable wanting, then there's this other kind of pressure. There's this other kind of capacity to absorb change, capacity to take all these inputs and be able to sort of absorb that into our system that we have to pick. We can only focus on so many things. We can only absorb so much of that change. And so I feel like given we've got this infinite set of possibilities for what we can do, what we can choose, and a limited capacity to absorb changes, then one of the most important things we can do is meta up on that problem and go, "Okay, how do we identify the right problem to solve? How do we invest our cycles such that where we're focusing our attention is going to have the best benefit to us?" And if we kind of backtrack that all the way down to Virginia Satir because every Greater Than Code episode should somehow go back to that of like what matters in life? What do we really want to optimize for? What do we care about as humans? We're creating this game that we play during life. We've got these constraints. We're amused as humans by solving puzzles because it's fun. So it's like, I think if we think about the system of constraints from the ground up, we erase the old thing and we go, "Okay, what are the little sinks and tanks and how do we want the system to go?" And we look at all of our different capabilities that we have as kind of one of those infrastructure systems in a moment in time. We've got a set of capabilities as people that we can model the constraints of a certain way. And then we've got this dimension shift of time and what we can be, what we want. How do we build like a world of awesome with our set of capabilities? And I feel like we need to start thinking about the problems that we're solving, our organizations, our businesses as a way to look at the constraints to dream this vision of what we want together. Enlist people in this creating of that dream.

WILL: I really love what you just described, particularly this idea of focusing attention. I think if you read Flow by Mihaly, there's this idea of we have a limited amount of time and attention and energy in our life and where do you want to focus it. And thinking a little bit about what you described, something that occurred to me is this idea of top down architecture where you tell people you have to build on these primitives, you cannot deviate. And then there's this other, I wouldn't call it bottom up, but more organic architecture where people are able to move both the primitives. You can actually ask for new primitives and you can actually build on the existing ones. There's a bit more of you can explore the solution space a little bit more fully and how that allows us to invest more energy and more attention when we have this top down rigidity that actually really focuses our attention very clearly in the product space. But then sometimes in ways where the actual outcomes are kind of quite worse because we're abusing the wrong tools and the interplay there is kind of interesting to think about.

REIN: As a systems thinker and as a manager, you're both an observer of the system that you're trying to modify, but you're also a part of the system. So how do you as an observer participant deal with the fact that you're a part of the system you're trying to change and anything that you do that impacts the system also impacts you in turn, and so on?

WILL: My third superpower is this kind of quest for justice in the everyday process of our professional lives, which is a true superpower in a sense of accomplishing really important things and also really gumming up the works at inopportune times in terms of this pursuit of consistency that is sometimes hard to obtain in the pursuit of maximizing shareholder value or whatever it is that we do on a daily basis here. A couple of thoughts. I think the first is it's pretty easy to take the privilege and kind of design something that you abscond from. I think one of the most important things for folks as they get more senior in their career to think about is not designing systems for other people. Designing systems that they themselves are willing to be bound by. And I think that's the minimum obligation as the manager system designer is to design the system that they're willing to be bound by and willing to participate in.

JESSICA: Yeah, that makes sense. Because this is saying that unlike in the hard sciences where we try to make the system separate from us so we can observe it from the outside that if you're making a system for people, it's in a sense unethical to try to stand outside.

WILL: Unethical is the sort of word I like to use but I find that people don't like it when you use it sometimes because it implies that they're unethical if it's usually like a bad start to the conversation. A different way to frame it is this is kind of like user acceptance testing where you as a -- it's easy to be like Jess, as a designer of the systems thinks we should have unlimited PTO and kind of unlimited budget for travel. And then it's a different thing for like Jess as kind of like the responsible party who's actually getting questions from finance about the budget every month and has to defend that. And then I think it's just putting yourself in the system forces you to actually think about the constraints of it in a powerful way where if you aren't impacted by it, you don't care. I think on-call is another great example where it's like, "Do we really need to do this?"

REIN: I was literally just about to mention that. One of the things that is really helpful is getting a manager to experience the pain that they're causing for the people on call because they didn't realize what it actually means for their lives.

WILL: Yeah. In my last job, I was the third tier on call for every alert at the company for awhile, which was pretty traumatic since this is a company where people's phones will run out of charge when they were on call because they got alerted so often. But it was a powerful feedback loop.

REIN: Gordon Pask might that when you're an observer there's a part of the system, one of the things you have to be able to do is update your own models and your own understanding of the context to account for the changes in the system that you are causing and that are being caused on you and so on. So it's not enough to have a static model, but your model itself has to be updated over time.

WILL: I think that's right. Something I think about is, particularly in a fast growing company, as a leader, you're trying to extract yourself from the system at all points. So I think this problem might be less dangerous in a fast growing company where your job as a manager is no longer be necessary for any of the function of what you're doing today. So I think that can be helpful. Conversely, I also think when you're not doing this well, the amount of work piling on you tends to be a very clear indicator that you're not doing it well. One of the things I love about growth is that problems become extremely apparent and you can't hide from them because they get so bad so quickly as a function of growth hiring, business growth. And so it's actually, I think a really powerful feedback [inaudible] general to have problems get bad so quickly.

JESSICA: On that thread, I was thinking about something you said earlier about limiting [inaudible] 90% and you get a task to 90% and pull it off the board and having to really be honest with yourself and this desire to believe that you've been successful. I'm wondering as you get into this tension of all of these problems coming to the surface and people being in this mode of really, really wanting to believe in their success and pull things off the board, what are some of the patterns of dysfunction that you've specifically seen and been a part of?

WILL: One of my jokes that no one thinks is very funny, but I keep trying to tell anyway is that executives never missed their goals. They just redefine the goal so they hit. There's a certain level of seniority where you kind of can never be held accountable because you yourself are the accountability function. And if you're willing to kind of reset your target in a way where you always hit it, who's going to stop you? Showing value where you have a new senior leader come into a company and they've been brought in to fix something. And so they have to show value as quickly as possible so people can see their impact. And if you read The First 90 Days, that's a pretty solid book in my opinion about going slow, observing before you try to change. But I think this tension to be a leader and this hero model of how organizations change, which is it's not the combination of our systems and our people. It's instead these heroic individuals come in and shape this organization. It's like the great man theory of history which has sort of been out of style for some decades in history, but I think is still kind of in style in tech as the most causal way to understand how change works, and consequently the most comforting kind of model. Because it implies you have any ability to fix problems quickly when in reality these problems are pretty slow to address. So that's the first one that I see the most often is where folks just kind of decide they're done.

The second is I think folks often, no, they're not done, but there's a new priority that's so important they can't finish. And they allow work to get pulled off. And this is tricky because there's I think two different dimensions. First, each time you do this, you leave some technical debt which slows down your future work. You're actually mortgaging the future for the present. But second, there's external work you can't not do. I think that sometimes as a rigid thinker, I want to say we should never violate this rule. But there are some times, I think for Stripe as a good example, is like the strong Consumer Act in Europe where there are regulations coming out in Europe about certain types of payments and you can't not be compliant to them. You just must do it. And so that's something that we weren't two years ago thinking like this is our most important priority. But as we started to understand that regulation, it was just clear like this is table stakes and really important. Something we have to do. I think GDPR is another example on the California privacy regulations coming out or another example. So I think figuring out the balance where you at least can track the extent that you've mortgaged the future and not kind of silently accrete the debt is at a minimum a good starting point.

REIN: The heroic individual myth is interesting because it's the other side of the coin to the scapegoat. The scapegoat is the person you pile all of the blame on and the heroic individual is the one you pile all of the credit on. And neither of those things are generally accurate. But if you are the sort of culture that will engage in heroic individual myth building, you are also likely to engage in scapegoating.

WILL: That's interesting.

REIN: Because it's really about an externalization of cause and effect. It's not the system, it is the person. Whether it's good or bad, we attribute things to an individual.

WILL: Personally, I've seen a bit of both. I think company by company, I've seen kind of different versions of this. Most of the companies I've been at have been sufficiently unwilling to hold folks accountable for their work. I actually haven't seen scapegoating happen much. I can't think of one place I've worked in my career where that happens or effectively everywhere I've worked through my career, I've seen the hero myth kind of come into play at least to some extent.

REIN: I think the thing that works against it is that scapegoating is generally seen as a bad thing and crediting people is generally seen as a good thing.

WILL: I mean it's such a powerful myth because otherwise you actually have to understand how these human systems of hundreds or thousands of people work, which is quite intimidating because the things you actually need to do are adjusting your sprint schedule or changing how planning works to give enough time or making it cheaper to say no because you're spending all your time saying no to things. Or even though you're able to de-prioritize work, the costs of doing is higher than doing the work. And so, everything gets prioritized. It's a lot easier to say so and so isn't the right leader and we need to pull in shinier leader number seven or whatnot to kind of fix the problem.

REIN: This also reminds me of Jerry Weinberg has a set of cultural patterns for organizations. The first one is a variable which is generally characterized by the belief in the sort of rockstar developer myth. If we can just hire the best people and let them do the work, then the work will get done. It's called variable because the results are generally as variable as human performances. People get sick, people have bad days, maybe you were incorrect in your assessment of their abilities and so on. And the next step or the next pattern that many organizations move into because it's explicitly not a maturity model is routine, which is characterized by the rockstar manager myth where if we just hire the right manager and they make everyone do what they're supposed to do, then we can succeed.

WILL: Sometimes it's easy to come across in these conversations kind of like the state of management is doomed and companies are doomed and the agrarian culture is the only one that will ever work for us. And I think that's definitely not how I believe. What I really believe here is that through consequent thinking where we look at the behaviors we see around us and think about how to solve them, we can make well-run healthy organizations. And I think my [inaudible] is that we should just do a lot more of that, looking at how things are actually working to your point, looking at that variable culture and trying to find a way to understand where the productivity is truly happening. And the costs, the underbelly of the 10X engineer myth is that these folks create so much complexity for other folks that everyone else is not able to function and they kind of suck the oxygen out of the room. Where they're able to do an incredible amount of work, but no one else is able to do anything which then confirms the myth itself. By being thoughtful and intentional, we actually can have well-functioning effective orgs. And I think just that intention is where I love to see people spending more and more time. I really do believe as an industry we're getting better at this but a little bit more of a statement of faith than a statement of fact, that I like it to be.

ARTY: There's other factors that affect things too of whether things trend toward better or not. We've got generational effects as one thing that you see eras of culture shift across generations. One of the things that seems to come up as a theme is the initial open source movement started as this community collaborative, let's work together and lower the cost for us all kind of sentiment. And we as a community will own the things. And then generationally we ended up with, there's all this software out there for free. And in order to be able to stand at the same level as everyone else in the industry, we have to leverage all this free software that's out there. And then we build on top of this mountain where 90% of our software is third party parts. And in order to compete, there's this pressure to shift further and further toward this space of not knowing what we're building on top of it. And there's this momentum in the industry that drives us this direction that creates very real costs of say, everyone in the industry in general shifting to some new technology and leaving an old one behind. And the community around that thing dies and you're dependent on Googling your way through it as a way to do development. And once you ended up in this land of unsupported, it's not really a real option. And so there are shifts in the industry, there's ways that we move as a whole that creates this gravity, that creates this pressure that's driven by the nature of momentum of the system as a whole, that we get caught up in ways that I think the system as a whole isn't necessarily moving toward the best interest of the people inside of it, because the decisions of our past feed these dynamics of constraints. And so it's kind of like this old crufty software now that we're like living inside this bubble that becomes our constraint. And so, I've been thinking a lot about how to genuinely change that. And I started listening to this podcast called The Portal. And one of the key lines of thinking is taking a step back in a systems thinking, look at our socioeconomic system infrastructure, thinking about all this stuff as software. Thinking about all this with our systems thinking skills and go, "Okay, what have we learned about all these things?" And to not stop dreaming, to not give up as we're just all helpless to the machine. And instead go, "Okay, we've got a whole lot of really smart people." And at least in the moment, we're doing pretty okay. So how do we work together and put our heads together to put our maps together of all the things we see together so we can create the opportunity for a better future, for a world that we actually want to live in. I see systems thinking, I see the possibility of modeling constraints in more mathematical, geometrical, Pythagorean sort of terms that we can create sort of universal language to share ideas with systems thinking, to share our models and share our constraints, to be willing to live inside those systems, experience what they're like and work out the kinks so that we've got tools that are good for making decisions. And I see this process of creating shared language, the types of stuff you're doing as really the core to the things that have the capability to bring humans back together and come up with answers.

WILL: There are probably like 15 things I want to say thinking about that. The very first one is, I don't know if you've all read Vernor Vinge's Deepness in the Sky. But it has this idea of code archeologists in the distant future where no one writes software anymore. They just kludge together existing things they don't really understand. I do think at times what you're describing there starts to feel more and more like this future of code archaeology where we're not really sure what happens or how it works, but we're cobbling.

REIN: The Warhammer 40K Universe is also similar for those of you who are fans of the grimdark.

WILL: The other thing that I would want to go into, and there's again probably like 20 things, at least it's going to grow each time. But I think so often when we think about these systemic problems, particularly when you go beyond your team or your company, they get pretty overwhelming to contemplate. A simple one a lot of companies might deal with is, we didn't hire a very diverse start and we have a non diverse company from a certain kind of dimensions. How do we start fixing that? And I think even something like that, which can feel overwhelming to folks when they start thinking how to solve it, it's too late. But then you think about something like how do we try to deal with the consequences of generational wealth created by mortgage programs in America? And you're like, "Look at your 200 person company." And that seems hard when you think about something like that and it's all of a sudden insurmountable. And the most important thing I found when thinking about these problems is try to understand the big pieces but also do something. I think a lot of times folks use the overwhelmingness of the problem to shield them from accountability, responsibility of doing something. And so, I found that each one of these overwhelming problems, finding any way I can make progress and doing it. It's not obvious initially how that little thing will contribute to the larger problem, but just not taking the excuse. Or another variant of this is I'm going to keep sharpening my tools indefinitely because sooner or later I'm going to use them. But I'm just going to keep sharpening because it's quite uncomfortable to take the actual next step. And so not getting caught up into the enormity, but just taking real daily action towards progress, I found to be the most important thing that folks often try to avoid doing. You know you got to do it, but it's easy to convince yourself not to, but it's such a powerful way to make real progress even in these large complex, confusing systems.

JESSICA: That gets to the thing we talked about earlier about getting in the habit of having change and not holding still.

REIN: One of the things, the resilience and safety communities talk about a lot is being poised to adapt or having adaptive capacity, which is preparing yourself to be able to deal with uncertain and stressful situations. You don't know what form the crisis will take, but you can be pretty sure that they will arise. So you don't know how to solve the crisis, but you'd know how to put yourself in a better position to be able to find solutions.

WILL: Something I think that ties into that is this idea of when you talk to someone who's been in tech for like six years, seven years, often they have this idea that they are almost done with their career.

REIN: I have learned all of the things.

WILL: Well, there's two different dimensions, right? I think that one, there's this kind of senior engineer after two years phenomenon. The actual career path is not well-defined for folks. And I think a lot of companies don't know how to leverage the skills that people who have 15, 20 years in. A second dimension is that the industry is extremely difficult environment for a lot of folks. And I think a lot of folks don't think that they've learned everything. Instead they have decided to opt out of this lived experience that is overwhelming and draining for them. I think as we go onto this other side, we still don't have enough folks who have actually been in the industry working in these environments for like 20, 30 years to see the full consequences of a lot of our decisions. A co-worker of mine worked at Nintendo. Nintendo is I like 100+ year old company, I believe. I don't remember what they did initially, but it was not on video games, I guess. And the conservatism that that brings is I think really fascinating to talk to him about and just thinking about how would you make decisions differently if this was the 20th time you were picking a programming language versus the second or something like that.

REIN: Our entire field is very new compared to how long many things have been around.

WILL: And consequently I think we rarely see the results of our decisions, and getting to the points in our careers where I think moving jobs is really powerful because you get to see different strategies for different problems. But there's also staying somewhere longer once or twice and seeing all of the consequences of your brilliant ideas as they kind of age out of solving the constraints is equally humbling and been like really, really powerful for me as well.

REIN: If you can't engage in the feedback loop, you can't really begin to learn. So if the feedback cycle is so long but you're at a different company, by the time the chance comes around to learn, you'll never really engage in that loop.

ARTY: And so many of our decisions in software that are like that. You think about the everyday decisions we make around refactoring for the long term or optimizing for this long game and often never really find out if the decisions we're making are making things better or worse, really. And there's this side effect of it's easier to work with because it's more familiar because we just rewrote it. So you've got this familiarity bias that is extremely strong which is always going to kind of anchor us to think that we're doing things the right way to reinforce whatever our beliefs are. And granted, it doesn't mean that our wisdom that we've learned over our careers isn't relevant or real. It's just that there's so much likeliness for echo chamber effects to sort of pollute that as well, where we just go, "Hey, I just confirmed my own beliefs, isn't this great?"

REIN: I think another one of the challenges is just that the rate of change is so high, the technological development. Even in the history of humanity, we're in the far right hand of the hockey stick. For thousands of years, people rode horses and then it took a hundred years to go from cars to planes. And the same thing is happening in the last 60 years in our industry. If there had already been five paradigm shifts in the five-years that takes for you to learn about something, how can you learn about anything?

WILL: And I think part of that that I think is interesting is that it's now become conflated that career development is learning new things. And there's a power to that. But if you talk to folks at smaller companies, they're often like, "We've picked these exciting new technologies so we can hire the most brilliant new minds who want to use the newest things." But it sort of means that we have channeled some of, by some perspective, most talented up and coming folks into things that will be obsolete in very near future. And we're almost making the folks earlier in their careers obsolete by how we are focusing their attention. So we're almost preventing competition in a certain way, but one that also devalues our own experience as well. So I don't know if it's working well for any of us.

REIN: Yeah. Stafford Beer has a model for this that I really like in the context of how companies keep up to date with changes in technology and so on, which is they have to keep making bets to keep their hockey stick moving up. As the technology they're on now or the decision they've made now starts to trail off, they have to make a bet on a new one that they think will keep pace with innovation. And they have to keep doing that. And as the rate of innovation increases, if you do that more and more often, then the bets become more and more risky.

WILL: Something I've been thinking a lot about in that context is the types of bets that are cheap versus types of bets that are expensive. A project we did at Stripe is Sorbet, which is adding this Ruby typer. So static typing to Ruby. And at first I was like, "Oh, this seems super risky. What if it fails?" What would happen is there would be a lot of comments that didn't do anything in the code, like identifying these types. But actually there'd be no consequence. We just would've had that fixed amount of investment into the project. But the actual consequence of failing that type of project, that enhancement of the existing stack basically zero. Some opportunity costs. And then others like moving to another language, the cost of failing that is two live living things in which things are cheaper or expensive.

REIN: I think maybe the sort of most obvious one to point to you is microservice. There's a bunch of companies that are placing a bet that microservices are the way to keep pace with growth. And that this innovation is the thing that's going to enable them to keep that hockey stick going. And there are some sort of short term results that are promising, but we don't really know how it's going to pan out yet.

WILL: I think that that one's particularly topical, and actually a place where my thinking has changed over time, which is the sign of, I think, personal growth. But I'm not sure. I can't really attest to it. But I think particularly something that I've learned in the last three years is the power of having shared repositories. Even if you have microservices, but just having a shared repository and be able to modify all of them in a single pull request. And there's all these kinds of variants of like, it's not in some ways just microservices versus monoservice or monolith or whatnot. There's like all these different combinations that kind of changed the risk profile and I think really important ways that are not kind of obvious initially when you kind of think about it without having seen the playbook run out three or four times in different ways.

ARTY: The biggest thing I've seen happening with the shift to microservices land is the knowledge for how the system works get separated into the silos such that when you walk around and try and figure out how the system works, nobody actually knows. And various people have simplified models of their interaction with the world that aren't quite right then end up stale with respect to the actual reality that can shift beneath their feet. And they don't really understand the characteristics of these other systems well enough to be able to reason about this hole. At the same time, we've got this influx of complexities that has to do with all the interaction factors and we've got interaction factors from these distributed parts exploding in terms of complexity at the same time as we're reducing our integration knowledge. And in theory, we should be decoupling these things. And so we shouldn't have to know how the whole system works anymore. But I'm not finding that theory to play out too well in practice. And there's a significant risk to giving up our integration knowledge. And I feel like what you're saying with respect to having a shared repository, modifying all the modules in the same pull request, it's a way to create a structure that matches an idea of principle that all of these things ought to integrate in some way together, and we ought to know what that is, it ought to have a meaning of correctness as a whole.

WILL: This is a topic I've been thinking about a lot recently and I've kind of developed a vocabulary because the first step to solving any problem is to develop a vocabulary to describe it that no one else can remember and then you stop using probably. I'm thinking a lot about systems having properties then also behaviors in the distinction here being property is something you can actually statically analyze. A behavior or something you can verify happens through alerting, through integration tests, through fault injection. But you really can't, it's not something you designed in necessarily. And so, when I think about service boundaries, I think a lot of the times these are the boundaries we're going to actually have properties. Property could be all write requests in a service are transactional neither succeed or fail atomically. That could be like one property. But once you start thinking about a mesh of services, there's no way you can assert anything about the transactions across these different boundaries. It's sort of impossible to any sort of meaningful extent. But you can have behaviors that you can actually assert or monitor that they roll back correctly across all of them where there's no hanging state during the failed request or something like that. And I think figuring out these tools to actually make these larger systems comprehensible is quite hard. And when I think about the move from the monolith to the microservice or the services, I think one of the big challenges as you move from being able to [inaudible] properties, many of which you can actually analyze in a static fashion through these behaviors, which you can only reason about. But often you don't even have the instrumentation to verify that the behaviors are consistent over time. And I think we're just missing and that's me where fault injection is kind of the way of the future here is simply giving us tools to verify the behaviors in these large systems.

REIN: The other thing that happens is that achieving those properties in distributed systems becomes incredibly expensive. So, achieving autonomicity when you have a database is cheap, you just use the database. In a distributed system, you have to like know how raft works. So it's expensive both in terms of the expertise required and in terms of the network costs and so on. You can get those guarantees but it is very expensive. And actually most of the things we do in distributed systems are impossible. We just almost do them and pay a lot to almost do them.

WILL: And this is one of the things when we talk about technology changing, this is one of the really interesting things is we're actually in a world where things like spanner proport provide the abstraction you so desperately want in terms of strongly consistent, highly available setups in certain ways. And I think this is one of the ways it's exciting to be alive in designing software right now is some of these things that were, to your point, impossible and are still impossible, we've created abstractions that make them appear to be possible, which is kind of interesting to see.

REIN: They're impossible and we do them anyway because we're happy with five nines of the property we're trying to acquire even if we can't get to 100%.

ARTY: Software is so cool. I mean, just as a tool, how amazing is it that we can take these really abstract ideas of a property that we want and go, "I'm going to build something that simulates this set of properties, creates an abstraction that someone else can then use as a building block part to build something cool that they want to have. And we have this chain reaction of all these ideas that we weave together and build sculptures out of them. It's so cool.

REIN: Isn't it such a cool place to apply systems theory?

WILL: I feel like I need your reading list after this to go back and study and make sure I understand all the references.

JESSICA: Will, do you think as you study systems thinking, I mean you're talking about it in respect to people who write software. Does it also apply to the software itself? Did the learnings cross?

WILL: I have a number of systems thinking related dreams that are extremely unlikely to kind of translate into reality at some point. But I really think a monitoring system that worked on you modeled your production behavior and then you tracked it and you identified the inconsistencies between the model, the expectation of throughput load latency and the interactions, the relationships between these and the actual would be an incredibly powerful way to think about this and a really powerful tool that you could use to have a much richer approach to monitoring your system working how you think it works. All of a sudden you're alerting on your mental model being correct, which is kind of an amazing thing to think about. I do think there's lots of aspects where if we model here, I think this goes back to the TLA+. If you've ever used a tooling to actually run the model checker, it is quite bad. It is not a good tool to use. It doesn't have the usability. It is cross platform software which has the lowest common denominator of the interface. But the actual capabilities it brings are just phenomenal and kind of extraordinary. We have the tools, we have the technology, we just haven't really as an industry decided to get serious about using them. And so I think that systems thinking applies tremendously, just like model checking applies. And there's all these things that we know to do that we just don't prioritize in figuring out how we can do more of them. Particularly the distributed work and the infrastructure work, the foundational work, I don't know if you need a TLA+ model for your login system. Although saying that, maybe you do. Maybe getting login rights is actually one of the most important things from security perspective you could ever do. So I think we just don't have familiarity with the tools, but I really think they apply, they're useful and I'd love to see more of them.

REIN: I feel exactly the same way about a dependently typed programming. It would be so cool if it was a thing that people could do in general in the industry, but the usability isn't there. And that's one of the main reasons I think we're not.

WILL: I was talking to someone about literate programming a few days ago where there's all these ideas that are kind of known good. They just aren't operationalized into actual interfaces to tools that we can use. Or they are operationalized but they're operationalized as two major modes in Emacs that you have to use multi-mode major mode to use or something and you're like, "This is not good." But yeah, why these tools aren't turning into industry products is I think a really interesting thing to kind of think about and think about how we can solve, kind of to our earlier discussion.

REIN: You say that as I literally just wrote an architectural proposal in org mode so that I can make like sequence diagrams in line in code.

WILL: I literally was talking to someone about multi-mode last week for literate programming, which seemingly actually would work pretty well. You can have like a regex with the transition [crosstalk] mode. So you could actually do literate programming really well this way. It's just sort of [crosstalk].

REIN: There's a new [inaudible] now called polymode, by the way.

WILL: I look forward to not knowing what that is.

REIN: It's basically like multi-mode but it works better and it's also called polymode, which I'm into.

WILL: I don't know if you've used Visual Studio Code, but it's actually quite good. The thing that I found really powerful as they've taken the extensibility and turned it into one command you can use to type commands and run, and it's like a usable version of a really powerful and free editor. So I've been spending more time there and learning to not be so sad about not having Emacs.

REIN: I really like Visual Studio Code mostly because I can't get Emacs to do things with JavaScript that make it so I don't hate developing JavaScript and Emacs.

WILL: For me personally, the reason my personal projects are rarely in a statically typed language is that I want to just go bang them out quickly and I find the tooling overhead for a Rust, for a Java, for a C++ is high enough that you end up in an IDE and that prevents me from doing more of it, which I obviously should.

REIN: By the way, welcome to editor chat, everyone.

WILL: Unavoidable.

JESSICA: Tools are part of the system. They're in a tracking part of the system because they're easier to target and change the dynamics.

REIN: And if you think about it from a sort of representation theory standpoint, look at how much power these tools give us to modify the systems that we can only see sort of dimly through the lens of our editors.

ARTY: I've been thinking about properties and behaviors and these missing tools to verify behaviors. And if I think about software and what gets hard, like what is the ultimate constraint to solve for? And effectively, it's our human understanding. And so as the systems get increasingly more complicated, we need some way to make the behaviors observable so that we can run experiments to get an intuitive understanding about how all these things work together and then start learning what kinds of information work for us as humans. How do we increase our own throughput with being able to take a ball of confusion and reach a sense of understanding of how something works. As I was listening to this talk about observing what's going on, my research is in a data driven approach to learning an improvement by measuring what I'm calling idea flow or the flow of ideas between the developer and the software or the flow of ideas between people and measuring the friction with respect to that flow. And friction, I'm measuring as the frequency and duration of our WTF moments. So like the moment you run into some unexpected and you're like, "That wasn't supposed to happen," until the moment we get that thing resolved. And studying the patterns in those troubleshooting sessions as sort of the gold mine of experiences that will help you figure out how to improve, will help you figure out how to design the information to optimize for the humans. And when you talk about getting serious about doing science together, doing systems thinking together and really taking our industry the next level, I think that's the new frontier really in our industry is exploring that space.

WILL: Are there any tools that you've seen that are demonstrating this? Like this line of thinking?

ARTY: I haven't necessarily found a whole lot, at least in the space of software. There's some emerging in the DevOps space. But I quit my day job and said I'm going to be an entrepreneur. I'm going to go build [tools]. That's how I solved this problem. I think it's very much a need. I spent a good bit of time figuring out how to make this a palatable thing for everyday usage. How do you drop the difficulty level such that we can kind of ease into it? And it's kind of like learning TDD. It's like a new habit you have to pick up on and it's hard. So how do you make all that easier? And so I've been focusing with one team and working on solving that problem. And so now, we've got these tools that are like super slick and easy to use and I'm still pretty MVP, but I've had the models for how do you measure and identify these risk factors for a while. But this other problem of, "Okay, so now we know what to measure. How do we actually overcome the barrier of gathering that type of information and being aware of it while we're working and all the influx of ideas around what we ought to change that we're not ready to absorb yet?" There's this whole class of new discoveries that come with a paradigm shift, but we got to be ready for it. And there's definitely people that are ready, I think, for that next level of growth and maturity in our industry. It's going to start with a small group and expand from there. That's how these things go, right?

REIN: I keep coming back to the importance of making these models explicit, and a number of the things we've talked about touched on that. We started talking about Kanban and work in progress limits. And one of the things that Kanban does is it makes the flow of the work through the system visible so that you can actually see how much work is in progress. And when we talk about sharing these system models and communicating and talking about building a shared vocabulary, it's so that we can communicate our models to each other, isn't it?

WILL: One of my finest pieces of thought leadership is this idea that there is an infection theory or sorry, there's herd immunity for information propagation where you can never actually get people to know anything in a large group. It's just impossible to get a hundred people to know the same information. But you can have enough people scattered around that know the information you need so that the information, when people need it, they only have to go one or two hops before this kind of person who's immune to not knowing this information can prevent the propagation from kind of proceeding further. So, I personally am sort of bearish on the entire idea of communicating things uniformly and widely. But I do think to your point, in small groups, you can highly communicate, highly align. And I think being able to communicate your models and those groups is remarkably powerful, figuring out how to do that well. And I think the systems diagrams, like one strategy to try to do it. If you ever read Good Strategy Bad Strategy, I think their definition of what a strategy document is is really effective. That's kind of a useful document for sharing mental models. That's honestly been the best one I found in probably the last couple of years.

REIN: Yeah, that's really interesting. If you look at Stafford Beer's viable system model, he spent the most time designing that model on how information moves through the system and where it needs to go just because it's such a hard problem and so important.

WILL: I think it's easy to [inaudible] and I think that's one of the reasons that it doesn't happen as often as you need it to. But it's clear that actually designing the information flow and consequently the feedback loops is how systems evolve. But I think often it's viewed as kind of trivial or kind of unimportant and fixing that perception is part of the overall journey here.

REIN: Yeah. One of his ideas is that competent information should be empowered to act. And the question then is how do you get, what is that information? How do you get it to the people that need it? How do you in a timely way, how do you know beforehand who needs what when so that it can be delivered at the right time and so on.

ARTY: It's one of those things that once you see everyone's like, "Duh," it's so obvious you took the time to write down this diagram of how the information flows across and saying, "Okay, yeah." But then once you realize that everybody's aware of the information flow and operates now in this context because everyone's on the same page with it, you like go, "Oh, okay. Maybe we should take some time to draw some simple diagrams so that we're all on the same page about how this whole system sort of needs to flow together."

REIN: We should do reflection.

JESSICA: Okay, yeah. I got a lot out of this conversation. One of my favorite things was the difference in reasoning about properties versus reasoning about behavior. I think we're taught in school how to reason about properties in math and logic and those make sense to us. But I think we could learn a lot as a culture about reasoning about behavior. This is not easy for us. And it's interesting. Fault injection was listed as a tool. And I think there's a lot of research in social systems or social sciences about reasoning, about behavior that we can learn from. So, I'm excited about that.

WILL: In addition to collecting at least three or four books that I'm going to need to go purchase, and actually three or four authors that I need to go study kind of the oeuvre, I suppose. I think that the ideas that really struck out to me is [inaudible] sometimes look at yourself as the lone practitioner trying to pull the industry forward. But to me it's just so heartening and exciting to hear all of you talking about different ways that you're looking at these problems. And each of us doing a little bit each day to advance the state of the industry is how it's going to get into what we want it to be over time. So to me just listening to the ways that you're all thinking about this and making concrete improvements each day and each month to me is really encouraging about where we're heading. And I'm, in general, quite optimistic about where the industry is heading in the long run. This made me think even more optimistic about where it's heading in the short run as well.

ARTY: The thing that really struck me was the discussion around decision making. And what I started to see as systems thinking is a way to think about how to optimize the quality of decisions. There was lots of discussion around dimensions. And looking at a problem and I see two dimensions here. Let's see, we've got dimension A and dimension B, and this deconstruction of problems and ways of seeing a system of constraints, a way to make decisions. And one of the dimensions in the decision making was always time. So we've got these time horizons that we're optimizing for short term decisions and long term decisions and learning how long those feedback cycles are can impact this too. So we talked about not seeing all the consequences of your really brilliant ideas. I think one thing I'll take with me is looking around and seeing this chain of feedback loops, a chain of flows that impacts the decision making as the system of constraints that I'm trying to see.

REIN: Cool. I had a different reflection, Arty, but I like yours so much that I'm going to try to build on it instead. Russell Ackoff has a saying that I really like, which is that a problem is an abstraction extracted from a mess by analysis and that's a lot of mouthy words. What it means is that a problem is a reduction of the system. When we think about a problem, we've done it by reducing the system in some way. Maybe we're only going to consider these dimensions when in fact the full space has many dimensions, or maybe we're only going to think about this part of the system in isolation when in fact this part of the system interacts with many other parts of the system that we're not choosing to consider. So one of the most important skills for a systems thinker and a problem solver is the ability to form a problem with an understanding of the complete mess that we're choosing to not think about right now. And one of the challenges here is that for a model of a system to fully predict all of the possible behaviors of that system, it has to be at least as complex as that system. That gets back to Ashby's Law of Requisite Variety. And so what we do when we -- Will was talking about models that can simplify a complex system. And the reason that those models are wrong is because they don't have enough variety to compensate for the variety that's in the full system. So we choose models that are wrong because they're helpful because the reduction helps us to analyze things that are interesting to us and we try to ignore, to the extent possible, things that are less to us for our purposes. But it's important to know how not accurate we're being by having some understanding of the more complete system we're extracting this model from.

ARTY: There are variations that we're concerned about, that we care about, constraints that we care about optimizing for, and there's other things that we can sort of leave as variables and not care. However they turn out is fine.

REIN: Yeah. And when we do that, it'll reduce the predictive power of our model, but hopefully in ways that don't matter to the result we're trying to achieve. That's the sort of bet we're making when we make these models.

ARTY: That makes sense.

JESSICA: Thanks, Will, for being here. Thank you, Rein. Thank you, Arty.

REIN: Thank you, Jess. And if you enjoyed this podcast, et cetera, et cetera...

JESSICA: You can contribute to our Patreon. It would be really helpful because Mandy does an amazing job editing, and it's our Patreon that makes that possible, managing and everything else she does, and putting transcripts on the website, et cetera, et cetera, et cetera.

ARTY: So yeah, contribute to our Patreon. What is it? Patreon.com/GreaterThanCode. And then join our Slack if you want.

JESSICA: And it'll be fun.

REIN: The Slack has all of us and a lot of our guests and a lot of really nice, supportive people that have really cool conversations.

JESSICA: Yeah. Pop in, ask a question, recommend a guest. It'll be fun.