Eric Anderson: Most pieces of software that are open source by large companies are terrible. Some of them are good, most of them are recruiting tools. It's not that they're bad pieces of software per se. They might be beautiful pieces of software, right? But they might require a team of people to maintain or they might require reading this extraordinary set of docs. If you're lucky if they were documented or they might solve a problem that only exists due to the confluence of other technologies that are used at one of these big companies, but they never solve the problems of the average business in the world. This is Contributor a podcast telling the stories behind the best open source projects and the communities that make them. I'm Eric Anderson, I'm here with Jeremiah Lowin who is the creator of Prefect and rather than have myself described , I'd rather hear from Jeremiah. Jeremiah Lowin: Sure. Thanks, Eric. It's great to be here. That's always the most challenging thing, right? Is trying to sum up what a framework does but at its core Prefect is a tool for workflow orchestration for making sure that the components of how you work with data and how you build applications around data do the things you want them to do and the way you want them to be done. So we are a tool for defining the flow of data through a system and ensuring that it gets where it's supposed to be. Eric Anderson: Great. And then we want to spend most of our time in the story behind Prefect which I feel like is just the story of you Jeremiah, to a degree. Jeremiah Lowin: Yeah. I don't think it's entirely me. I think maybe I was there for the initial spark of it but honestly, a lot of the... Yeah, a lot of the hard work and a lot of success is owed to the team that we've brought together since those early days. But there's some piece of that maybe it's interesting to go back and explore which is, why build this thing and more importantly maybe what problem does it solve? And this is just me talking, right? But the thing that's most interesting to me is that this is a problem that is extraordinarily obvious to literally anybody who's ever worked with data. They know this problem, they know the frustration that we're describing and this is a problem that has been attempted to be solved for not just years but decades in various forms and all of the solutions are terrible. Jeremiah Lowin: And I think one of the reasons they're all terrible is they're all designed for one thing and then attempted to be ported to a second thing. And it's just not the way this works. So this is such a fascinating thing because it's not hidden, it's not small, but it has for really decades defied any attempt at... Or solving it. Just to be blunt, it just all solution suck. And so it's quite something to put something in the world and find this incredible response to it. And this is excitement and literal happiness from the people who use our software that it's solving their problems. Eric Anderson: And maybe we're getting ahead of ourselves a bit but part of the reason there's so many attempts at solutions is because people had to do something. And so everyone came up with a solution and to your point, others picked it up and probably took it to places they never expected it to go. Jeremiah Lowin: Yeah. I think that's absolutely correct and actually we can go a step further. Another reason is that, this is a problem that feels easy. So let's talk about one example of that, right? When you put code into the world and you want it to do something frequently it doesn't do that thing, it fails. It doesn't matter why it failed, it failed and so the first thing that you'll do is you'll add some form of a retry, right? If no request is received, if some success criteria is met, whatever it is, run my code again. And you'll just do that, you'll do that as an engineer, it's practically trivial to introduce a retry to your own code. The moment that you've done that you have crossed the line from writing code that we would call positive code in the Prefect vocabulary to code that is negative code. Jeremiah Lowin: You've started writing code that's defensive, it's making sure something happens and therefore, whether you like it or not, you've begun writing a workflow orchestrator. And so you can imagine that very few people who encounter this class of problem ever turn and start looking for a third party solution to it. They'll always start by writing their own code. And if you look at, for example, we know how people arrive in the Prefect universe and more than half of them come from homegrown tools. And that's for this reason, there is not an engineer in the world who has encountered what we call a negative engineering problem and not try to solve it themselves at first before they go looking for another solution. And that is how you have this proliferation of tools in our space and our orchestration space. They all come from usually an engineer and he, or she looked at some problem that they were solving and decided, you know what? I can solve this and by the way my solution should be used by other people too. Jeremiah Lowin: And for any number of reasons it's not quite the way that other people think about the problem. And therefore we have another check box in the list of tools that have been introduced to solve workflow orchestration and another failure in the list of tools that attempt to solve workflow orchestration. And that misperception that this is a trivial problem is why we have this explosion of interest, this universal agreement that this is a problem with very few truly universal tools. Eric Anderson: Yeah. And you're speaking like the enlightened one who's been through all of the stages. You did the retries, you've probably built your own till you tried all the other tools. Maybe tell us what was the problem you face, or what was the point that you reached where you knew you had to do ? Jeremiah Lowin: Sure. Look, there's a little bit of a ridiculousness into what I'm saying, right? I'm saying to you, everyone has pointed this problem and been like, I can build a good tool. And of course I am doing, I am saying that. And so what right do I have to say that I have the best ones? That good question. And we can talk about where Prefect came from I think we will in a moment, but a big piece of that is Prefect was never written for anybody but me. Prefect was never designed to be a company, it was never supposed to be opensource, Prefect is literally a tool that I built to solve problems that I had and I think this is the interesting thing which happened to be across an extraordinarily large number of disjoint domains. And therefore that may be the key insight maybe but there was never a piece of this which was, we're going to open sources or we're going to build a company on this. This was a selfish tool at the beginning. Jeremiah Lowin: And then it was when people tried to buy it light bulb goes off and hey, maybe we should get this out into the world. So I don't know, I think that I should at least acknowledge the hubris one could accuse me of having for leading with this introduction and then being one of the same people I am laughing at. Eric Anderson: You built this to solve your own problem. And I don't know whether to start with the problems you had to solve in order to build this, or the moment when you're like, it's time this becomes more than just my own project or my soul... Yeah, my own thing. Jeremiah Lowin: It's great question. So let's dive in. I'll tell you the story and we'll see where that happens. As you may know or some of your listeners may know, so I'm a PMC of a very popular piece of software called Apache Airflow which also solves the problems that I am describing here. It is used to orchestrate workflows and it was one of the first tools that I think in a really effective way introduced the idea of orchestration as code or workflows as code, right? You write Python code, it's translated into a dag and put out into the world for executions. It's a very cool idea and a very effective idea. And in Airflow at least implemented in a very, very simple way which on the one hand has allowed an extraordinary number of use cases to be mapped onto that simple interface. Jeremiah Lowin: And on the other hand makes the tool in my opinion unsatisfactory for any one use case, but can't do enough. It does a lot of things, it doesn't do enough of them well, but I spent a number of years building Airflow. I'm extremely proud of the work we did there and as a consequence of that I got exposure to thousands of data engineers. And more importantly their complaints, their problems, their emails, that they would send to our support list for example, and the issues they would open on GitHub. When you are one of many authors of an incredibly popular tool and you are inundated by what people hate about it and what they're trying to do with it, you learn a lot. And I attempted to turn those lessons into constructive actions in the Airflow world. Jeremiah Lowin: I introduced something called XCOMs once upon a time for moving data around, I introduced attempted desk integration which has been abandoned I understand, I introduced proposals to run things off schedule, there was all this stuff. I had one of the first concrete Airflow 2.0 proposals four years ago. Airflow 2.2 is very famously not out and I wouldn't expect it anytime soon if I were you, I attempted to make these changes and in many cases they were shot down. And I say that with enormous respect for the other folks on the PMC, I came from a very different place from many of them, I am a data scientist by background, I'm a machine learning researcher by background. And most of the folks of the PMC are not data scientists they're data engineers, and our understanding of the use case that we should optimize for was therefore somewhat different. Jeremiah Lowin: And I think that that's an important facet and many of those folks have been and continue to be great friends. A number of them are investors and advisors in Prefect now, and it's been a lot of fun to continue working with them as their careers and objectives have expanded and their needs for workload frustration has expanded as well. But this was where we were at loggerheads. I wanted to do things that I couldn't convince the rest of the PMC were important, this tool. Which was viewed as primarily a slow moving batch on schedule data engineering orchestrator. And I lived in a very different world. Eric Anderson: Yeah. I'm hearing from you that you're not necessarily being critical of anyone as much as you are pointing out that projects can reach a point where they have to satisfy two masters. And that's a tricky thing to navigate. Jeremiah Lowin: It is, people ask us all the time when will Prefect kill Airflow? I'm like, we don't have to kill Airflow, I love Airflow, I won't kill Airflow. That was a great tool. If you're doing the things that Airflow is good at, and if you're not doing those things, you shouldn't be using Airflow. One of the early things that we put up when we first introduced Prefect is we put up a comparison of writing some code in Airflow and writing that same code in . And I got I thought odd criticism of the comparison. People said we had cherry picked something Airflow was bad at. And I was so confused by that, because I wasn't trying to replace Airflow. I was trying to build something that attacks the same problems that Airflow suffered at, that Airflow couldn't attack. Jeremiah Lowin: And so of course our comparison was something that was hard to do in Airflow, right? That the Airflow was bad at. But it was so interesting to me that Airflow was viewed as such a general tool even though to me, it was so obvious that it isn't that we receive criticism for that and we took that down, and after months of work, we introduced a blog post called why not Airflow, which has been received very, very well. And is in all honesty and extremely balanced view not about use cases and writing code but about the choices that were made to build Airflow that we don't think are applicable to a modern data stack. And I think that we successfully tapped into the mentality of the crowd that we're trying to reach in presenting a balanced explanation for why we had to do something, why Airflow is on satisfactory. Jeremiah Lowin: Anyway, to jump back to our story though, I was frustrated. I had things I needed to do that Airflow could not do. I was overseeing risk management for a very large investment firm at the time and had this very broad portfolio and broad mandate. And every day I needed to do things ranging from a simple as sending out emails and risk reports. And back-testing potentially tens of thousands of trades for a quad fund that I was overseeing. And this diversity of activity demanded a workflow system that could keep up with it. And one of the more interesting things that I kept hearing when I would make basically these complaints about Airflow's ability to handle it to the rest of the PMC is some of them ran data for some of the largest companies in the world. Jeremiah Lowin: And they would say to me, well, Airflow works for us. How can it not work for you? It doesn't make any sense. What do you mean it doesn't scale? Of course it scales. We run all these giant companies and I would point out yeah, but you have 50 people whose only job is to run Airflow. And I have me and my job is not to run Airflow. I actually don't want to think about this at all. And this sounds crazy in retrospect, I convinced my boss to officially allow me to spend my time building myself a workflow system, because I used Airflow to show him that in the right domain a workflow system can be credibly powerful and drive value and save time. And I said, I know enough to do this, to add value to my own job but I need a data science first workflow system. And with your permission I'd like to spend my time building it. Jeremiah Lowin: And he said, "Go for it." So I for months took everything I knew about Airflow that made it great, stripped out all the things I hated, which turned out to be all of it. We had to start over from scratch and ended up with what would later become . And then this light bulb moment happened when a whole bunch of firms that I work with came to me almost by coincidence in the space of a couple of weeks with some workflow related problems and they asked me for advice, I just knew I'm plugged into this world on PMC of Airflow, et cetera. And they were trying to understand how I solve this problems in Airflow. And I said, you really can't but I have this little thing that I've been running my own workflows on, why don't you take a look. Jeremiah Lowin: And I gave it to them and they came back. This is a head of data science, head of engineering, a couple large companies and they said. "This is great. If this was a product, we'd probably be interested in deploying it." And I said, huh. And I went to my boss and I said, "This thing I've been working on I got people want to buy it, can I sell it to them?" And he said, "Sure. But why don't you figure out what that means exactly." Literally, what does it mean to sell this? And I said, "Oh I really don't know." And so I did about 50 phone calls and I started paper selling a product. And very literally I would ask people what their data problem was. I learned very quickly, you never tell people what your solution is. No one will ever agree that you have solved the problem that they haven't told you about yet. That's a terrible way to pitch product. Jeremiah Lowin: Instead, I would ask people what their problem was, their number one most frustrating data problem. And they would tell me, and I would ask them if I could do the following things, would that solve your problem? And they would say yes or no. And if they said, yes, I would hold up this little piece of paper where I had written down I think eight features. And I would say, great. So that's numbers one, three and seven on my list. And they would say yes. And I'd say, and if I could deliver these to you, that's a product you'd be interested in? They'd say yes. And then I wrote down... I dug this up recently, I think two thirds of those calls ended with someone basically saying, we'll buy this if you build it. Of those 50 calls. Jeremiah Lowin: And so I said, great. And I was now armed with people who are interested in the software, I was armed with prototype of this software and I was armed with eight key features that if this software had I now knew I could solve... Let's just say two thirds of the sample of 50 problems that I had just collected. And what was very interesting was that even though the features that would have solved those problems were very small in number only eight, no two people had the same problem or if they did, they didn't describe it the same way. And this Snowflake aspect of data problems is one of the reasons I think that most attempts to build a unified framework for solving orchestration fail. Because you ask your people with their problems to conform their problems to your framework and that's very unnatural. Jeremiah Lowin: So we have a belief at Prefect that the role of the orchestrator is to help you in your code fails. But since nobody wants to write, nobody anticipates code that's going to fail, nobody expects that, every single moment that you ask someone to adopt their workflow to your workflow software is a cost that they are incurring for using your software. And so the more that your software diverges from their expectation or even from how they would express their own problem, the greater the cost you are asking someone to incur to use their software. And I am the person who I've had more than my fair share of PRS to Airflow that introduced syntactic sugar and magical things. I love when software becomes invisible and does things for me, right? To a fault almost. So I was very, very, very into this idea of, okay, I've got 50 different problems and I can basically apply a dimensionality reduction to them to end up with eight principle components, if you will. Jeremiah Lowin: So horribly bastardize this metaphor. I had eight things that if I solve them would explode and actually let me cover the space of data science and data engineering problems. And so armed with all that, we renamed the whole thing . We set up a company, we transferred the software to it, and we began this process to open source it and build a business. And that's when I accumulated this fabulous team that we have now and we created these important guidelines for how we operate and the values that are important to us both in our business and our open source and fast forward a couple of years and you get to today where we have this really thriving community and this extremely rapidly growing open source product. And this just really this young, but just fantastically exploding commercial business as well. Eric Anderson: You mentioned something I wanted to go back to in there that on this show we've talked to several people who come out of big companies and we've talked about the merits and values of that. They have captive user base, they have cutting edge scale problems that sometimes are hard to find. And I think I'm realizing from your discussion here that there's a con to that, as you point out you can solve Facebook or Uber's problem. And then you try and push that on the rest of the world. And maybe the rest of the world's problems don't quite fit so well, but you can extend things and massage them into the framework. But in your case, you didn't have a single perspective at this problem as much as you had a heterogeneous one. Jeremiah Lowin: Yeah. I think that's been a hallmark of my career to be honest with you. I've worked for an enormous multi-strategy hedge fund, I've been a machine learning consultant with every client of mine had a different problem, I've worked for this diverse portfolio investment firm. The nature of the problems I've had to solve don't just vary from portfolio to portfolio, they sometimes varied from just day-to-day within the same domain. And this requirement of being flexible and adaptive has been incredibly important to me. And we do see this. We see this in most pieces of software that are open source by large companies are terrible. Some of them are good, most of them are recruiting tools. And there's a good reason for that. It's not that they're bad pieces of software per se. They might be beautiful pieces of software, right? Jeremiah Lowin: But they might require a team of people to maintain, or they might require reading this extraordinary set of dots. If you're lucky, if they were documented or they might solve a problem that only exists due to the confluence of other technologies that are used at one of these big companies but they never solve the problems of the average business in the world. Which doesn't think about things that way and doesn't have this same reliance on technologies and doesn't have the same intersecting spheres. And I believe that this was true so strongly that when we started Prefect the company we explicitly did not target enterprises, because we straight up thought... Mistakenly as it turns out, that they had solved the problems that we were solving. We believe that they threw money and engineers at this type of problem built in health systems, some of which have been open source and you can have a look at them. And we just figured they would have no interest in us with our flexible little package. Jeremiah Lowin: Now that all change there because you and I have discussed when one of the largest technology companies in the world showed up and basically just bought almost sight unseen our pre-release software because they were so fed up with their internal tools. And I remember this team said they had used Airflow in a previous job and straight up refuse to use it again. And they literally just came in. We wrote the login screen of our software so that this team could literally log in to a piece of software they just purchased access to. And that was another light bulb moment for us, right? We're like, huh. Well, this is one of the largest companies in the world, one of the best technology companies in the world, they seem to have a problem that we can solve, that must mean something. Jeremiah Lowin: Also, this is a company that somewhat famously is not always eager and not an early adopter let's say. So armed with that knowledge we figured there must be lots of other companies out there that would be interested in our software and our similar size. And we very much changed what we were doing. And without abandoning the small, medium businesses and individuals that we had spent a lot of time curating we decided let's do both. Why not? Our software seems to solve the needs of both large companies and small companies. There's no marginal cost to us pursuing them other than time and that's a talented salesperson can evaluate that ROI on their time very well. Let's go figure this out, let's see if we can do both. And for the last about 12 months probably has been an important part of the story of how we've built this. We can serve an opensource customer, a relatively inexpensive customer, all the way up to some of the largest companies in the world. Same product, same platform, same eight things that solve this incredible array of application frustrations. Eric Anderson: One of the things I'm starting to believe is that I don't want to project is of course, more than just the projects this community and good communities develop their own vocabulary and almost colt way of viewing the world. And you've mentioned the several things in this podcast I want to come back to it. So one was this blog post that seems to have made some waves about, why Airflow, why . And then the other are these ways you talk about problems like negative engineering. And I feel like that speaks to people, it identifies you as a group, identifies people within the group. We all share the same concern and it's as unifying as it is the solution. You present a solution but you also presented a way for thinking about and talking about the problem. Jeremiah Lowin: Yeah. This has been a really important thing especially for me because I wasn't joking when we started this off and you asked me like, what y'all do. And I just like, well, that's the hardest thing. First of all, I ramble a lot as it's probably evident to anyone who's still listening to us but also it can be hard to say in one sentence something that ties all of this together. And so we really experimented a lot in the beginning with how we communicate this problem. And of course you remember one of the early learnings was we don't tell people what our solution is. And we certainly don't tell them what their problem is. We need them to volunteer their problem to us in a way that they can then see that our solution maps onto. That's the only way that this works efficiently and so negative engineering was the product of days, may be more. A long time of trying to different ways to communicate this problem qualitatively, not specifically. Jeremiah Lowin: And the reason we call it negative engineering is honestly because negative engineering is the word that we started using that literally everyone who heard it started nodding and was like, yes, we have that problem. It didn't matter if they were an engineer or a manager or a CEO, or it doesn't matter who they are. Negative engineering is a thing where obviously anybody who hears that they have something negative will start in and be like, yes, we need to get rid of the negative thing. But beyond that, once we explain the negative engineering is this need to write this defensive cruft that is orthogonal to a person's purpose. It's not, if I'm hired to do something I'm very capable, I write positive code, I get my job done, I also spend an enormous amount of time. Jeremiah Lowin: We were finding it to be about 90% of my time writing what we would call this negative engineering code. This just frustrating, repetitive, minor stuff I have to do to make sure the code that I was hired to write actually runs. That is something that resonates with people, even if they've never written a line of code, even if they're just thinking about organizational problems or just problems that exist logistically or operationally within their business. You can find examples of this and so it's a phrase that really resonated. And when we discovered that we decided to make it ours and we made our company's mission to eliminate this negative engineering. There's not a good reason for it but nonetheless, it is critical and necessary. The career that I referenced earlier, I spent most of it in risk management or overseeing risk management functions. Jeremiah Lowin: And it's a very similar feeling. Risk is orthogonal in many areas in finance, right? It's orthogonal to the objective which is returns, people have all crazy beliefs about what you can do is just management. And the truth is you can't do that much. Once the risk has been encountered, your done. So the true value of a risk process is the ongoing measurement understanding and generation of let's say, experience and intuition such that when you encounter the problem, you minimize the time it takes to figure out what steps forward you can take and how to contain it. Prefect is very, very, very similar by being a genuine partner to our users and minimizing their cost of adopting our software we can be there, come on the ride for them and then the moment they encounter a problem, we cannot unfortunately prevent the problem from taking place in some circumstances, right? Code's going to fail. Jeremiah Lowin: But what we can do is at that moment we can suddenly become useful, we can suddenly point them to the log, we can suddenly point them to the task, we can take any semantic cue that they've decided to imbue the system with and deliver that insight. And so this idea of being a risk manager for code is a philosophy that I've spent my whole career just thinking about, not for code necessarily but just in general, right? Risk management and that is where this negative engineering idea comes from. We cannot make the negative engineering truly vanish in the sense that no one will do it, but by bringing best practices to it we can condense the amount of time that any individual has to spend worrying about it. And that's really our goal here. Eric Anderson: Yeah. I think that's really powerful to speak. There's a lot of chest stumping about solutions and I don't know that people always identify and want to clean to those chest talking about solutions, but when people talk about problems it unifies folks and I think you've done a fantastic job there. Jeremiah Lowin: One of the things that we struggled with in building Airflow, and I think Airflow is still struggles with is Airflow really wants to help people do things as ridiculous as that sounds right? And because Airflow lacks a strong I don't know, a hand on the runner, right? There's no strong sponsor of it, it's very much designed by committee. It very much accepts pretty much anything, any new piece of functionality whether it adheres to some philosophy of how these things are supposed to work or not. And as a consequence it just becomes spread very, very, very thin. Whereas for us we don't want to get in people's way, it just sounds so crazy to say out loud but we don't want to do anything for anybody. If you're a machine learning engineer and you're trying to build a certain model you should use the tools that are great for building that model. We are not going to pretend that we are the best tool for that. Jeremiah Lowin: But once you have that model once you have that code written, that's where we want to plug in. And we know that that's our domain and we are very excited to stay in that domain. And by being able to exercise that opinion, we turned down feature suggestions that we worry would take us too far into basically competing with great tools that skilled engineers can deploy in the pursuit of whatever they're doing. What we love is when people integrate us with those tools so that we can hand off and then receive back feedback from that tool in a really non-invasive way. And so I think that's where this chest thumping comes in. Everybody wants to solve the problem, but the engineer is great at solving it. But let the engineer solve the problem. We just need to be there to support them and make sure that their solution works. Jeremiah Lowin: It's just such a funny thing to say I almost feel silly saying it, but it's so core to this negative engineering idea that we are not in somebody's way when they are writing good code. We are only there when they need our help. We are insurance [inaudible 00:30:15], so we think about our company, we're an insurance company. Eric Anderson: And then another topic that you alluded to that I'd be curious to hear your thoughts on, you mentioned at the beginning that you didn't set out to make an open source project or a company. But eventually you were pulled in these directions and the open source vision has unfolded over time. It was we'll open source this, and with time you realize that there was more to open source, I'd love to hear how you discovered that along the way. Jeremiah Lowin: Yeah. So we have the saying there's a difference between what we make and what we sell. For all companies there's difference between what you make and what you sell, right? For us in particular as an open source company we make workflow software. If we sold workflow software then we would compete with ourselves as an open-source company. I would argue and Eric, you probably are in a better seat than me to decide whether this is a true statement or not. But I believe that many open-source companies are open core companies that fail are companies that never really figured this out. And they think that the code that they put into the world it's also the thing that they sell. And maybe it's just a managed version of that code which I think is ridiculously terrible business model or maybe it's you pay us and we unlock more features of that code. Jeremiah Lowin: And in all cases you're competing with yourself, right? You're giving something away but you also are selling that same thing. And in the early days of Prefect we decided to give away the tools that our users needed to build their workflows. So the core engine really, and we have this whole thing we haven't really talked about with this hybrid setup where every single one of our users is basically an on-prem deploy but we manage all the infrastructure. And it's very cool how it works and keeps costs down and keeps privacy Hines, it's really great. But as a consequence that we had to make sure that we could deliver the entire engine to our users. And so that was the thing, we're open source's workflow engine, and then it became so popular and people started expecting a much more full suite of workflow tools. Jeremiah Lowin: And this was a bit weird for us because on the one hand, we had a free version of that but on the other hand, it was proprietary code that we supply to is a SaaS software. And it was actually when COVID settled in back in March and at the time we didn't really know that we'd be in the situation we're in now as we record this one in early September. But back then, it just seems like this event that had just come through and just decimated so many of the things that we took for granted. And so we made this decision... well, to be honest, we didn't just make this decision, I came up with a new set of assumptions about how the world would work and how the world of software would work. And some of those assumptions were quite scary, right? Jeremiah Lowin: They involve things like, nobody wants to buy software right now but people are going to be desperate for open source software. But open-source development is going to stagnate because most open-source maintainers are just volunteers and this is certainly not a priority for them. Now there's this list of things I came up with. And as we looked at this list we said, well, the obvious strategic decision here is that we should open source what had formerly our proprietary platform. And the reason we should do that is that people could really benefit from it. And we always say we are trying to deliver value to people, we are not trying to extract value from them. We're trying to deliver, we're trying to solve a problem that they have. And here's an opportunity where we can take this action and really align ourselves with people at a moment where at least in our judgment, they really need something like this. Jeremiah Lowin: So we just believe that the appetite for our platform was much higher in that moment than it had ever been before. One thing that we had struggled with if I'm completely honest with you, is that we save companies money. So we agree to say we struggled with, why do I say that? Because companies in 2019 were a lot more excited when we could demonstrate how we would make them money or something more top-line correlated. But it's really, really easy for us to show how we can save you six figures. All of a sudden in March of 2020 that changed, companies became incredibly interested in anything that could save them money. And so, even though we made all these assumptions and just make these strategic decisions and decided to open source our software our commercial inbound went through the roof, because we're sitting there saying we can save you lots of money and you'll keep using your same code and costing nothing to implement us. And you can do a POC in a few hours. Jeremiah Lowin: And so all of a sudden we were in this bizarre situation of our opensource took off and so did our sales. And what we decided to do therefore is not undo our decision, we decided to lean into it because clearly we had found a way to make something workflow software, and sell something of an insurance product built on top of that software that was resonating in a non-competitive way. We had these two great balanced businesses that were growing. And so a few weeks ago we relaunched that platform as an open source product and did a lot of the things that frankly we didn't have time for in March when we were trying to move quickly, created a shared code base, proper open code, so this stuff. But the result was again, this incredible interest and most shocking to us was right after we announced this release and we said, here is our biggest release ever, Prefect was the number one trending repo on GitHub for a few days. Jeremiah Lowin: And the way I can quantify that as I think one third of all of the GitHub stars that have been given to our open-source project showed up within the last 30 days. Which is an extraordinary, crazy thing to think about. So our community exploded and we saw this in our Slack community, we saw this in our three tier users, we saw this in our commercial inbound as well of course in the most visible metric, our GitHub stars. So this has been a very interesting journey of deciding not just what to open source but how to open source it. What coach we would put out, in what form, with what support guarantees, how do we document it? How do we deliver it? To be blunt, how do we as a young team, as a lean team, provide the coverage that people expect from such a broad suite of software as the one we're offering? And the answer is we can't in a lot of ways. Jeremiah Lowin: So we have people who they try to use our software and they're like, well, for example, I don't know, why don't you support push button deploy on AWS? And the answer is because we have a very, very small team that is focused on what in our judgment are the most high value pursuits. And that's not something we've had time to do yet, but that's the wonderful thing about building this great source community is we're working with folks who are building that. And so while we expend engineering hours to oversee and make sure that that's done in a way that complies with our design objectives. By getting an excited and motivated community together we can collectively build the product that is most useful even if any one piece of it might not be in our company's highest ROI pursuits, if we want to be extremely just transactional about it. Eric Anderson: Yeah. On that point as we wrap up here, I'm sure there's people listening to show that want to be more involved or just excited about what you're doing. Maybe what are the upcoming milestones we should be looking out for? What are the maybe things that people can do to learn more about the project? Jeremiah Lowin: So the easiest thing is obviously check out our GitHub, it's github.com/prefecthq/prefect or just Prefect IO. If you want to see our site itself that's a jumping off point for the code, for our Slack community, for our GitHub discussions, for getting in touch with us, for just kicking the tires, for signing up for our SaaS is everything can be done from there. But from a roadmap standpoint we're rapidly approaching the place where our cloud product, our SaaS product, and it's open source version with Prefect server is reaching a level where I think if we're completely honest, someone who signs up for it right now is in some respect an early adopter. Just by virtue of what the software looks like. At the end of this quarter our literal product roadmap is to no longer have that be true. Jeremiah Lowin: And so this remaining long tail of features that will satisfy the 80 percentile needs, will be available and present in our platform and that's very exciting for us. And so the question is on what's next. And here's a crazy thing to say, but we've talked today about this data orchestration problem and the fact that we've built Prefect to solve it in a modern way. But if I can be completely honest, it's very amusing to us at Prefect when we get referred to as a NextGen System, because we are not a NextGen System. That's a very honest assessment. We and many other tools like us are just the latest implementation of this decades old paradigm. The difference is that we run a modern hardware and we interface with modern software and we do it a lot faster and we do it in a lot more easier way and we scale better and all this other stuff. Jeremiah Lowin: It makes us feel familiar to a modern data scientist but there's nothing NextGen about that. It's just the natural upgrade that frankly I think Airflow is supposed to take the 2.0 upgrade five years ago. And for whatever reason has elected not to, and just sit there. What we are going to be turning our attention to is a few months ago internally... We haven't really publicized this yet, but few months ago internally we drew up our new product vision. Because we made this product vision in 2017 to allow people to run any workflow anytime, anywhere. And that was our governing statement. And at the end of this quarter we will basically be saying that is achieved, people who are using our software no longer early adopters of this theme. This is a real thing that we're putting into the world. So what's next? Jeremiah Lowin: So basically our goal is to do to ourselves what we think we have very successfully done to Airflow. Which is to look at Airflow, look at where it fails and the use cases it fails at, we identified those eight key words and we said, these are the things we need to do to be compatible with the modern world. So one of the nice things about the relationships we've built with our users and with our customers is that they're not only telling us what problems we're solving today but they share with us the problems they'd like us to solve in the future. And so, very similar to sitting on that Airflow support list and hearing about all the things people were doing and trying to do, and learning about this incredible wealth of activity that people were frustrated that Airflow couldn't meet. Jeremiah Lowin: I know the same list of complaints about Prefect. Again, feels a little bit ridiculous on one hand we here like, we make great software. Now they have to be like, but we don't do everything. But it's true, we don't do everything. There are things that are not really compatible with the way that Prefect exposes vocabulary, but our users very much would like to be and some of them have even begun hacking that together. And we don't want people to be hacking our system that's how I forgotten the trouble, we want people to be embracing our system and we want to deliver that utility. And so in the fourth quarter, we'll be designing a lot of this and probably in 2021 we'll be introducing a lot of new features. They're not going to change how Prefect works, Prefect is going to work exactly the same. Jeremiah Lowin: If you wrote a flow a year and a half ago, that first customer I mentioned they deployed flows in I think August of 2019, but are still running today with no change to code. And we were like super committed to maintaining that compatibility and we will. But there is this set of problems that is very obvious to our users in the same way that the problems Prefect solves are very obvious to Airflow users, that they can't do it for Prefect today. And so we have this entire roadmap about how we're going to solve that. And most of them to be honest with you involve breaking this stag paradigm that Airflow certainly didn't introduce, but very much popularized to the point that it's what we call a flow, they call the dag. We don't call it a dag because that is too inflexible for our uses. Jeremiah Lowin: And we need to go well beyond that. And to give you a preview of some of the things we're thinking about, we're going to basically let people break a lot of the rules that today we enforce, we and other tools like us. An example of that would be, right now when you have a task, it's pending, gets scheduled, it runs, it succeeds, it's done, we don't revisit it. There's no concept of a state after a finished state and maintaining that state hierarchy is super important to us. It's how we have the integrity of our system, but we have a lot of users who it turns out they're using Prefect and Prefect flows to monitor infrastructure. And it's very confounding to them that if they're using a task state to represent the health of that infrastructure for confounding them to rerun that flow every time. It breaks a semantic understanding they have because their infrastructure is health exists on a continuum, not in this batch interval, this check, this ping's heartbeat. Jeremiah Lowin: And so, one simple example of a thing we'd love to allow such a person to do is it let a task go back into a running state and evolve through a number of finished states. It's successful, it's successful, it's successful and event takes place, it's now failed. The events resolved, it goes back to being in a success state. And that's a real time continual file you streaming to use an extremely loaded word in our universe way of working with the states of things is heresy in the current database Prefect world. But something that I think you'll be able to see in the future from the tools that we support. So that's the stuff that we are getting extremely excited about is this incredible variety of use cases that we know we can't deliver today. But once we check the box on our stable platform in a couple of weeks, that's what we'll be turning our attention. Eric Anderson: Very exciting. I can totally imagine a persistent state for everything. And it's just continuously available as opposed to a thing that cycles through it, very good. Jeremiah Lowin: We hope that it's as much of... I'm so stupid to say, but some of our users thought it was a real idea revolutionary when you could run something off of schedule, because Airflow doesn't allow that, right? You run things on a schedule. So the fact that we can just run your workflow at anytime for any reason in response to server less computer, that was revolutionary. So this to me feels like the same degree of sea change in terms of what use cases we'll be able to support when we go to this continuous setting, it's going to be very cool. Eric Anderson: I'm excited. Jeremiah as we wrap up, I should also direct people for a good time to your Twitter handle. I just think you're a funny guy. What- Jeremiah Lowin: [crosstalk 00:45:12]. Eric Anderson: Can I ask about... I shouldn't and we can edit this out if you want, tell me about sobbing gifts. What [crosstalk 00:45:24]- Jeremiah Lowin: No. We don't have to edit that out. It's just embarrassing I guess. I don't even know what it is, not embarrassing. I'm certainly not embarrassed but I do it every week. Eric Anderson: No. I just left this great inside joke that I feel I'm not quite a part of but trying to be. Jeremiah Lowin: I think you are as much a part of it as any of us including myself or Jim, if I'm completely honest. So why did this start? I don't know why this started, Jim O'Shaughnessy for those who don't know is first and foremost a legend in the financial industry and someone who I've been fortunate to know for a very long time. And the O'Shaughnessy Jim and his son, Patrick are great partners of ours at . And Jim is a master of gifts. If you follow him on Twitter is a lot more rewarding experience than following me on Twitter, you know that, right? That's his most common responses are gift. And for some reason I decided to start tweeting these horrendously bad jokes at him. I don't know why, I don't know why, I love puns. I love terrible puns. I just really enjoy them. I like wordplay, whatever, I liked a lot. Jeremiah Lowin: And someone sent me a really great joke once. And I was like, I want to tweet this, but this will be a terrible tweet, I at least recognize that. And I was like, but if I could just bring someone in to be the straight man basically to think back to... I don't know, Laurel and Hardy or something stupidly old fashioned foreign prominent. Yeah, exactly. I was like, who can I bring in? Whose reputation can I not quite tarnished but still bring into this. And so I just picked Jim and used this ridiculous format and it did surprisingly well at the time. I didn't have nearly as many followers I do today and I was shocked by this. And so the next week I did it again with a little bit of a different format, it didn't do so well. I did it the next week with the same format, it did really well. And so every Friday for about two years I have tweeted this idiotic series of jokes at Jim O'Shaughnessy, who every week responds with how he'd like to murder me and that is my relationship with Jim O'Shaughnessy. Eric Anderson: Now we're all in on it. Got it. Jeremiah Lowin: And it's in public and the whole world can join in on this insanity that every week I apologize and people actually write it. And they're like, this week was really good or this week wasn't so good. And I'm like, no, no, no, no. That's not really the point here. No assumption of quality in anything that's happening in this weekly series. Eric Anderson: Jeremiah, thanks so much. I've loved this chat and you've been a good friend as we got to know each other the last year, and you're doing an incredible job with this work, we're all excited about it. Jeremiah Lowin: Well, thanks Eric. I appreciate it. And I appreciate the invitation to come on. It's been a lot of fun, it's been wonderful to meet folks like yourselves who just have this knowledge and this domain expertise that for a company that's trying its best to simultaneously do a lot. And as I said, do nothing as weird as that sounds but being able to share ideas and get inspiration from folks like you means the world to us. So thank you. Talk to you soon. Eric Anderson: Bye. You can find today's show notes and past episodes @contributor.fyi. Until next time, I'm Eric Anderson and this has been Contributor.