Egil Østhus: You can always train on the engineering capabilities, but if you also have this open-minded willingness to learn, willingness to share, celebrating this open mindset, I think you're building something very, very strong. Eric Anderson: This is Contributor, a podcast telling the stories the best open-source projects, and the communities that make them. I'm Eric Anderson. Eric Anderson: Today, we get to talk about Unleash, an open-source feature flagging project that is quite popular. It's had a really good run the last couple years among community development and adoption. I'm joined today by Egil and Ivar, who are creators of the project. Maybe you two can introduce yourselves so that we know your voices. Egil Østhus: Absolutely. My name is Egil Østhus. I'm the CEO of this company behind Unleash. Ivar Østhus: Yeah, and I'm Ivar Østhus. I'm the little brother of Egil. I'm the CTO, the technical head of this company. Eric Anderson: Fantastic. We got the whole family here in the house. Probably not the whole family, but the important part of the family for this discussion. Tell us quickly what Unleash is. And then of course, we'll go into the history. Egil Østhus: Sure. Unleash is an open-source feature management platform. We are delivering feature toggle, or feature flagging to basically any programming language that you can think of. It is open-source, built on top of an open core business model. Yeah, it's a great tool. Eric Anderson: And others may be familiar with feature flagging tools. And there's kind of a spectrum, it seems like, on some that are a bit more marketing oriented, help marketers experiment and measure the results of experiments, and some maybe more for engineers to make safe deployments. Where do you feel like Unleash fits in the spectrum? Egil Østhus: Yeah. Good question there. Unleash is very much focused on the developer. We are very focused on developer experience. We definitely recognize the importance and the need of experimentation, and AB testing and such. Where we are very strong is on our capability to work on top of DevOps and infrastructure components. Very flexible when it comes to also application context, and exactly where the application runs, and allow you to do a lot of segmentation capabilities on top of those. Would you agree, Ivar? Ivar Østhus: Absolutely. I think it's important to also remember the backstory of Unleash, and we are coming from a very developer engineering type of culture. Eric Anderson: Ivar, maybe take us back then to the beginning. I believe it may have started with you. How did Unleash come about? What were you doing maybe that led up to starting the project? Ivar Østhus: Yeah. I care deeply about developer efficiency. And for me, I was working in a team in a very forward-leaning company. I was the tech lead of that team, and basically in this company, we had automated all the codes deployments to all our environments. We could ship, in theory, codes to production every day, but we didn't. We had our weekly sprints, but even for a week, we can not always release code to production. Ivar Østhus: And this bugs me a lot, because we ended up having uncomplete features hanging around in long-running feature branches, because they were not ready to enable all the uses yet. So we had to do more testing. We had to write more of the edge cases, all of that, before we could ship that code out to production. And for me, this was bugging me a lot, because it added a lot of complexity to our workflow. We had to manage all these feature branches. Ivar Østhus: It could be very hard to actually merge them back to the main branch. And we're also postponing all the learning opportunities. So I started evaluating, how could we solve this better? There has to be a better way to solve this scenario. And for me, I very soon discovered that there was people talking about feature toggles, and I felt that this was a very compelling solution to this problem. And I started to investigate that a bit more. Ivar Østhus: You have to remember, this was back in 2014. There wasn't a ton of solutions to select for. There wasn't any closed-source vendor at the time that provided this as a service. So I could either do what most people did back then, was a static conflict file in my application. That helped me a lot. I actually started with that. But I still wanted something that was a bit more dynamic that allowed me to change the configuration run time, so I could have more control on who got access to a new feature. Ivar Østhus: I could enable it for myself and nobody else, for instance, in production. And this was a key win for me. And this was also why I wanted to create something like Unleash, because I really wanted something that could work across all our microservices. It could centralize the need I had to take more control on how we release software. And I just felt like this was an obvious choice to open-source it from day one. Eric Anderson: Ivar, you were kind of the original. I mean, you said there was other people doing feature flagging and feature toggles then, but it was still quite new. I mean, this was something that maybe was popular in Google and Twitter, but hadn't really made its way into most of the world. It feels like you were ahead your time. Ivar Østhus: Yeah, I feel like that every day. And I still feel like we were early. It's like even today, people are discovering this technique, and it kind of amazes me. But still, the market is growing in. And I think that a lot of people see the value of doing more trunk-based development, that it removes a lot of complexity. And I'm so happy to be in this space at this moment in time, that this is actually the perfect time to be in this space, because there are so many talented developers out there. And there is so much pressure to work even more efficient with software, and deliver quality, and be able to verify your things in production. Ivar Østhus: And also, there is so many reasons to do that. You have the privacy concerns. Today, it's not that easy to just dump all your production data into a test environment anymore. And having the ability to actually test in production without exposing it to everyone, it just makes so much sense. Eric Anderson: So you wanted at release an open-source, and so you publish on GitHub one day. How do you find the first users, or who stumbles into it? Ivar Østhus: Yeah, that's a funny story. So I think it was officially open-source the 1st of January, 2015. Nobody was using it, except my team. It was out there and no traction at all. It was a repository in GitHub, that's it. But I still felt like this was a fun project. It was a fun project to keep. And also some of my motivation of open-sourcing it was to allow myself to use some of my spare time on building this project. So I wasn't going to be tied to the company I was working for at the time. Ivar Østhus: I kind of kept it as a side project. I used it obviously in my day job, and my company was very encouraging towards this. They felt like it was cool that we were open-sourcing it and all of that. But what I'd experienced was that it was very easy to adopt for the other teams in this company. There was more than 20 other teams that saw that we were using it. We used it on our Sprint demos, where we showcased that no, we are actually releasing to 2% of our users on stage. Ivar Østhus: And the defect to get from that is that everyone wants to work like that. Everyone wants to take that level of control to the software they're delivering, and just spread organically within the company was never a push. Nobody was ever forced to use the tool, but they wanted to and they started using it. Ivar Østhus: And the next thing I saw was that when developers left this company, one of the first things they did at the new company that joined was to adopt this tool, and introduce this tool into a new company. This was the beginning, I guess, on the growth you can find for the adoption of Unleash, is that it was basically word of mouth, people that have already experienced it, or heard someone use it, and they wanted to try it out. And they saw the value of it. Eric Anderson: Yeah, no, it's the power of those demos, I'm sure, at work where people can... Hey, they've unlocked their feature velocity. They can ship, because they can hide it behind toggles or flags. That's great. And now, at some point you leave the company. Was that in conjunction to wanting to go full-time on the project, or were there other things? Ivar Østhus: Yeah, so this is fast-forwarding a bit, I guess. So it was late 2018. It has started again, a lot of momentum. A lot of companies around the world were already using the open-source, and there was coming quite a lot of inbound request towards issue request, and other type of more enterprise features and all of that. Ivar Østhus: And I literally also got emails from companies, asking me to host this for them. "Can you just host it? And we will pay you." It was literally that. And this was about the time I brought Egil to the table. So me and Egil, we are brothers. And sometimes we go out, and we have beer and food, and I started talking to Egil about this project. And I'm explaining to him like, "Yeah, I have this open-source. It's getting a lot of traction. There seems to be a lot of people using it all over. Maybe there is an opportunity here. I'm not sure, but it feels like there could be a potential to actually build something around this." Egil Østhus: I remember this dinner. It was a fantastic evening. It was actually in the mid of, I think January something, and we were having burgers and beer. And at the time, I was a product development director at a great company, called Visma. It's a M&A company, a shop. It acquires a lot of mature software. And as you say there, Eric, Ivar is very much on the front-end of development and engineering culture. Egil Østhus: It's fair to say that a lot of the software products I was responsible for, I was maybe less so mature when it came to engineering capabilities. But the interesting part is then when he started to explain the capabilities, the use cases and the possibilities, I perfectly saw that this is solving a lot of my needs as well. So although it fits very well into this DevOps, CDCI kind of pipeline way of working, also monolithic application is like a payroll and accounting, maybe not super interesting, but something you need to get right. You don't want to get those numbers wrong. It allows you for opening up a new way of thinking around how you can run this organization. Egil Østhus: So for me, it was obvious that there is a ton of value here. Honestly speaking, I also need to admit, open-source was this interesting twist on everything, but I didn't really have all of the details set out. So for us, it was a great start of a very interesting journey to explore this opportunity together. Egil Østhus: Obviously, Ivar coming from the very technical. I'm bringing in this kind of, how do we look at this from a product market fit? So what we ended up doing, and I think actually this was a great decision, was to decide we wanted to do this as a side gig. So it forced us to actually be very focused on where do we pay our attention, and where do we spend the time? Egil Østhus: And literally, every moment we spent was to understand how does the competitive landscape look like? And where do we fit into this picture? And we were very direct and honest with, we are not starting to build this feature game, running, catching up game with the other vendors. Let's try and understand what is the value of what we have already. And also, where does it fit into the picture? So I think that was the first year or so, just working crazy hours late night and weekends, understanding the customer and how it fits into the market as good as possible. Eric Anderson: Great. I'm imagining this dinner with you, and now your lives are changed. You can't sleep that night. Ivar, you basically became a community manager here at some point, it sounds like. I can see from the GitHub history that you're still doing a lot of contributing, maybe more than ever. But at some point, who's engaging with the community? Is that a new kind of work for you? Ivar Østhus: Yeah, absolutely. There is many ways of looking on how we have been working with the community. And in the beginning, it was more like some people were coming in. And usually, whereas over the time where there have been the most contribution is on the SDK side. So in order to use Unleash, you will have an SDK that you put into your application that would do a lot of the heavy lifting. Ivar Østhus: Actually, it would be the SDK that would evaluate the rules. So it would not need to reach out to the API to check where our toggle should be on or off. And I will do all the synchronization in the background with Unleash API. This allows us to be both resilient and super fast, and all of that. But that also means that there is some complexity in actually building the SDKs for the various programming images. Ivar Østhus: This is where the first major contributions were coming in. So there was one person coming in and building the Go SDK, for instance. I didn't know this person from before. It turned out that he actually, for some time, worked for a sister company of mine, but I didn't know that at the time. And I didn't know him personally. And they were obviously also using Unleash. And I saw that it seems to be easier to contribute to those more concrete SDKs, which would be a limited problem area. It would be kind of isolated, and it also had some kind of API surface, or some contracts that you should implement. Ivar Østhus: I also improved on this over time, so we created a specification for the SDKs, so that it would be easy to validate that they were implementing the protocol correctly, and all of that. There were just more and more contributors coming, providing different SDKs for different programming languages. And for me, I always look at the human side of things. So when there was someone building an SDK, instead of just gladly adopting everything as an official SDK for Unleash, I would actually engage a lot with the maintainer of that SDK. Ivar Østhus: I will look at how this person reacted to issue requests, [inaudible 00:14:48] request, or feature requests. And if this person was very welcoming and open to suggestion, or naturally curious and learning more, I always thought that that is a good sign. And for me, that is even more important than the quality of the code itself, because you can always fix the quality of the code if you're willing to learn. Ivar Østhus: But if you're an asshole, sorry my French, it's really hard to fix that. At least doing this back then as a side thing, I didn't want to spend any time working with people that couldn't behave themselves. And I actually saw that, and I steered away. And I always chose the SDK that were having a really welcoming maintainer. Egil Østhus: Yeah. And I think I also need to add to that, because looking at who Ivar is in the community, and also in our company now, and I truly believe it also comes with myself, it's very focused on values, very focused on who are the people, and always striving for this better for the community, better for everybody. And I dare to say that this also ties, I think it's fair to say that the community members and the employees of the company, the same culture and value is applicable to both parts. And we look at this as an extension of our company, is that it's part of the big family. Egil Østhus: I mean, as Ivar says, you can always improve of quality. You can always train more on the engineering capabilities. But if you also have this open-minded willingness to learn, willingness to share, always celebrating this open mindset, I think you're building something very, very strong. And I perfectly feel this is who we are, both as a company and as a community. Eric Anderson: Fantastic. Switching gears a bit to some of the things that the product does. You mentioned some keywords before that I'll just repeat here, and we can talk about them. You mentioned trait-based, and testing and production, privacy concerns. Maybe help us unpack some of the value that you've described earlier. Ivar Østhus: Yeah. I can try to start with the privacy concern. And maybe because we are a European company, I don't know, but privacy is a huge thing. And I think it would be more and more important in the future. And for me, it's always important that if you are collecting some data, you should have a reason to collect that data. It shouldn't be just because you can. Ivar Østhus: And this also comes back to Unleash. So I've been very careful that we shouldn't collect user data that is not useful. And also, I've taken that a bit further. Today, Unleash will not collect any personal identifiable user data. And it goes down to how the whole architecture of Unleash, it means that we have built artifacts to make sure that we still can solve a lot of these important use cases. Ivar Østhus: So for instance, we have built something we call the Unleash Proxy. The Unleash Proxy will be something that sits between the Unleash API and single-page application and native application, because then we don't need to see the end user. They will not connect directly to our offering. That will be something that the companies can host themself, and connect directly to the end user. Ivar Østhus: I feel like this is super important, because you will use a tool like Unleash to configure your software. You will use it to configure how you are releasing your software. But we are not building yet another analytics tool. That would be a different tool you're using for that. And that's why we're also looking into how we can make it easier to pipe that data to the appropriate analytics tool, or your internal data lake. Or maybe you have some machine learning team that needs the data to learn which version of a new feature that actually performs better. But we don't pipe all of that data back to Unleash API just because we can. Eric Anderson: Yep. Great. And then the testing and production. Because I have feature toggles, I can feel more comfortable pushing a feature out to production, knowing I can pull it back. Or I can roll out even a single feature to just a portion of the population. Egil Østhus: Yeah. Maybe I can start with an example, Ivar. We are shipping a new feature, very fundamental change in our architecture. And obviously we are using Unleash to develop Unleash. So it was a very crazy story. We did the launch, by the way, it was on a Friday. We launched software on Friday. And we did this behind the scenes. So the customer was getting this new code in production. They didn't really care about it, or were made aware of that. And I remember Ivar coming back and saying, "Oh, I did some..." I think it was a lock challenge he had on a database that was pretty bad actually. Ivar Østhus: Yeah. A deadlock scenario. Egil Østhus: So basically, by launching this very silently into production, having customers running larger part of that new code, without even being aware of that, we were able to identify these kind of... I would guess it's a sort of [inaudible 00:19:53] scenario when you look at a test plan. But it was important to find it, because it was creating big challenges for the customers. So this allowed the team to fix that problem, update, roll it out again. And now it's fixed that we are starting to onboard private customers to start using the new feature. Ivar Østhus: I would actually like to extend on that exact example, because we actually did launch the new feature in parallel. So it was actually not causing any problem for the customer at all. And we were able to detect the deadlock scenario and disable the feature that they didn't see yet, and roll it back. But it wasn't impacting customer at all. Ivar Østhus: And that's also something to be aware of, that we knew that we were doing crazy stuff with our database query here. And we were a bit curious how this would scale. So that's also the reason why we thought that this was super important that we did it carefully, did test it with real data before we enable it for everyone, basically. Egil Østhus: Yeah, so this also adds to this where we are seeing ourself, because we are very much focused now on the feature management, the feature toggling capabilities. But when you really start to think about it, everything we talk about is developer efficiency. And this is also when you mentioned this trunk-based development. So trunk-based development is one of those other buzz words that is in the developer community, where we make it quicker. You don't want to have that merge conflict, and all of that difficulties that you have to get long-lived feature branches laying around there. Egil Østhus: But actually, what a feature tool does is to move that complexity into production. So it's also on our mind that even though it's a super great improvement, compared to the feature branches and moving towards trunk-based development, it's not like the silver bullet fixing all problems. We are creating new problems, to be frank. Egil Østhus: And this is also important for us, how we think about Unleash. Because of that, we are very focused to make sure that the teams has the overview of where is the feature toggles? What is the lifetime? Should I start to really clean up stuff as well? Because this is actually technical theft, when I think about it, to have this feature toggles laying around the production, It's super easy just to forget about it, just later. It's done its work and the trick is done, and you move on to the next feature. And I think we can all relate to this is a behavior that often comes into an engineering organization. Egil Østhus: So for us in Unleash, we are very focused on having reporting capabilities in the tool to make sure that the team is at least aware, and can make it cause a decision of cleaning up. And what we see is also often engineering teams, at least my personal experience, need some support to highlight this, also to upper management or to business, or whatever you want to call it, to drive conversation on we actually need to do something that doesn't really impact the feature for the customer immediately. But by cleaning up, we can make sure we are having this capability to stay very fast, and have a short time to market on what we deliver. Eric Anderson: You're helping me understand that I think this ability to feature flagging or toggling is fundamental to be able to do true DevOps and continuous deployment. I think I hadn't connected the two. When I was at Google, we would put things in features. We also, as you would imagine, didn't have a QA team, and developers own their own code. Eric Anderson: But it's hard to imagine owning your own code and not having a QA team, if you really can't control a feature after it's put out there. If everything is just put together, you start to see why you have to have a QA team, and why it's hard for a single engineer to manage a release. I think you've helped me realize how interconnected and how vital feature flagging is to a true DevOps motion. Egil Østhus: For sure. Eric Anderson: Great. Take us to the future of Unleash. We skipped a little bit of the end story, how the company formed and how it's grown recently. But what are you working on today? What do people have to look forward to? Ivar Østhus: Yeah, so it's a good question. This week actually, we have dedicated the whole week with the team to look into the future. So we are actually doing a full week of hacking, just to explore the opportunities. And one of the opportunities that we really see, and that we want to double down on, is how to share impression data from Unleash to any tool in the market. And when I say impression data, I mean when you're using a tool like Unleash, Unleash will help you control the rollout, allow you to define your segments, who should have access to this new feature. Ivar Østhus: The Unleash SDK will know exactly who was exposed to this feature, and in which point in time. And we can actually create a reliable authoritative screen of this information, and make that available to you. So you can integrate that to any tool you're using. It could be, you want to push it on Kafka, or it could be that you want to push it into your own Snowflake instance, or any tool that you are using to analyze how users behave in your solution, or in your platform. Ivar Østhus: We're also doing this by also addressing the privacy concern of this. So we just don't want to just start collecting all of this data and take care of it for you, without you having control on where the data ends up. We strongly believe that this is something the customer should decide how and where this should be stored, and how this is piped to the next service. Ivar Østhus: We don't have all the solutions yet, but we strongly believe that this is a fundamental need when you're working in this way. When you have a lot of features, you're experimenting a lot with new features, you also need to tie that together with not only your business KPIs, but also your performance KPIs. So you may have a data target. You may have APM tools, auto tools, and it could be that a feature is negatively impacting your memory consumption, or CPU consumption, or you may increase your failure rates for the service. And all of these are different tools, probably you're using to analyze all of this. And it will be very beneficial to connect all of this together with the feature toggle information, and who was exposed to which feature at that point in time. Ivar Østhus: And obviously, building on top of that. That's the first part. Start sharing this data. But you also I want to think about taking the next step there. How can we pipe the results of these information back to Unleash? And not per user, but we want to move towards service level indicators, like what are the important safety metrics that you care about for this feature? And have that defined, and use that to control your rollout. And then you can start thinking about much more automation on top of this data. Ivar Østhus: So instead of a person saying, "I want to do 5%, 10%, 100%," what if you could just define, "This is the business KPI I care about. These are the performance KPIs I care about. It should never be slower than, I don't know, a hundred milliseconds," or something like that. And as long as all these KPIs or indicators are within their threshold, I just want to roll it out as soon and as safe as possible, and just have the tool do it for you. Egil Østhus: Yeah. So basically, how we see the market today is very a configuration of a feature. It's a very manual and tedious process to say, "This is the rules. This is some of the segments I want to create. And this is how I want to enable my feature." And by the way, it's also important to emphasize, we look at any kind of change to code as a feature. So if it's a backend, below the surface there, it's very much still feature enablement that is important. Because anything you do, you are impacting the behavior and the experience of the platform and product you put in front of your customers. Egil Østhus: So what we are saying, we are taking the feature configuration as it is today, and we are turning that into purpose-driven feature management. Exactly as Ivar says, you should look at what are you trying to achieve, and this is your concern, rather than thinking around, do I use that activation strategy, or that kind of cohort, or this kind of way of looking at the world. So I think it's going to be a lot of innovation in this space, as we move forward. Eric Anderson: Fantastic. Ivar, Egil, thank you so much for coming today. I think we covered quite a bit of the Unleash story, but it sounds like it's just beginning. I took a glance at one of the few objective measures I can, and the GitHub stars are just accelerating. I mean, I think you'll double in this year, what you've started with, it looks like. Ivar Østhus: Yeah. And that's also quite amazing, that building a company around the open-source project actually accelerates the adoption of the open-source. It kind of builds an additional layer of trust that this will be maintained. This is actually something you can trust over time. Eric Anderson: Makes sense. Thanks for coming. Egil Østhus: Cool. Thanks for having us. Ivar Østhus: Thanks for having us. Eric Anderson: You can find today's show notes and past episodes at contributor.fyi. Until next time, I'm Eric Anderson and this has been Contributor.