Ev Kontsevoy: It's quite the responsibility because on one hand, you need to make sure that only quality contributions get in, but on the other hand, you want to be nice to people. I want people to be excited about programming, first of all, but also be excited about participating. Eric Anderson: This is Contributor, a podcast telling the stories behind the best open source projects and the communities that make them. I'm Eric Anderson. All right. I have the privilege of being joined today by Ev Kontsevoy, one of the creators of Teleport. Ev, thanks for coming on the show. Ev Kontsevoy: Thanks for having me, Eric. Eric Anderson: Ev, we have lots... We could discuss. Normally, we want to cover the origin story for Teleport, and I know some of it, but I would love to hear the nitty gritty. First, maybe you can tell us what Teleport is, to give everyone context on what we're discussing. Ev Kontsevoy: Absolutely. Teleport is an open source tool that engineers used to access their Cloud resources. Cloud resources means SSH Servers, Kubernetes clusters, databases, whatever you have inside of your Cloud accounts. It's basically a much more modern replacement for something like SSHD, but except... Unlike SSHD, it speaks more protocols than SSH, and it gives you a lot of this modern practices like SSO, second factor, unified audit log, web UI, and a bunch of other things that engineers expect to have and enjoy these days. Eric Anderson: Perfect. You've been noodling on this problem space for some time. Tell us the story of how Teleport came to be, and feel free to go back as far as you'd like, because we'd love to hear your story as well. Ev Kontsevoy: Of course. Look, I love going back. I started my career as a... I was building applications that today you would call them desktop, right? You would have something like Windows in T or some other operating system and then you would be building applications that run on a single computer. I enjoyed it. The process was that you sitting in front of a box, right? Say you have your keyboard connected to it and you just typing in instructions, you creating your software and then it runs on this machine. It's a very instant experience. You make something and you see it live, it starts walking in front of you. It was awesome. Then, we started building things to run on the internet. If you just starting, it feels similar, so you also have a single box and it's somewhere in the data center and that box, instead of throwing pixels at you, it basically has a CTP protocol that you would use to build web UI. Ev Kontsevoy: Then, as your application grows and it becomes more complex, it starts spreading goes from one box to many. Then, you start using all kinds of middleware components and you have databases and radius cache, and then you have elastic search and then you have multiple regions and multiple Cloud accounts, even sometimes. It gets crazier and crazier and crazier, and the fun of interacting with it just goes away. This is kind of a personal level, what I've been observing with Cloud computing. Meanwhile, as your application is growing to consume more and more and more of these components, how do you access those components like that? Intimacy of having a keyboard and the mouse plugged in, just say, machine in where everything you need to runs on. I call it a local host experience. Local host experience is less and less irrelevant today. Ev Kontsevoy: Even to build applications now, you basically need all of these Cloud resources. To recreate that intimate experience, that's really what we wanted to do with Teleport. If you just imagine that all of this Cloud accounts, all of this VPCs and availability zones and regions, all of that stuff is like one giant computer, and your internet is your USB cable. Eric Anderson: Yeah. Ev Kontsevoy: How can we make that experience to connect into this one giant computer connecting you to the matrix like in the movie? That's why we built Teleport. It's a way for engineers to have instant access to everything they have inside of their Cloud accounts. That's also why it's open source. In fact, Teleport was originally not created as a business. It was almost a side project of ours. It did not have a price tag. It did not have a commercial license. It was just the product of love that we've created because we wanted this experience to be true. Ev Kontsevoy: Then, over time, companies figured out that the way Teleport does security, the way Teleport does access controls. It's incredibly appealing to them as well. That's how this business was born. We started building something to make programming easier to make building Cloud Applications easier and ended up being in this security and access and compliance space. A lot of companies now are paying us money because paradoxically, while Teleport is extremely easy to use, it also happens to be... Basically, it's like a consolidation of industry, best practices for access, as you can just download and run. That's the story. Eric Anderson: Got it. You have a personal passion for this, but you didn't do it alone. Right? How did the team come together to build Teleport? Ev Kontsevoy: The team that built Teleport, we actually, it's our second project we're working on together. Originally, we started assembling the team around 2009, 2010. We were working on a company called Mailgun, it was email delivery in the Cloud, because early Clouds, they all sucked. The very first version of everything always sucks. The EC2 and AWS and the early days, they didn't have the ability to send to receive the email traffic because SMTP was blocked. We figured, we'll solve this problem for early Cloud pioneers, we built this company called Mailgun, which is still around. It's doing really, really well, so I'm very proud of that. Then, we got acquired by Rackspace, one of the cloud providers. In the process of growing Mailgun team and the process of growing Mailgun, we witnessed how complex Cloud environments become over time. Ev Kontsevoy: For example, just migrating Mailgun infrastructure from one hosting provider to Rackspace, took us many months. That's what complexity does for you. It can even move around your workloads. The dissatisfaction with this growing complexity started there. Then, once we were at Rackspace, we kept thinking and thinking about what we can do to reduce that complexity, to make applications run ideally everywhere by themselves. It just so happens that the access problem is number one problem to solve. If you want to enable this simple computing, I create for simple computing in the Cloud, and the team here does too. Eric Anderson: Got it. You're with this group doing Mailgun, you go to Rackspace and then the band gets back together to work on something that eventually has a side project called Teleport and Teleport takes on a life of its own. Ev Kontsevoy: That pretty much true. When we at Rackspace, we wanted to figure out, "Can we make applications run in fully autonomous mode?" If you picture... Like your laptop, right? The laptop has... It's actually a very complex machine. It has a lot of complexity in it too and it has a lot of applications so you probably have many, many millions lines of code in the written all kinds of languages running on your laptop all the time. Then, an interesting thing is you get software updates, is it Apple or Microsoft? One of those operating system vendors, they do a software update where you get new version of everything. All of this happens without any DevOps people, right? If you look at this from... If you can zoom out and see what is actually going on, how is it different when Apple runs software update to Apples versus doing a Cloud deployment? Ev Kontsevoy: It's actually pretty awesome. They are deploying a software to millions, maybe even billions, I don't know, of servers scattered all over the world over untrusted connection, over unreliable connection, and all of this happens without DevOps people managing these boxes. That's pretty amazing. We wanted to build a similar technology for data centers and that project was called Gravity and Teleport came out from that. We wanted software to be able to run by itself without any DevOps teams globally and in order to make it so, we realized that if we want to run it everywhere, then you need to actually enable secure connectivity to everywhere first, right? Because how it even gets there, that's the number one problem you need to address and that's where Teleport came from. Converting, again, all of these Clouds converting this different form factors of compute into a single seamless environment, that's the first step towards it and that's what Teleport is. Ev Kontsevoy: You put Teleport on your infrastructure and it starts to feel like it's all in the same room with you. That's why it was called Teleport. Yeah. I encourage everyone to go and play with it. It's a single binary. You could run it as a Linux Daemon, you could stick it into your Kubernetes pod. I have my own... Well, company is paying for it, but I have a half a rack of equipment in our Fremont Data Center here nearby and I have another one in my house. I have to look for it running everywhere and it feels like, "How all of the servers with me all the time?" It's pretty nice. Eric Anderson: You mentioned something earlier. I think we need to dig into, you said that this was a protocol aware solution, because I think many of us have experienced resources behind the VPN or something. We've requested access to that VPN and thus the resource, but you're describing something different and I'm not sure we've captured the nuance here. What do you mean by protocol aware? Ev Kontsevoy: Yeah. Let's get technical. I love that. Eric Anderson: Yeah. Ev Kontsevoy: If you think of, "What is the best way to implement the access" Right? You have... Let's just use databases or SSH Servers as an example. Teleport could support additional resources, but I like those too, because we all use databases and we all use SSH. When you try to connect into, let's say Postgres database, or you try to connect into a Linux box via SSH, how do you authenticate? How does that machine or that box knows that you are who you claim you are? There're basically two approaches to that. One of them is to use some shared secrets. The simplest form of it is username and password, and we all have agreed that it's basically not a good idea to use those because people tend to reuse passwords. Ev Kontsevoy: Then, the most of the time today, most organizations and teams, they use a slightly more advanced version of it based on keys. You're having a public private key, something like that, but that's still a shared secret. That's also can be stolen, so it's basically anti-pattern. Another anti-pattern that people frequently employ is reliance on perimeter security. If you have a database, for example, and let's say, it's listening on two sockets, one of them is a socket and local hosts, or it could be a local network. If you connect into your local network, there is no password, there's no encryption. Why? Because I'm on my own VPC. I trust everyone who is on the same VPC. Zero trust movement, the purpose of that movement is to eliminate that. You should not be relying on perimeter security and every resource needs to think that I'm on a public internet, anyone can try to connect. Ev Kontsevoy: That's another thing that people oftentimes rely on this perimeter security and a third solution to access is organizations now realizing, "Well, we have a lot of servers, we have a lot of databases. We have a lot of apps and environments, and we have this teams that are distributed." Which just cut access to as many people as possible. No access is the new default. In combination of those three things, share credentials, perimeter security, and bureaucracy. That's really what it is. That's the state of access to the most organizations. Not only it pisses people off, but it's also not awfully secure because I can look into an organization like those five people are allowed to access to production. All right. They're going to hack into their laptops, get the keys, and then they get access to everything. Ev Kontsevoy: That is what Teleport solves. Teleport basically says "No. No form of shared credentials ever." Instead, Teleport relies on your identity that you have within the company. When you show up for work and you authenticate via something like Active Directory or some organizations use Google Apps and smaller teams might be happy with just GitHub, right? You go authenticate against GitHub and you have that credential. Teleport brings this down to the protocol level. What it means is that you try to use a command line like pgSQL or you try type SSH or kubectl if you're using Kubernetes. Once you try to run these commands to a portable check, if you are authenticated against your identity for your company, if you not, then it will ask you, go and do the SSO [inaudible 00:13:23] with multifactor and all of that. Ev Kontsevoy: Then, once you are authenticated, Teleport will create a certificate. We believe that only certificate based access should be allowed and we implement certificate based authentication and authorization and we can talk about difference on a protocol level. We bake you a certificate for SSH, and that's how you access a SSH machines and then we make your certificate for Kubernetes or for a database, and that's how you access those. The difference between a certificate and a shared credentials is very simple. Certificate is unique to you. The resource you accessing like server, for example, it will know that you're Eric, you're not Ev. You're not hiding behind some alias like admin or a dev, because a lot of this shared credentials, they share it for a reason. They have very generic names and which allows these resources to make decisions to grant or deny certain actions to based on who you are. Ev Kontsevoy: That means your authorization is now based on identity as well. It also allows you to have extremely detailed audit. For example, when you try to run a Sudo command in the middle of SSH session, Teleport might actually say, "You know what? You need that additional permissions for that." That capability was not even possible before, or if you're trying to drop a SQL table, like another obviously dangerous operation. You got to use your YubiKey one more time, just because you're about to drop a SQL table. That's what Teleport enables. It's basically extremely easy to use thing. You don't have to change anything, but you getting the state of the art access technology for all protocols that we support. That is how Teleport is fundamentally different from anything else that's out there. Eric Anderson: Thank you for clarifying the nuance. These are good examples. The Sudo command dropping the SQL table. Do users then specify the things that they think are sensitive? Does Teleport just come out of the box with like, "We know dropping SQLs is probably something you want to care about." How does someone tell Teleport this is the stuff that needs approval for and who to go to approval for? Ev Kontsevoy: Well, that's the software engineers do. When you design your stack and you decide, "Okay. We're going to be using Linux, then we're going to put Kubernetes on top. Then, we're going to use Postgres." For example. All of these things you need to decide ahead of time, what is privileged operation? What is not privileged separation? Then, you have to rely on something that we've all been familiar for, like role role-based access control. You have to say, "Okay. I need to have roles that are not privileged, I need to assign this roles to people in your organization." You can do that using even GitHub. In GitHub you can create teams, basically creating groups and for different groups, then you can define permissions. To create groups, you use identity management. I already said, Active Directory, Google Apps, GitHub, whatever allows you to create teams and groups of engineers and assign memberships. Ev Kontsevoy: Then, in Teleport, you then go take these groups and you specify which actions are allowed, they're not allowed for these groups for individual resource that Teleport support. Then, what Teleport will do as I already said, so we will enforce certificate based access, but we will also utilize native access controls of that particular resource. Basically, means that's what we put into that's a phrase that I often use that we bring identity down to protocol level, means that your group memberships from Active Directory will go all the way on the wire, all the way to database and database we'll see, "Oh, you're a member of a different group and you getting access denied." Ev Kontsevoy: If you want to insist that I need to have access, maybe because there's this mission critical ticket that I need to troubleshoot, then you can request access right there in the middle of a session and your team will be able to approve or deny your access using something as slack, which means that your permissions will be elevated temporarily just to do one privileged operation and then dropped back to the original level. This is why Teleport is so much better than relying on this bureaucracy where. No access means no access, simply because we realize that the world is extremely dynamic and you have to be using modern access solution that gives you exact same ability to grant to deny access. Eric Anderson: Got it. I show up to work. I request access to do my common development and Teleport knows discovers who I am. It tells the database who I am and I'm off to the races. Then, as soon as I want to drop a SQL table, it's going to intercept that command and be like, "Whoa. You need elevated permissions. Do you want to request those?" Ev Kontsevoy: Correct. Eric Anderson: Yeah. Okay. Yeah. Ev Kontsevoy: I Want to make one correction. In the beginning you said like, "I show for work and they request access to what I need." Actually, no, things that you need access every day by default, you have access as long as you authorized with your identity provider, because it's extremely important for your access solution to be convenient. If you force engineers to be inconvenienced by it, you're causing all kinds of problems. Obviously, people will become less productive, but those who are a little bit more adventurous and energetic, they will build back doors into your own infrastructure. It's really hard to make all of the engineers to use [inaudible 00:18:58], so to speak or Inconvenient Access Methods. They will just inject this as HG into their own stack and they'll create their own socket the own, even know about it. Eric Anderson: For my standing permissions to run queries against the test database, I don't need to request anything special, but as soon as I want to run queries against production, or I want to tear down that test database, it's going to intercept my command. Tell me I need elevated privileges and give me a path to request those in slack or something. Got it. Ev Kontsevoy: Exactly. We try to resolve this conflict that exists across many organizations, security versus productivity. We believe we found a way to deliver amazing security and a lot of compliance controls as well, without pissing off engineers, without sacrificing and you have productivity benefits. In fact, I would even argue that if you're using Teleport, you're way more productive than if you're using standard source tools that come with. Kubernetes, it has its own kind of access layer and the SSHD is on every server. To start using those, you need to be creating jump hosts and proxies and figuring out how to configure this IP addresses to be discoverable. With Teleport, the way it's architected, we can give you access to anything, anywhere without VPNs, so we're going to take a look into documentation how to do it. Ev Kontsevoy: Spoiler alert, it's based on a reverse tunnels, so every resource can be connected to a Teleport proxy via reverse tunnel, which means that you can connect to things running behind [inaudible 00:20:36] on someone else's infrastructure with Teleport like a self-driving vehicle or some IOT drone flying around. All of these capabilities are going to only contribute to how productive you could be if you still a port faxes. Eric Anderson: Now, take us back to the narrative, if you will, now that we understand how this all works. You're building this autonomous deployment platform you discussed earlier and release Teleport on the side. We mentioned that it took on a life of its own. What did that look like from the creator's point of view? How do you see that there's activity around Teleport that's picking up? Ev Kontsevoy: Yeah. Yeah. There was really confusion is the word. We initially I do remember, I think I took a call with a very early open source user who wanted to run Teleport on smart colors, that you put on [inaudible 00:21:31]. It was funny to me because we always talk about, "Let's treat servers, like we treat cattle. [inaudible 00:21:38] pets." Here we go, that actual person who's running code deployments into [inaudible 00:21:45]. I think the interesting thing was that someone offered us money before we even wanted Teleport to be a product. I think someone reached out and said, "Do you have a license? Because we want to be using it in a mission critical deployment and we want to have a relationship with the vendor." We continued not to take Teleport too seriously, simply because it's lightweight. It's not particularly expensive and we had this other product that we were selling at the time called gravity. Ev Kontsevoy: Gravity was this autonomous Kubernetes cluster technology. It's also open source, it's also out there, but we don't actively develop and evolve this anymore. That was quite unexpected and then I was shocked by how mature companies were. They were reaching out to us and asking questions about Teleport, because normally what you would expect if you have a project like this on GitHub is home lab for people would reach out and say, "Hey, I have my VMware in the basement or I have like OpenStack cluster on a bunch of raspberry pies and I'm tinkering with Teleport." I expected that, but we had people stock exchanges or Silicon valley companies who just raised a series of funding or companies that are preparing to go IPO. Ev Kontsevoy: They started reaching out to us and saying that we actually have Teleport and open source version and we would love to use it in production if only we've had additional capabilities or integrations. It wasn't sudden, I don't think it ever is, but I would say in the matter of six to eight, nine months, it became pretty clear that we have something that the world desperately needs. Eric Anderson: Got it. Then, you started putting some more attention on Teleport. It sounds like it's been a busy year for the project. What's been going on? Ev Kontsevoy: At the of last year. Actually, there'll be a year before. At the end of 2019, it became absolutely clear that Teleport solves a really, really big problem that a lot of people are experiencing day to day and we need to focus companies attention on Teleport simply because opportunities are huge. Mark my words, I don't believe that five years from now, it will be the norm for companies to manage access for every single protocol for every single layer in their stack that kid's going to go away. The way you accessing databases, Kubernetes clusters, even internal web apps or associated server is going to be through same proxy that needs to be identity aware. That is what we believe will be true. If everyone will be using it and if we are the best solution out there, so as you need to focus on it and completely own the space, that's what we've decided. Ev Kontsevoy: Since we've migrated all over engineering efforts from our previous project to Teleport, we nearly doubled the size of product in less than a year. We originally only supported two protocols, SSH and Kubernetes API, because Kubernetes, I act as in Kubernetes is exact same problem with SSH, you have many clusters and they have different permissions. How do you make sure it's all synchronized and so on and so forth? We added support for two databases and just the few months, Postgres and my sequel, we rolling out MongoDB support soon. We've added support for HTTPS, meaning that if you have your internal web apps, something like Jenkins or Grafana Dashboard, and you have plenty of those sprinkled all over your infrastructure so how do you centralize access to all of this internal web apps without having to expose every single one within a load balancer, X.509 certificate, public AP, and so on and so forth? Ev Kontsevoy: All of that was launched in the last few months. It's extremely exciting. The product is growing very, very quickly. There are again, plenty to do. There are so many different things, people run inside of their VPCs and Cloud accounts. Once you experienced Teleport [inaudible 00:25:51], you wanted for everything else that you're using. We need to build more, build faster without sacrificing quality, of course. Eric Anderson: Tell us about the Teleport community. You got this strong team, building a bunch of product and I'm impressed at the GitHub Stars, which is just one metric, but there's some a lot of people have discovered this thing and seem excited about it. At the same time, there's not a lot of a... If I want to go hang out with a Teleport community, that's still in its infancy, it sounds like. Ev Kontsevoy: Guilty us as charged. Absolutely. Eric Anderson: Yeah. Ev Kontsevoy: Absolutely. I think it maybe has to do with a DNA or personalities of the founding engineers on a project, right? Eric Anderson: Yeah. Ev Kontsevoy: If you go and do get log and Teleport source code, then you start looking at the first three contributors. People who like our CTO, [inaudible 00:26:45], and couple of other engineers. They're not particularly visible out there, some of them actually extremely pro privacy. We, as a company, in the early days, particularly, we were extremely introverted. Most Teleport contributors prefer to be coding as opposed to be visible at conferences or on IRC or on Twitter or something like this. This is slowly changing. We just recently launched public Slack channel for Teleport users and they instantly got hundreds of people. They showed up out of nowhere. I don't even know how they discovered. Ev Kontsevoy: I think it's because we added Slack channel to the README, that's on GitHub. I don't even think we have it on our website. We slowly learning how to enable community to exist, how to build this relationship. We need to rapidly get better because even companies that pay us money, like enterprise users, they have support, they have account managers, they enjoy amazing treatment by Teleport team, but they still want to interact with the community. Maybe for a variety, that's why community is a called community. It's just that a place for people who have something in common to hang out and have a good time. Yeah. We need to absolutely be better at building our community and interacting with it. Eric Anderson: What's the future hold for Teleport from here having... I heard the story and about the last big year, what gets you excited about the next year to Teleport? Ev Kontsevoy: One interesting thing is that we will go back to our roots a little bit. Remember I started my career as a building desktop software windowss, apparently east of Silicon valley, half the world runs on windows. Windows support, support for protocols like RDP, it's on the [inaudible 00:28:34], so this is something we are actively working on. Then, we get a lot of requests from users to support more and more database types. Because on one hand, data is why we actually do computing. It's the most valuable thing. That's really what the computers do, they transform one data sets into another. Companies these days, they have variety of databases in different locations. If you want to make sense of this data, you have to convenient access to it, because if everyone in your organization is getting access denied to all the valuable data you've collected, then what's the point then? Ev Kontsevoy: That's another obvious vector for a Teleport to be evolving. I also believe that we could always... If we just remember where it all came out of the frustration with complexity, I want Teleport to continue to be simpler and easier and easier to get started with. Basically, user experience and making it appealing, not just to security professionals or a very experienced DevOps people. I want the 14 year old to be enjoying Teleport. You get yourself your first digital ocean account when you tried to get into Cloud computing, Teleport should be obvious way how you access everything that's inside. Instead of you building your own jump host, following tutorials where you don't even understand what you copy pasting. That's what the future holds, the near future holds for Teleport and the team. Eric Anderson: Yeah. Then, the community sounds like that will be turning a corner here as well. You've got the Slack channel live and you'll be doing further engagement there. I'm imagining, for listeners, a few of them love Teleport and want to meet each other. Then, for those who don't want to jump in, the Slack channel is the place to go for now. Ev Kontsevoy: Yeah. Absolutely. I think the best place to start tinkering with Teleport and understanding how it works is probably go to GitHub page. If you could Google a GitHub, Teleport, SSH, and then this keywords you'll find it, or you can just go to goteleport.com. From there, you can go to the open source area. Eric Anderson: What's the state of outside contributors? Is there a place for that here? Ev Kontsevoy: Absolutely. Absolutely. Having a project in the open is quite the responsibility because on one hand, you need to make sure that only quality contributions get in, because look, it's a security product. If someone is using Teleport that basically controls access to everything I have. The security angle and just overall level, like the weight of this responsibility is enormous, but on the other hand, you want to be nice to people. We've had examples where college students that are working on a project, for example, learning something, they would send us a contribution. They don't want to be discouraged. I want people to be excited about a programming, first of all, but also be excited about participating. Ev Kontsevoy: At the same time, it's unrealistic for a college student to just expect them to have top quality contribution into a project like Teleport. Working with contributors, working with them on improving quality of the pull requests, it's extremely important. We try to balance all of these considerations, and it's not always easy. Sometimes people not aware that having a popular open source project out there creates this tension, but it's unavoidable and it's a sign of something good so you just have to be very pragmatic and just be nice to people when you go through this. Eric Anderson: It sounds like your approach isn't to just ignore outside contributions, it's to engage with those contributions as they come in and improve them to the point where they're ready to go in the project. Ev Kontsevoy: Yes. Yes. I do believe that one of backhands that Teleport supports for storing artists for MongoDB, I believe came from the community that we had to go back and forth to make sure that the compliance and security constraints that we have there. They're still addressed. Yeah. We also learning how to get better at it. Yeah. It's a balancing act. Eric Anderson: Fantastic. Well, you've done great so far. I have all the confidence you'll continue to navigate the balance. Ev Kontsevoy: We will do our best. Eric Anderson: Ev, thanks for coming today. Go find Ev on Slack and GitHub and we'll catch up with you later. Ev Kontsevoy: All right. Well, thank you. Thank you, everyone. Eric Anderson: You can find today's show notes and past episodes at contributor.fyi. Until next time. I'm Eric Anderson and this has been Contributor.