Ben Edelstein:
Hello, and welcome to PodRocket. Today I'm here with Ravi Lachhman, who is a principal product manager at Harness. How are you, Ravi?

Ravi Lachhman:
How you doing? Well, Ben, Thanks for having me.

Ben Edelstein:
Yeah. We're really excited to have you on the show, and want to talk about a bunch of things today. I want to learn more about Harness. I've seen the company grow from the sidelines, and seems like the product suite is expanding, and a lot of really interesting developer focused products and features and stuff that I'd like to learn about. And I know you've given some talks recently on developer experience overall, so I want to learn about that. But maybe we can start with Harness. Maybe give us a quick overview for folks who aren't familiar, what is Harness?

Ravi Lachhman:
Yeah. Yeah. If it was a couple years ago and you would Google Harness, dog harnesses would be actually the number one result. We actually did an April Fools joke two years ago that we made Harness dog harnesses, and sent them out for people who requested one. But Harness is a software delivery platform, so it's a really wide brush style paint there. But there's a lot of steps between you as a developer and getting your idea out in production; there's confidence building steps, there's infrastructure. There's, how do you make sure something is running after you deploy something? And there's a wide gamut.
Harness is a platform specifically designed for software delivery. Harness started in the continuous delivery space, so that would be how do you get literally deploying something or delivering something. But this Harness platform has expanded to multiple modules, so it focuses also on build with continuous integration. It has a security orchestration, it has SLO management, it has chaos testing. It has just a bunch of features and functionality out there just to get your ideas any which way into the hand of the end users. And also experiments. I think last year was a feature flight platform that they introduced.

Ben Edelstein:
Got it. A lot of tooling to, sounds like, take you from pre-production to production and everything in between. I'm curious, how does what Harness offer differ from other products out there? There's a lot of tools in the CI/CD space, everything from CircleCI to Amazon and Google have CI/CD offerings. What's different and unique and special about Harness?

Ravi Lachhman:
Yeah. Harness takes a little bit more of an opinionated approach. Harness will actually, for the core continuous delivery platform, mimic what you and I would do together. For example, let's say you and I were on a team and you're the senior engineer, or you're the development manager on a team. And you tasked me, it's like, "Hey Ravi, go and deploy something." Harness, all of your intrinsic knowledge is locked in with you. You know when something's going right, you know when something's going wrong. I have a lot of stories actually about when I started out as an engineer of, I remember... This would be a little more of a long-winded answer there. My very first real job out of school, I was working in the federal space for IBM. TSA was the client. The very first time I went to prod, I started seeing all these errors starting to appear. More and more and more error, error, error and WebSphere(MD), and I was flipping out.
I was like, "Oh, did I crash reporting at airports?" This is not going to be good. And I remember the more senior engineer, she was with me, she's like, "Oh no, no, that's normal." And that's like, I've never seen these in my local environment. Are you crazy? What does this mean? She's like, "Oh yeah, we see them all the time when the application starts up and they'll just taper off." And it's like, I had this blank look on my face that was even possible. But that was such an intrinsic thing. That senior, eventually everyone moves on and when that senior left, there was no one there to teach the next generation or the next set of people who came on. Harness would mimic that interaction. Harness has a lot of machine learning. Word of the day, AI/ML. IT has a lot of that learning in it saying, "Oh, these were actions or metrics or telemetry that was happening before, it's normal or is deviated."
And it takes that across entire platform and helps you, even more so cranking that along the way, solving more intrinsic software engineering problems. For example, test coverage. Was there a flaky test? Do you have too much coverage? Do you have too little coverage? Can test be ran in a more efficient way? And that keeps going module to module. What is the most high priority thing to look at? What is the lowest priority thing to look at? And that's where Harness differentiates that. It embodies a lot of that engineering or DevOps platform manage knowledge into the product into a simple to use, and also, it's a very consistent user experience and developer experience across the product [inaudible].

Ben Edelstein:
And one of the things you also mentioned is feature flagging, that's another area where there's a lot of tools in this space. But I'm curious, how does Harness's future flagging maybe tailor and tie-in nicely with some of the CI/CD and deploy pipelines and things to offer... My guess is the whole is greater than some of its parts there. Yeah, why build feature flagging in the first place, and how does it tie in with the existing feature set?

Ravi Lachhman:
Yeah. There's been a rise in feature flag vendors out there. The whole progressive delivery movement that was made popular. Feature flagging, it wasn't anything... I would say, as a software engineer, a conditional statement has been there forever since this first gen. Take it back to the 60s, if statement was out there. But where feature flagging became a thing was... I know it's a long winded answer again. Once I got at Facebook, they invented one of the first feature flagging platforms. They used to call it dark features or dark rollouts. The ultimate AB test. I used to be subject to them. I would ask people, "Hey, why is my Facebook distribution two gigabytes in the app store versus 750 megs?" "Oh, you're part of a test." It's like, "What were you testing on me?" "We don't know." I had some buddies who worked there, they didn't know what it was. But just like, that's probably what's going on with how you're getting the application distributed to you.
It really brings the power back to people who need to make the change, so it limits the blast radius. That your question probably, how is Harness is different than other feature flag platforms out there? It's the same thing, the sum is greater than the individual parts there. And also, going back to whole intelligence thing. How do we provide a clean way of enabling something? How do we provide a clean way of, when is appropriate to enable something? How do we provide a clean way to give you metrics on, was your experience successful? A common thing, from a developer standpoint, it's always hard to get that data. Usually it's locked away. And a product manager, non-product manager. Right? You talk about I already hear [inaudible 00:07:29]. Sometimes that data doesn't trickle down into the hands of the people who have to make the change. Or as a developer, your core to the innovation, you could have a different lens or different idea how to solve a problem. Harness tries to democratize that saying, hey, this data is available for everybody. And also it's easy to enable and disable a toggle or a [inaudible 00:07:53].

Ben Edelstein:
Enjoying the podcast? Consider hitting that Follow button for more great episodes.
You mentioned before, Harness now does something that helps with chaos engineering. For folks that maybe aren't familiar, what is chaos engineering? Can you just give us an overview of that?

Ravi Lachhman:
Yeah. It came out of Netflix. It's a way of testing resiliency or robustness in your infrastructure applications. There's a concept I talk about, it's a distributor systems concept I talk about, it's called the fog of system development. It's a cliche on the fog of war, it's situational awareness. No one person knows the entire end to end flow. I guarantee you, there's not a single person at Netflix that can tell you every single microservice for every single service 100% what they do, and what's the upstream, downstream, the aggregation of what comes back to you. Because the systems are so distributed and complex, so as an engineer you focus on your domain or you focus on the services that you own, that's it. Again, it's hard to say. There's always just limitations to your purview that you have, even though it might be very wide.
How do you test robustness or resiliency from a systemic way? Going joke, I was taking services out of production way before chaos engineering. It was at one of my specialties. But it's injecting faults in certain places. For example, a standard chaos experiment would be something is unavailable. So you have arbitrarily knocking, let's say... I forget what the chaos monkey specifically targeting specific processes and dockers, or specific machine AMIs are running. You see what your application or your service will do. Now similar thing with chaos, chaos engineering has gotten a lot bigger, so there's a lot more false and experiments to do.
And also from a platform standpoint, going back to, well you have all this data, what happened in variant A versus variant B? Variant A might be, you suffered from a network black hole. So there's something that wasn't able to connect. You can't reroute it back to yourself. Does a thundering herd appear? These are all SRE type of things. How graceful was your handling of this problem? Did it manifest to the user? And that's what chaos engineering's all about. How do you limit manifestation of errors by actually testing these things that you probably wouldn't have thought about before?

Emily:
It's Emily again, producer for PodRocket. And I want to talk to you. Yeah, you. The person who's listening but won't stop talking about your new favorite front-end framework to your friends even though they don't want to hear about it anymore. Well, I do want to hear about it because you're really important to us as a listener. What do you think of PodRocket? What do you like best? What do you absolutely hate? What's the one thing in the entire world that you want to hear about? Edge computing, weird little component libraries, how to become a productive developer when your WiFi's out? I don't know. And that's the point. If you get in contact with us, you can rant about how we haven't had your favorite dev advocate on, or tell us we're doing great, whatever. And if you do, we'll give you a $25 gift card. That's pretty sweet, right? So reach out to us, links are in the description. $25 gift card.

Ben Edelstein:
You recently gave a talk at Dev Innovation Summit 2022, and the titled talk was KISS: Building a consistent developer experience. What does that mean?

Ravi Lachhman:
When I think of a kiss, I think of a Hershey Kiss. I always think about food a lot. But KISS, it's actually an acronym for, Keep It Simple Stupid. It's a design principle. It actually came about from the US Navy in the 1960s. And so, simplicity certainly has its virtues, but when trying to build very big and complex systems, overarching goals should be try to keep it as simple as possible. Because if you start adding multiple complex decisions when you have to do something in a hurry, or you have to train somebody. We all know what complexity do does. If you have two solutions and one is simple and one is complex, and they both achieve the same thing, why don't you just keep things simple rather than making it complicated.

Ben Edelstein:
And I guess more concretely, what are some examples of when building developer experiences where maybe the natural inclination would be to overcomplicate things, but it's important to keep things as simple as possible?

Ravi Lachhman:
Yeah, for sure. It's always funny if you take a look at where the ramp time actually is for a software engineer, and this is more of an intrinsic thing. Let's say you and I have switched teams all of a sudden. We were plopped down into a new team. It's even ironic, even folks in the same company or organization, they go through the same learning curve that if they started another job somewhere else, which is interesting. Your first week you get your development environment set up and you get a local build going, and you're able to change something or make a material change locally or in a development integration environment. But what takes the longest time is learning how to go to production. I must have been, I'm counting 12 or 13 engineering teams before I stepped out of day-to-day development work.
And every time I changed teams it was like, hey, my background was in Java, or distributed Java. And so I was like, well I'm very familiar with Java or JE, but I'm not familiar with how we go about decorating this to go to a deployment system, to validating that the changes were correct, to validating that going to production was the same. And so, usually a lot of complexity that what the engineer, or he or she, the software engineer's not dealing with on a daily basis, they're not typically deploying to production multiple times a day. Now, you might say, Facebook, Amazon, Netflix, they'd employ every 11.6 seconds. I'll make some numbers up there. But it's rare to do that. Take yourself to the other 80% on [inaudible]. It's a big deal and you have a lot writing on it, and there's a lot of unknowns that you try to work your way through.
And a lot of confidence building exercises, that's again, the fog of the development. A lot of folks are not exposed to all the rules in their pipeline. I've been in some very complicated deployment scenarios that people had to sign off on it. It was the regulatory stuff. Or, I didn't know what the next step was, I had to use my nose to edge forward every time. And that's where a lot of complexity is. At the end of the day, you want something going to a hand of end user in a safe and sustainable way. What's the path of least resistance for that?
Funny, going back to a bank I was a consultant or deployed engineer for. It was so funny because they had such a lengthy change control process. But if there was a problem, let's say there was a production issue, 80 to 90% of their policy went out the door. Just get it fixed. But the same result happened. We were able to deploy in an hour versus we sat there for a week arguing. Well, they were trying to probably prevent something from happening. But it's like, why not just use the instant deployment path all the time? Because it works. That's where you could overcomplicate things, putting a lot of toil in front of getting their idea out there.

Ben Edelstein:
And what is WAM, it's a acronym you introduce in the talk?

Ravi Lachhman:
Oh, yeah. Yeah. WAM, it was more of, wham! It was more not an ac... Maybe I capitalized it in the presentation.

Ben Edelstein:
Okay.

Ravi Lachhman:
It was me trying to say, there's a lot of decision that have to be made when you're deploying. Why is there a lot of decisions? I remember reading in a textbook, and if anybody gets a copy of the presentation, I had to search Google near and far to find this. It was just like picture. It's called, How many regulations go into a hamburger? It was from the 80s, so it was like 41,000 regulations. It was a picture of a hamburger and it's like, there's 41,000 regulations that go into this hamburger. It's from the transportation to how the raw materials, how it's cooked, to how it's sold. There's a lot of regulations and rules to follow. And that wham, the next slide. Wham! Okay, we're not talking about food here at Dev. I got away with it by using Hershey Kisses and the hamburger. I don't think I'd get away with it anymore if I keep talking about it.
It was basically a very lengthy CI/CD... Or not CI/CD, but a very, very lengthy deployment flow. You have to follow a branch and then you have to get test it, and then you have to document it, then you have to make configuration for the next environment, and you have to work for QA, then you have to take the feedback to QA. And it went on and on and on. This is not that different than what a lot of people go through. There's multiple environments, there's multiple levels of rigor. As you increase, there's the subjective and the objective you're dealing with. And that was the wham part. How are there 41,000 regulations to a hamburger? You don't know that. As an engineer you're like, how are there 100 steps to go through?
I know, long-winded answer here. I always remember, that first TSA project I was on was so influential to everything I did, because that was where I started as full-time employee. And I remember their change, they had a Word document that I remember the number had 158 steps. 1-5-8 was to get it to publish something in that portal application we were building. It's like, yeah, that was a lot. My machine, there was three steps. Why is there 158? Crash my machine opening up Word document, how big it was. But yeah, that's where complexity can certainly come in.

Ben Edelstein:
And talk to me about ownership in developer experience, and the concept of who owns things, who owns everything. And in the presentation you relate that to Conway's Law. Talk to me about that subject.

Ravi Lachhman:
Yeah. I'll give a base explanation of developer experience. Developer experience, it's like user experience. Its acronym is DX. But that's similar to UX, user experience. Except the user is a developer, so the feelings of how he or she or the developer as they interact with the systems in place and the tooling, and how they get their job done is their general feelings and consensus and ability to get their job done. So the next part is, who owns that experience? I came up with four separate personas, because Conway's Law is this organizational concept that system design follows communication pattern. So if you have an operations team and development team, they both develop systems based on the way they like to communicate. And so it's, who owns your developer experience? Is it a DevOps engineer?
Is that part of your DevOps or dev toolings team, or platform engineer team? Is it the system engineer? The person who controls where the stuff goes eventually. Does the developer own it themselves, does he or she have full reign? And it talks about, hey, it was more of an open ended question of getting folks to think no one person will technically own it because each one of them brings in a different piece of expertise. A lot of folks I interact with are DevOps engineers, and their internal customers are developers. And so there might be a little bit of relationship between, hey, the developers are the internal customers. And so if your internal customers happy, your external customers would be delighted also because they're able to produce better work. Or versus, a lot of people putting up these barriers into place, we can't change, makes stuff go wrong. If it's running right, why change it? There might be more barriers if a system engineer had to create that developer experience right. And so each one has pros and cons to it.

Ben Edelstein:
And curious to understand more about, what is DORA? That's another concept you introduced in the presentation, and I hadn't heard of that before. So curious what that means, and what's the impact on developer experience?

Ravi Lachhman:
Yeah. DORA is this DevOps research and assessment. And I think they got purchased by Google, I can't remember. But they were a group of very talented folks that produced a bunch of research. There's this concept of, you might hear it, the four DORA metrics. It's a way of measuring engineering efficiency. In the olden days, let's say, it might be how many lines of code did someone write to measure developer efficiency or output? How many commits did you make from GitHub? You better be green in every box. Those are asinine vanity metrics, they don't really show much. If you want me to produce more commits, I'll commit every sentence, every period. Every line I make, I'll have a dark green bubble, or square in GitHub, if that's what gets me my bonus. And so DORA I came up with, the authors came up with four metrics.
And those metrics to that I think really sufficiently measure efficiency would be... Deployment frequency is one of them. Also, there's a couple others that are telling. One would be lead time to change, so going from code to production. Lead time to change is outside the developer's hands. A lot of this would be how burdensome of your processes are. Time to restore. If there was an issue, how long did it take you to restore previous functionality? And also, change failure rate. When you try to go to prod, how often did it fail? That will help measure overall engineering efficiency. And again, no one person owns that. Does the developer own all the tests, or do they own the deployment making some? No, there's multiple people involved.
But learning where these items become bottlenecks, you can help tweak and improve and optimize and bring up the base level for everybody. Going back to, where does that go into developer experience? That's usually what people are measured against. Hey, we're all working towards delivering this feature. But also, there's also other objective measures to look at. A good developer experience will increase those metrics intrinsically across the [inaudible].

Ben Edelstein:
Well, Ravi, it's been great having you on the podcast. Really enjoyed learning a bit more about Harness and some of these concepts around developer experience. We're going to post a link to some of your social profiles for anyone that wants to follow Ravi to keep up with what he's working on. And we'll post a link to Harness as well. Yeah, thanks so much Ravi.

Ravi Lachhman:
Cool. Thanks for having me, Ben.

Emily:
Hey, this is Emily, one of the producers for PodRocket. I'm so glad you're enjoying this episode. You probably hear this from lots of other podcasts, but we really do appreciate our listeners. Without you there would be no podcasts. And because of that, it would really help if you could follow us on Apple Podcasts so we can continue to bring you conversations with great devs like Even You and Rich Harris. In return, we'll send you some awesome PodRocket stickers. So check out the show notes on this episode and follow the link to claim your stickers as a small thanks for following us on Apple Podcast.