Sheng Wu: The benefits of the Apache Foundation is they have over 300 top level project and have dozens of incubator projects. They are looking for cooperation. Like, people support us. People support SkyWalking. They are building connections between the projects. Eric Anderson: This is Contributor, a podcast telling the stories behind the best open source projects and the communities that make them. I'm Eric Anderson. Well, I'm excited that today we have Sheng Wu joining us on Contributor, to talk about SkyWalking, maybe to get us started, Sheng, as we customarily do, tell us what SkyWalking is. Sheng Wu: I created SkyWalking five or six years ago, and it is a very unique open-source project. It's built a commercial level, open source APM system. For APM, it means Application Performance Monitoring or Measurement. It's to measure how your system performs, especially for today. We are focusing on the coordinative distributed system. Eric Anderson: Fantastic. And APM is something that we think of often associated with New Relic. They invented the space, but I don't think we've seen a lot of open source APM. Sheng Wu: Yeah. Because the APM usually extremely complex. So the open source community is usually lack of people, lack of the contributor to build such kinds of complex system. But at the same side, we can see from the Kubernetes to the service, and the distribute team. All the words are building more complex system than before. Eric Anderson: Yeah, certainly. Well, take us back to the very beginning. Why is it you wanted to get into APM? Sheng Wu: The SkyWalking came from a chaos. A chaos from the real world. It's back to eight years ago, 2012. I was the first group to building the first country wide used telecom system, the first telecom system used for whole China. So in that system, they set up a vendor team to build a distributed system that based on the SLA. That basically not like today's microservice, but still is very distributed. And you can see eight vendor for one system. There will be chaos about, we blame each other. The system always fails with an error, and the stack came from to upstream to every downstream system. Sheng Wu: Everyone says, this is not my fault. So this idea comes out and we are thinking about, we need a solution. So that SkyWalking came out. But add back to that day, SkyWalking is only a training system, not a real system deploy on the product environments. The training system is for ... We have many new developers, and the distributed system is very new to them, considering they're using the J2E architecture before. So we hope that SkyWalking could loop everyone in about how complex the system could be, how the error or unexpected network issue could happen, and affect the whole distributed system. So that's the first idea we build that, kind of, the SkyWalking. Eric Anderson: So you're on this complex product, everybody's pointing fingers and blaming each other for problems, and you realize you need the source of truth for what's happening in the network and the application. Was this application you were working on unusual in any way, or this would presumably be a problem everybody has? Sheng Wu: I think at that moment the people are nearly moving into these problems in many ways. This is the first time people are get touched to these distributed systems and application performance. Because in the old day, we have many efficient performance tours to focusing on the traditional architecture. But today all of the same scores into the network, goes into the third party system, goes into the past cloud. Everything goes into the distribute. So new tools to resolve the new problem. Eric Anderson: And as you went about to resolve this problem, you began working on SkyWalking. Did you do that at your current work or do you leave your job? How do you go about building SkyWalking? Sheng Wu: That's because I am working on my free time. I'm nearly, only need to work three or four hours per day, in the daytime job. So I have a lot of flexible time to do other works. And then I'm beginning to write the first prototype of the SkyWalking came back to six years ago. Eric Anderson: Fantastic. So this was just you, by yourself, at the beginning of SkyWalking. Is that right? Sheng Wu: There were two of us. One junior developer and me. We are working on the very beginning prototype. Eric Anderson: How exciting. And did you have a pretty good idea of exactly what you were going to build at the beginning or was there an evolution to SkyWalking? No, okay. Sheng Wu: No, no. There's no idea what we are going to do today. Because back the day, we built very prototype system. We can still found what we did in the very beginning, based on the GitHub history. It's super different, and it's not that commercial level. We don't have the age instrument agent, we didn't know because the service mesh is not there. We don't know the Kubernetes. So basically that's another traditional tools based on big data. But with the time, we get rid of all of the traditional big data system, because the complexity too, because we need to maintain, to monitor the system easier than the targeted system. Eric Anderson: So you've been working on this a while. Does it predate Kubernetes and service mesh, or just you weren't aware of them at the time? Sheng Wu: I'm aware at the very beginning, but I set up the SkyWalking to walk on there, to take several years because the Kubernetes or the service mesh, basically for the developer are very new. And they take time to put such kind of architecture into the production environments. And the APM is designed for the productive environment mostly, so it's only focusing what people are using today based on. At beginning, people are only use SkyWalking out of VM, and then they ask for the Docker container. And then Kubernetes, and then move to the service mesh. Eric Anderson: Got it. So SkyWalking has evolved with all the architectures? Sheng Wu: Yeah. It's changed and refracted, rebuild it based on the architecture changed. Eric Anderson: Very good. After you build this, do you take it back to your work and on the project that you were struggling with? How do you get your first users as you're building SkyWalking? Sheng Wu: Actually, that's a totally accident, actually. Maybe just supposed to project down the GitHub. And for the first idea, we are not doing anything related to the open source marketing, to tell people what we are doing. We just put it there and the rest on blogs, and we post there. And we just continue our works. Because basically, this is a training project and we are doing that for fun. So this is not some urgent plan we need to focus on and to make sure it must be a success. So built this totally because it's maybe a lack of such kind of tools in the world. So people are finding this on GitHub. Eric Anderson: Okay, yeah. So you casually just put the code up there, explaining the code blog post and move on. And then people discover this and begin using it. Sheng Wu: Yeah. People take time. I think they take two or two and half years to get first group of people, get the system on their product testing environment. The first group of people using the product back to three years ago. 2017. Eric Anderson: Are you ecstatic that people are using this project? Sheng Wu: We are excited to have users, and the first time get feedback, how they are feeling about deploying the SkyWalking in their environment. And what they want the project to do. And we are always focusing onto looking for the contributors, actually rather than the real only end users, because we won't know that's how complex the APM could be. So we need to build up a very huge and very active contributor team to make all the [inaudible 00:09:17]. Eric Anderson: How do you attract contributors versus users? Sheng Wu: We are focusing on to answer questions related to development and the contributions. If the user are focusing on just about how to set this things up, we just spend less time to point them to documents rather than explain the questions, or we don't help users to find out what's the issue, what's the log says. And we are thinking they are user's responsibility. Actually, this is not a commercial support. Eric Anderson: Yeah. You're on your own if you're not going to be contributing code. Sheng Wu: Yeah. If you are contributing, then we are work as a team. Otherwise, mostly you are on your own. We can provide very limited help. Eric Anderson: No, makes sense. Great. And at some point, Kubernetes and service mesh come into the picture. When was that? At this point or earlier, or later? Sheng Wu: I think a little later than this point. Actually we were adopting the service mesh and we build nearly 50% of the system. Eric Anderson: Wow. Sheng Wu: We write the codes for making the SkyWalking could work perfectly in agent base, the system, or mesh based system, or at the same time. Eric Anderson: Very good. We should spend more time on that. But also when did Apache or CNCF get involved? Sheng Wu: I actually got touch with nearly Apache and the CNCF for the first time, at the very beginning, about three years ago. And back the day, I was at Huawei, trying to make the project and moving into some foundation because the mostly is about interest conflicts, because this is a personal project and I cannot continue working on that as a employee of Huawei. So I need to send them somewhere. And then the company would allow me to continue the work because it's belongs to the foundation. Eric Anderson: Good idea. Sheng Wu: So I got connected with CNCF, and they proposed to have a presentation, but one side is not China types of friendly. So I have to do the presentation like midnight. At same time, Apache is very connected with me because at the beginning I was involved in the Zipkin. So I have a lot of open source friends there and some of them have the capability to mentor the project. So we are choosing the easiest way. And at same time we recognized the Apache is more focus on the community rather than to vendor based. It's open source ecosystem. So we think that's special, be the way we like, because basically at that moment, we have 10 to 20 contributors. And they are from 10 vendors. So basically we have very individuals contribute to module and that's how Apache works. Eric Anderson: Go it. So you have a bit of a bake-off here. CNCF or Apache, they both could take you. And for a variety of reasons, Apache seemed like the better fit. Not only do they schedule things at a reasonable time, but the project is more contributor based, less vendor based, less corporate. Very good. And Apache has this incubator. I think they both do. They have an incubator model, right? Where you start and then you graduate. How does that work? Sheng Wu: Actually, the Apache most incubator process is measure your IP issue, branding issue, logo. And then it's mostly about building community. The Apache measures you need to set up a diversity community from people, you need to contribute from different companies. It's better to have contributor from different countries. But for SkyWalking, that's not a very hard requirement. We already have that when we joined the Apache Foundation. But it takes some time to make the SkyWalking release follow the Apache requirements. Eric Anderson: It's like taking a company public. You have to get all the ducks in the row and get everything structured the right way. Sheng Wu: Yeah, you need to take all of the items that they clicked to check, and you need to pass some exams. It's just take some time. At beginning we were not planning how or when should graduate. It just take 15 months and people begin to ask why you are still in the incubator. Because no one have, not interested in finish all of that paperwork. We was thinking, we are good. We are stay in the foundation and everything works. And we just keep the things where it is. But the incubator's people, they are asking, "Why are you still here? You have enough community, you should move on. You should be on your own." So we take three or four months to finish all of the paperwork, to make the project graduate. Eric Anderson: And what else does Apache provide? Did you find more contributors? I know that was a big part of what you were looking for before. Sheng Wu: Yeah. It's quite interesting about the Apache Foundation, that they are not promoting project much. But the benefits of the Apache Foundation is that they have over 300 top level project, and they have dozens of incubated projects. They are looking for cooperation. So once you join the foundation, people are showing up to set up the cooperation between the projects. We have support from different projects. And other project contributors are connected with us to set up, because they need to monitor it too. They need their users to keep their eyes on the system, how it works. So they are contributes plug ins and bridge with our system, and the two to make sure they are covered. We have 50 contributors before we joined the incubator. And then when we graduate, we are nearly 200. Eric Anderson: Wow. Got it. Okay. So you've been Apache incubating, and then graduating. You rewrote the project. And going back to Kubernetes and service mesh, you rewrote half of it to accommodate service mesh during this time? During the Apache incubation time? Sheng Wu: Yeah. Yeah. Eric Anderson: Great. And then how big a deal is graduation? Is there some boost to the project at graduation? Sheng Wu: It's actually the best part of the graduation from my understanding is you finally could release a project on your own. In the incubator, you need to approve from the incubator to approve your release every time. That takes time. And now we could make the release easier and we can set up the sub-project for the SkyWalking, because today we have over 10 sub-projects already. And we are building new sub-projects for the whole ecosystem. Eric Anderson: And then personally, you were at high rate some of the time, and then you left that, and eventually found your way to Tetrate. Is that right? Sheng Wu: Yeah. Yeah. SkyWalking is why I joined Tetrate. Eric Anderson: Yes. Now you can work on Tetrate for work. I'm also noting that you've since done a lot of other open source work in the Apache Foundation. Tell us about that. Sheng Wu: Yeah. I was called creator a protect called ShardingSphere, and that is a proxy level solution for My Circle cluster. So if you need dozens of My Circle instance and you want to access them, you have one My Circle database, and that's your solution. I build that project, but I left. I left four years ago and I only support that project from the idea of marketing a project, how to build your community, to support them, to have connection, to join the incubator. And now they graduated too. Also at the same time, I would involve with the Apache APISIX, open Rust-based gateway solution, like another version of Kong. So they're interesting people, originally from Open Rust community. And they set up a company. APISIX is one of their projects. Sheng Wu: So I involved with them because the SkyWalking has interested in monitoring Open Rust and Ajax. Because the other side was a part of the Tetrate. We are working on Envoy, but still the nginx is well used. So I want to involve. So I would write first version of the Open Rust plugin to monitoring all the traffic goals through the nginx. Also, I may involve some other budget projects because of something like that [inaudible 00:18:33]. Like the world widely use to a web visualization tool kits. And incubator project that opens scheduler to distribute version Airflow. So they are all very cool project. I just spend some the time to support them in the foundation and involve their... like people support us. People support SkyWalking. They are building connection between the projects. Eric Anderson: Yeah. That seems to be some of the Apache way. I was involved in the Apache beam project, and we had people from other projects that would show up and help us. Both from governance, but also just writing code. So I imagine this is how you give back to the Apache community. Sheng Wu: Yeah, yeah. That's how the things works. So Apache is recognize such kinds of contribution because I was be selected as a member of the foundation, like 2020, April. And I just take nearly two years after my first time join to Apache, and became a member of the foundation. Eric Anderson: Take us back to the SkyWalking. Where's the project at today? It seems to me that it's a bit of a hidden gem, I guess. It doesn't get a lot of publicity, but it's a fantastic solution. Tell us, what are your aspirations for the project? Do you hope more people use it or are you just looking for contributors still? Sheng Wu: I think it basically the contributor is contributor team is very powerful. And I said, because of the short term of the project really becoming such powerful. I just want to say the project is powerful in these two years. So, we have already donated all of the users inside China. So if you ask the people in China, they build what kind of open source APM system, you most likely, you only gets the SkyWalking as the answer. You have the only cloud, the Tencent cloud to the already support or the SkyWalking on the clouds environment. Sheng Wu: So also at the same time, from 2020, and we can see the user group expanding in the world. The user from India, from Europe and from US. They are showing up to adopt the system. And the same time, I think it takes time because the replace APM system is a big thing. It's a big decision you need to make, because I have heard about people are, you mean world, they're using SkyWalking to replace their work at Dynatrace. So people are highly measures these kinds of solutions because they can provide more chance to do the integration, because you need to build a reaction system based on the data of the APM provided. Eric Anderson: And the people that use SkyWalking today, some of them, as you pointed out, had Dynatrace or New Relic, and they left it. Are they also these Kubernetes and service mesh people? Sheng Wu: I think most user are still stay at the very beginning of the Kubernetes. They have widely used VM environment and they are adopting the Kubernetes with the time. So basically we only get connected with real open source mash users. They basically came from China and they are adopting the new technology very quickly. Eric Anderson: Very good. Where does the project go from here now? Do you continue to spend a lot of time with it? Does it still have a big roadmap ahead of it? Sheng Wu: Yeah. I think that will be a big map because in the previous several years, we begin with the distribute tracing at day one. And then we have service mesh. And last year we finished all of the adoption of the metric system, including the Prometheus, the open telemetry metrics. And other system is on the way. The topics is on the way. And the same time for this year and the incoming next 8.4 release, we are going to adopt log in. So they will be all of the three important parts of the observability. And then we are going to introduce a concept called events. So I will change to the targeted system. There should be triggered by it. So we are going to introduce that so user can set up the deploy tools like the Kubernetes or helm or operator to coordinate with the APM system. Sheng Wu: Then you can understand why this metrics change happens. It could based on your new code base or the traffic off site. But also could be both. So you need tool or you need the integration solution to tell me what is really going on. Also, we have a very big step. We are going to move. Because of today, it's very hard about Elastic changed the Elasticsearch's license from the Apache 2.0 to SSPL. Sheng Wu: And basically SSPL have some limitation about how user use that. And from Apache's perspective, we don't like that much. We could accept and we understand why this is happening. And also at same time we are provide the public goods solutions. So we need some better way. So today SkyWalking could use My Circle to [inaudible 00:24:09] the inference, to be the universal search as a storage option. But we are thinking about more. We are going to release the design in the next months about a native SkyWalking database. Eric Anderson: A new database, just for SkyWalking? Sheng Wu: Yeah. Your new database for the SkyWalking specifically, and partly for other APM system that will be observability focused, time series database. But don't like the before. They are for mostly inference DB or some some time series database. They're focusing on the metrics. So we are focusing on the metrics, logging and the trace. Sheng Wu: So basically it's a hybrid database that we build them in one solution and focus on the SkyWalking use case. And the people can have a very efficiency and same time, very powerful database. Because we are focused on time series and we need a huge number of data, not like metrics. The trays and logs that they decide is very huge. And as time, we don't need some concept like transaction. Most device provide that, but we don't like that. So we can have better performance. Eric Anderson: Actually, say a few more words about that because I think it's an interesting situation in the open source community or open source world. So how does the Apache Foundation and how do you feel about this Elasticsearch change? It sounds like what you're going through, a lot of projects at Apache might have to do, if they have a dependency on Elastic. Is that right? Sheng Wu: Yeah. So basically because of the change to SSPL, we cannot utterly depend on them as a solution. That not a discussable thing. Because of the Apache, we already determined that SSPL is a catalog X license. So no project could depends on that. We could choose another option. That's the maximum we could do. And also we cannot redistribute their client side to license, unless they change it to Apache 2.0 in the future. Eric Anderson: And to be clear, you're okay today, because you can rely on the existing code base. But you couldn't continue to get the upstream changes. Sheng Wu: Yeah, yeah. Basically is that if they finish their side of the work, they are still distribute outside the client side as Apache 2.0 license. We can still involve their driver into our distribution, but in the whole project perspective, we are looking for other options. Eric Anderson: And just to round out the conversation, did you consider the AWS fork of Elasticsearch, which I imagine is permissibly licensed? Sheng Wu: Yeah. So basically we are also discussing this to people involve in AWS has about this kind of things. And once they have finished all of their preparation and they're going to release the whole package of the version, or fork version of Elasticsearch. I'm pretty sure Elasticsearch is not a name for the final distribution because Elastic have the trademark, and we respect that. But at the same time, because they are used to licensing Apache 2.0, everyone have the power to fork that version and continue to maintain the work still on Apache 2.0. Eric Anderson: Very good. Great. So if you've peaked the interest of our listeners, what can they do to help out in the projects? Or what are you looking for from new users or contributors to SkyWalking? Sheng Wu: Today we are looking forward to hear more worse, more of the information front end user, all over the world, because we can see from the Google analyze. We have access nearly all of the country, in the world. Every day. But we are lack of the feedback, how they look like in their environment. We receive the issue from time to time, but we are expecting people can share what they experience, what their use case looks like on the blogs and the video, on the podcast, anything, so we can know how it's look like. Eric Anderson: Totally. It almost feels like it's somebody's good will and obligation maybe to give feedback, to open source code. You're given the code for free. You should tell people how it's going. Sheng Wu: Yeah. You at least to tell what is things going on because we are exciting to see. There's a Poland company, [Rep Debug 00:28:57]. They are also building commercial solution on the SkyWalking already, and they are making marketing things. And we are exciting to see more vendors behind this project. Eric Anderson: Fantastic. Sheng, thank you so much for coming on the show. SkyWalking is very exciting. I can't believe you've put so many years of work into this, and then given it all to the world for us to enjoy. Any last things you wanted to mention before we wrap up? Sheng Wu: Last thing is about how we named the project. Eric Anderson: Oh, yes. Let's do it. Sheng Wu: I think that's the same people are also thinking about SkyWalking as a turn from the SkyWalker, that's our original source of the project. But actually it's not. Eric Anderson: Okay. Sheng Wu: SkyWalking is from the skywalk because back to the day, I want to build the project. That was my first time to go to the US, so I go to the Grand Canyon, the national park. And there's a view on top of the mountain. And that view is built by the Chinese company, Glass Bridge. But they name it Skywalk. So back to want to create the project, I went there. So I chose name, SkyWalking. That's the place that I create the project. Also at the same time, I know there is people sometimes called X sports, when people are climbing and running on top of the building. They call that skywalk. So when you see a distribute system, you want a very high view and the very challenges solution to find what is going on. So I think this is also feeds to the project. Eric Anderson: That's a good name. And I love this story about the Grand Canyon. Sheng Wu: Yeah. And the other tricky things about project back today, because it's a personal project. So I inject some personal interest in the project's names. So if you'll see the first letter of the sky and the walking is the same first letter of my name. Eric Anderson: Very good. Sheng Wu: So that's just small tricky things I used. Eric Anderson: Well, any other Easter eggs or hidden messages, Sheng? Sheng Wu: Sheng is something, is the idea of things are growing very well. You're growing fast. That's the thing behind this name. Eric Anderson: Good. Thank you very much. Congratulations on the progress of the project, and appreciate you coming on and tell us the story. Sheng Wu: Thank you. Thank you for setting this up. Eric Anderson: You can find today's show notes and past episodes at contributor.fyi. Until next time, I'm Eric Anderson and this has been Contributor.