Alex Scammon I sort of wanted this technology protected to some degree. I also wanted it not to be purely on the shoulders of G-Research to just hire everybody and keep it going. I wanted it to be a community effort. I wanted other people to invest alongside us so that there were more voices at the table and there were more pocketbooks that were contributing. Jay Faulkner Hi, this is the G-Research Open Source Software Outreach Podcast, the Gross Out Podcast. I'm your host, Jay Faulkner, and today we've got a reprise of the very first guest, my grand boss, the co-CEO of all the things that are run around here. We've got Alex Scammon. So thanks for joining us again, Alex. Why don't you refresh our listeners on what you do over here at G-Research Open Source Software and how much it involves bossing all of us around? Alex Scammon It's very nice to be here. Thank you very much. All I do all day is just, I don't know, yell at people and accuse them of things and tell them to work harder and faster and threaten their jobs with replacing them with the AI, you know, the normal, the usual. Yeah, it's like that. For the general public, I'd probably tell them that I lead a team of around 30 mostly engineers who contribute to open source of all kinds for all the things that G-Research is interested in, which means that we get our hands dirty with a whole bunch of things and I try to do the shit strategically and help get the bright outcomes for G-Research over a very long period of time. So that's me. Jay Faulkner If you want to know more about specifically like what gross does and like what our general day today is, that's kind of what our whole episode one was about. So we're not going to go too deep into that here. You can go listen to that. Maybe appreciate that maybe we've gotten better at podcasting over the last couple of years. If it was the first one, but I wanted to talk a little bit about sustainability and open source, right, like that's sort of the reason why gross came about in the first place was sort of this idea that if you're a company and you're relying on this critical piece of infrastructure. That means you're also invested in making sure that critical piece of infrastructure keeps moving right. And it kind of feels like, and maybe AI is just a convenient blame for this, but it feels like as we're going more into the age of AI that like investments in those areas in like those critical pieces of technology or maybe getting pulled back some is that is that a pattern you've been seeing. Alex Scammon Um, actually, I don't think so. I see investments shifting, and people trying to catch up and trying to figure out where the puck is moving to. I don't think it's out now retreat from open source but what I have seen more often is are some negative impacts on open source. I don't think that the money situation is more or less than it ever has been. I think it's just sort of shifting a little. But I have to talk about the other impacts of AI. If you'd like that, that can go very far Jay Faulkner Please. Alex Scammon Yeah, I mean, the, the central version there is that. A lot of maintainers that of projects that we care about have been inundated with not necessarily AI swap just potentially drive by PRs and they may be fine PRs but it's not just enough to give a good PRs. To a maintainer, there's a communication that happens after that, and you have to talk about what it is and maybe lobby for why you want it over the line why should a maintainer take on a new chunk of code just because you wrote it. And actually that theme may come up again later. But we've actually seen a bunch of maintainers retreat from open source, even though their software is open source. They don't want to engage with the community because they're never sure who is actually a human behind some of these PRs. So then for us who are coming to the table, honestly and earnestly as humans wanted to contribute. We're not getting the attention of maintainers that that we need, even though we're there as humans, the maintainers don't know who's human and who's not. And one of the ramifications for us has been we've redoubled our efforts to engage socially in real human spaces because a large part of our ability to get things over the line depends on those social interactions and we can't make those social interactions necessarily in the the town square of GitHub anymore, we have to go outside of that, make the connections in real time and then come back and do the business with the code. Jay Faulkner Yeah, it's interesting kind of it brings to mind like the invisible work series, a blog post I wrote for G-Research where like I feel like that the use of AI as like this is what my maintainer hat on you know has accelerated our ability to get stuff done. But it's increasing that invisible work that goes around it the making sure things are secure the performing code reviews the keeping CI and CD healthy, replacing an emphasis on who the people are right like with the ex Z attack. And I thought like to that and now the AI revolution is sort of bringing an entirely different angle to that. I mean I've even had serious conversations with other maintainers about whether or not we need to in first some sort of proof of life requirement for core review now, because how do you know if someone's an actual person or one person instead of an amalgamation of person or or whatever the reality of the Internet is breaking. Alex Scammon Right, because even the people who are the people are we're all using AI to generate a bunch of pull requests for views and and comments response to comments. You know, shamelessly use AI to generate the comments for me even if I have the idea myself it'll still read like Claude wrote it because it was much faster for me to write it with Claude than it would be to get out those ideas myself so just like at the end of the day, you need to be fact of the database like this I'll have Claude write that but it means that with somebody looking at my PR comment, my go this is just another bot when when it's actually me so like you don't know it's very hard to tell. A good example of the invisible work that we've done recently was in the Delta Lake community, but we've been trying to get some work in the in encryption over the line. And one of the maintainers over there really sort of put it to us like, are you just going to throw a whole bunch of code over the wall and then walk away. I don't want to merge this if that's what you want to do and totally fair reasonable rational thing it's just more work for him to maintain. So, I ended up going to a Databricks conference to show up in person, we send a bunch of people to more emphatically assert that no we're here for the long haul we're using this product we care about it, we're going to be around and, you know, those kinds of invisible works actually showing up to a conference that's maybe thousands of miles away from you is now making a huge difference in the technical abilities of our team to actually get this code over the line. Jay Faulkner So that's just a recent example. You know that's actually that's actually interesting and, and you know, most open source maintainers the entire thing is often viewed as a dark cloud coming over as because we get hit with a lot of the impacts but you just kind of pointed out something to me that maybe is a little bit of a light there right that like we were moving in a direction and open source where it was becoming increasingly transactional right where it's like, I give you code I give you maintenance I give you 20 bucks a month. And my code merges, and now we're sort of getting back to some of the community roots of it where you have to create a human connection with someone to build human trust, because really, we've gotten to the point in society where you can't do that successfully over the Internet or at least without the point in the way that I know Sally, you know Sally, so you can know that I am the things that I say I am which is a weird problem to have. But I guess it's kind of always been looming over us on the Internet of you never know who's on the other end of that chat box it's just now a professional concern and not just a, you know, paranoia about concern for paranoid parents right Alex Scammon So, you know, there's real downsides to the proof of life thing because you know we can send people to a conference it's not available to a whole bunch of people. You know that to some degree cuts out the ability of a certain class of people to be able to communicate with other people who are doing open source of like it's, it's not perfect binding. But, you know that that's just where I've gotten to in this in this conversation myself. Over the past couple months that we're going to have to redouble our efforts in actual making so actually making social connections like that more than anything right now the code is cheaper and cheaper the social connections are worth more and more. And that's how I look at it. Jay Faulkner And honestly like, and I don't want to go down this rabbit hole because we've got better things talk about other problems that we're actually solving rather than ones that we're just kind of grousing about and don't have a good solution for But I do worry a little bit about long term with the open source community of like if we're going to have a schism along an AI fault line, because there are a lot of people in the open source community a lot of long time people who I respect to have taken on a policy of And so I think if I see that there's a clod in this repo I'm going to you turn if I see the day as in use and stuff like that and so I actually think it's really insightful to think we need to rebuild those human relationships those human concerns And then basically let let our humanity win over the the AI noise pro anti whatever right like it doesn't matter what tools in your toolbox it matters that we're two people and we can go you know share coffee together and talk about the problems But you have problems to solve right. Alex Scammon It will be fine Jay like everyone will come around with everyone will love it I'm sure. Jay Faulkner The thing is those that you know we we are in the business to solve problems right like we don't like to think of it that way but like from open source and and one of the projects that we worked on was was getting into a little bit of crisis and that's Apache Arrow. Alex Scammon Yeah Jay Faulkner I believe and so maybe talk let's let's give a little bit of background on like what Apache Arrow is and maybe why gross in particular was interested in it before we we delve too much into the story. Alex Scammon Yeah, it's a column they're in memory data store format that allows you to allows different applications to access the same memory and interrupt rates and it's all wonderful and kumbaya says all the applications and it's, it speeds up a bunch of the tools that you and I may use to do data science and machine learning And it's sort of really underpins a lot of the AI technology that we use today. It goes hand in hand with Parquet very often Parquet sort of like the on disk format and Arrows this is the in memory format. And so we are heavily invested in it, whether we like it or not. And we were always on the sidelines trying to make things better like one of the very first things that we did in gross was get nanosecond time sounds back into Parquet V two, which required a whole lot of plumbing And so we've always been contributing to this particular area just because it is so sort of fundamental to what we do it is the basic representation of data throughout so many systems at G-Research and that in the world. So that's so that's what it is. You mentioned the, the, the problems that the community was having. Essentially, the story is this that rewind to when that whole ecosystem is being created. Wes McKinney had started this thing called Ursula labs, which was not quite an on profit, but it was this lab that he had set up to push this ecosystem forward. And a bunch of people who were into Arrow and Parquet and contributing to all that stuff and that Ursula labs was funded by people like us G-Research we had a small sponsorship with them or to segment some other companies had invested in this whole ecosystem benefit in it. Great, brilliant. At some point, Wes and Darren and Josh and some others felt that it was time to turn this into something bigger than it was to use the technologies that they had been building in their own. And they actually built a product on top of it. And they formed Voltron, Voltron Data and they started working on a product that had this whole open source component they brought all of the practitioners from Ursula labs into the world. And everything was happy they had a ton of funding. They were pushing forward with Arrow and Parquet seemed great until the point where Voltron Data started suffering from fiscal financial problems I don't even know all the details behind what ended up happening which is Voltron Data folding. And at that point, there are a whole bunch of open source people who are essentially the community, the main the release managers, Voltron on all the main contributors to the project are all at Voltron there, all centered there. And they're all going to be scattered to the winds and maybe not have jobs that allow them to work anymore on our or Parquet. And we still depend on it. It's not like that our dependency on it goes away. And I was worried enough that I started chatting with Wes and some other people. And we put together a response which was to put together this little nonprofit called Arctos Alliance that is hopefully free of corporate problems or going to ways or whatever that that can be people engineers who are employed specifically to help this particular community, who aren't necessarily the, the, the whims of the stock market and things like that. So that's the story in a nutshell. Jay Faulkner So, here's a lie that I've had right like when I saw we were doing this and it's good and honestly it as an open source maintainer regardless if I was involved with this or not like seeing things like this, it gives you a little bit of inspiration right like there are people invested and they have check books and willing to make it happen, but why didn't we just hire all those folks. Why not bring them into GR-OSS why not make them all a part of us like what's the benefit in having that like intermediary like are we working with other companies here too. Alex Scammon I think for me, there were a couple of things one was that I felt clear that I didn't want there to be a company like this foundational technology at the whims of any specific company, including ours. We are still going to contribute to this. This technology it's not like we're going away. But I sort of wanted this, this technology protected to some degree. I also wanted it not to be purely on the shoulders of G-Research to just hire everybody and keep it going. I wanted it to be a community effort I wanted other people to invest alongside us so that it, there were more voices at the table and there were more pocketbooks that were contributing to this. Because it, it really shouldn't just be on any one company's shoulders it shouldn't be Voltron shouldn't be G-Research it shouldn't be. I guess two segment probably gave a lot of money at the beginning as my guests. Like it shouldn't be up to any one company to push such foundational technology along. That was my feeling and that's why I created it like this. Jay Faulkner I'm glad that was exactly what was first important you hit me with it at the end of the fairs as beautiful. The interesting thing to me with that is that I've seen like as an upstream developer in a place for like OpenStack is a pretty distinct community separate from the software like it's not as, you know, direct line from maintainer code there's there's a little bit of bureaucracy there and what I wanted to see is that we have a lot of contributors in OpenStack, who maybe that community can be used as a buffer to to keep away bad ideas to prevent, you know, like maybe too fast initiatives from one company spiraling the whole project because you have like this entity that can pump the breaks and almost like you want the project itself to be that entity you want the project itself to have an opinion a project itself to have the scope to draw the lines around what it's doing and what it's not doing rather than maybe someone who is invested in, you know, what are we doing next quarter what are we doing this next specific company project problem I'm trying to solve that that sort of encourages that sort of long term thinking is that is that, does that reflect your experience as well here? Alex Scammon Yeah, very much although I would take a step back and say that Arrow and Parquet are still Apache projects themselves, and so there is a project, you know, the PMCs and, and the whole structure that Apache has to do some of that work itself And we see Arctos as just a helping hand to that community, but still, to your point, we don't want any one company just to come in and be like, "Baa, we're going to set the table for everyone here" And like, you know, with four or five developers in Arctos we have the ability to drive a lot of development, we make a lot of decisions about what we decide to work on and the bugs that we want to clean up and the projects we want to address We could easily put our thumb on the scales of development and strategic direction if we wanted to, and if that was driven by one company and one company loan, that would not be great, I don't think, and yeah, so it's definitely to create a bulwark against that kind of relationship. Jay Faulkner People don't like talking about a lot, but I feel like building good software needs an adversarial nature to it at some degree, right? Like you need the idea of like two things pulling in opposite directions and that sort of helps you little things down I was actually talking to someone in ironic about this year, the day about how AI is enabled us to go faster, but sometimes that's actually a problem because sometimes the fact you're going slow, the fact that maybe you trim off the edges of the feature because it's too much code, it's too much scope. So if you end up with something in the end that's general purpose that's better because of that adversarial situation that's happened that you maybe wouldn't have gotten inside echo chamber be it human to human or human to AI echo chamber. Alex Scammon Yeah, one-hundred percent. There are a number of initiatives that we're taking up right now that think Rock who's one of the developers on Arctos, he's doing some work around tensor types and he's getting input from a variety of different sources And that's important that like left to left to their own devices if any one of those companies wanted to come in and sort of define that work, the output would look like it was tailor made for them, rather than something that would work for the entire community And because it's Rock who's independent and working on Arctos as opposed to, as opposed to working for a specific company, he gets the liberty to look at it sit back and look at all the inputs that are coming in, and hopefully define something that is better for the overall everyone, as opposed to just being able to serve one specific vector database or something like that. Jay Faulkner And that's actually the secret to like sustainable open source success right because in a way like we don't ever put it this way that much but with open source you're hoping for some type of economics of scale to come in, where, you know, if we have a large number of And that's the key to this is if if an entire technological ecosystem is resting on this pillar. It comes with it a more easy path for getting contributions it lowers the amount that each person's that's using it has to contribute so there is like A project value to getting other people in that project and this is actually a piece where we do so well that it's almost implicit we don't even think about it outright. But this right here is the piece that's missing it so many places I've worked at that's tried to embrace open And I couldn't see past scratching their own itch in the moment to see the opportunity that laid in front of them over the course of two, three years if they stuck to it. Alex Scammon Yeah, I mean it's really tempting for us to be hell bent on the financial applications that we desire because we're G-Research, but as we've gotten into this. And we've been at all of the geo applications that use Parquet and realizing that they needed a whole bunch of specific stuff for them. And then we're doing this, this life sciences grant application recently, and we said, Oh, who in life sciences uses this we don't know Oh my god we then looked it up through, you know, Google did and Claude and all and turns out Arrow and Parquet everywhere in that ecosystem to you know you just every vertical place you look Arrow is underpinning a whole bunch of the technologies And so like, to your point, this is so broad that there should be an economy of scale here for this particular foundational technology. Everyone uses our every everyone was able to give just a little bit. This would be really well funded and everything would really be the technology would be really good we'd have all sorts of fancy things on it. We can only dream of right now and it sounds funny because it's like, it's just a data format like whatever, Jay Faulkner Data is everything data everything. Data is everything data everything. Alex Scammon Yeah, Jay Faulkner Look at the data how data is stored and moved around a system you understand the system that's like a core tenant of systems architecture, the data is all that matters. Alex Scammon And there's, and there are still so many things that need to be implemented in Arrow and in Parquet that it's not even funny like if we got to everything that we want to get to. We can hire another like 5-10 people and then work for another 5-10 years and maybe maybe we'd get somewhere. Jay Faulkner I mean, let's just attach a call to action to this right if there's someone listening to this right now and you're a CTO at one of these companies that depends on this if you're startup is based on these technologies you know, if someone hears this and they're like, that's a piece I want to be a part of what they need to do in order to get involved and I'll mention that any links we mentioned here will make sure they make it into the description of the podcast and the YouTube video. Alex Scammon Thank you for that. ArctosAliance.org is where you can get more information. It's an open collective page I think it'll direct you there, where you can find more information you can find out how to get a hold of any of us. You can also have a little team on there, and you can donate money right then and there but probably best to actually just reach out to me Rock, Alinka, Nick Wes, you know, any one of us who are listed there and get into a conversation about what you use this technology for and why you want to support it. I think that's the best, first most thing to do. Jay Faulkner I'll say I'm looking at the page right now, and I will say that you can be a friend of Arctos for $5 so maybe that's not the message for the CTO. But, if you're Bob and you like what Apache Arrow is doing or Sally or whoever out there, go ahead and go in there and toss five bucks the way. You know, I promise you that we're going to be, like at least as me, a person at the endpoint of one of those dollars that get put into open source by GR-OSS. We take it really seriously and it's it's kind of a privilege almost to be one of the people get to work that much upstream. Alex Scammon And I'd say to the to the CTOs, like, don't ignore this. Like so many people are clearly just taking this part for granted. If you use Arrow Parquet throughout your systems really take note. This community was almost destroyed last year by, you know, the persititudes of corporate life. And the things that you depend on may have gone south, were it not for a bunch of those contributors being picked up by Arctos or Quansight or a bunch of companies who rallied to the cause. But it's a serious thing. It's, and a little goes a long way right now for the small amount of funding that we've gotten so far, we have, for example, there's a whole bunch of technical innovations that we'd be moving on around encryption and around tensor types that I mentioned before. But also just like simple community clean up community health, things like the 4600 issues that were just climbing inexorably. We're now, after a lot of patient work from Alinka, Nick, Rock, the team. We're down to like 2200 maybe less now. They've halved the number of issues in a period of three or four months. And that's the kind of thing that actually, yeah, sounds like a trivial, "Ahh, you closed some issues or whatever" but it actually makes a project welcoming and it makes it so that people who are coming to contribute are not overwhelmed or they know where to start. It looks inviting to actually start working with people. And to that end, because of the advocacy that we've been doing a bunch of people have been reaching out on LinkedIn and other places just wanting to offer to support with with contributions. So it's not merely the contributions that are making directly to Arctos but, but it is, it's sort of like a contribution match almost. The money that has gone into it is multiplying as we are accumulating more passionate community members, whether they're part of a company or not, to contribute directly to the cause, so a little goes a long way is my end message to the CTO people. Jay Faulkner So, then it kind of makes me wonder right like if this problem exists with Parquet or Arrow, are there any other areas of the open source ecosystem that we might have something brewing? Like I get to have you on a couple of years from now to talk about, you know, "Apache Target" or something I don't know. Alex Scammon So, I think, I think I'd have to default to like, there are a ton of things happening in the AI space that I think parts of that need to be open source commoditized you know harnesses and sandboxes and governance frameworks and I feel like there will be pieces there that emerge for us as, oh, this is foundational and we need to really invest in this for the future. I can't quite perceive of what those are yet. There are some other pieces that we're looking favorably upon these days, things like Polars is another thing that we've been putting a big bet on. Thy've been coming out with things like distributed Polars and we've worked with the Polars team on a number of core pieces. What else? I mean, the list of things we work on is endless so like you've asked me a question that just sort of like, opens a Pandora's box of ideas and I don't quite know where to start with any of them like, oh, we can start with security. Oh, no, no, wait, we should start with all of the PyTorch stuff. Wait, what we should really do is I mean I could, I could, so yeah. Jay Faulkner So what so what this really tells me in a way is that we kind of, are prepared to act like as the EMS if one of these projects falls over. We have some expertise in house. We have the experience of going through and trying to rally folks around the cause and stuff. Even if there's not a candidate, it's nice to know we're kind of invested in those things, that if there's a open source project we're invested in and something happens to it, we're going to be involved in the adjudication of that if it be migration, or like, you know, I mean in Eventlet, we helped drive some of the migration there along with Red Hat, you know, in some of these cases you know Arrow we pick it up and we make it keep going. Alex Scammon Yeah, I mean there's a bunch of different examples of when we've done that kind of thing. You know, whether those, all of those situations rise to the level of like we shall create a foundation to support it is another question but sometimes it's, it's like Sparkmagic we just cleaned up the repo to the place where we could invite everyone back to the site. We took over Console.net because the maintainer was didn't have time for it. Or, you know, one point etcd, sort of a foundational piece of Kubernetes was asking like hey, anyone wants to maintain this was a couple years ago where it had lost almost all of its maintainers. And just last week I think somebody from the inside of G-Research an engineer properly on the inside of G-Research not part of GR-OSS but, but one of the platform engineers there, how contributed to etcd for the first time they got a major piece of optimization and then we got etcd across the line. And, you know, those are places, that's just another example of where we will rally around the things that really mean a lot to us right now. And yeah, when the time is right when the foundation when the technology is sort of so core and foundational. And that's a problem so big in that community. Maybe we'll we'll start another foundation to, to raise some money to specifically target that. But yeah, like that that kind of support comes in all sorts of ways for us. So, that the answer? Jay Faulkner In some ways I wish it were as like as easy to attribute things to like, organizational reputation as they are individual reputation right like, in a way I kind of feel like at least I've worked here like for five years and I've I've seen quite a bit of stuff it's like, I feel like if someone from GR-OSS comes into your open source project, there should be a thing where we should have the goodwill at this point to promote good but I don't think generally in open source communities that ever gets attributed to organizations and always gets attributed to individual folks. Alex Scammon I think you're right, although a little by little people have started noticing who we are. Outside of this podcast we've not done enough I think to publicize the amazing work that the team has done. But little by little things get noticed and I do, I do think that over the last couple of years people have recognized us before our names are personal names have shown up. Lovely chat with somebody from Marantis the other day who was really, really happy to talk to them because I'd spoken to them two years ago when they got a new CTO who wanted to restart the Ospo. And in a way that is very similar to the GR-OSS way where we have a lot of engineers on our team, they hired all engineers and it seems to be working for them. And, you know, I think that the more that success stories like that can appear the more that people actually start knowing company names when people show up as a company, an open source team. So that open source team. Yeah, no, that's a, that's an engineering team. We should, we should pay attention to them. Jay Faulkner So, it's a weird thing because I almost think only negative gets gets attributed at the company level not as much positive it's Alex Scammon Yeah, Jay Faulkner and it's almost like some of the big names and open source that can almost be scary if they get involved in your project because there's so much larger than you might be but I think one of the things we have to give is that we aren't that giant, we are small, we are not going to bring an army of developers to take over your project. We're going to go, you know, be there be your, be your buddy and get it going. But, you know, it doesn't always work well. We do the best we can but you know there are bumps there, there are valleys there hills along the way. But we always figure out how to get from point A to point B so, you know, for anyone out there who's maybe in the middle right now who's in one of those valleys who's trying to climb one of those hills. Do you maybe have a gross story of one of those moments you can share with our listeners and sort of talk about how it got to there. I mean, honestly this whole podcast is kind of about a gross moment, but you know, we're going to we're going to try to get a sound bite anyway out of your up. Alex Scammon It was a situation that we really didn't understand because we come to the table with an open heart and an open mind and really just wanted to contribute positively to the world. And when we're unique that we are not trying to pimp some products and sell our thing to anybody so we more than most can can honestly say that like, we're just here to make the technology better because we're going to go use the technology. Yeah, we've got some biases but like, we're not trying to sell you a thing, which is very key to our pitch. The moment that jumps to mind where things went south and we didn't understand it was, we were trying to contribute to the Yunikorn project, Apache Yunikorn, and we were working with them for a long time. And it's funny that we were contributing to this project at all because we have our own Kubernetes scheduler Armada, but for specific reasons around Spark, we wanted to use the unicorn scheduler to schedule some things and we wanted to, I think, make some improvements in the UI or come up with their own UI and we wanted some hooks into Yunikorn so that we could get some hooks into some data to display into the UI. And we came to the community and we asked how we should do about this. And that it's been a couple years so it's the details a little hazy, but my recollection is that Rich proposed some approaches, they got down, we worked with them to come up with some other approaches, they thought this other approach would work. We worked with them on the PR for that approach. They approved it. We continued to do development on it. And then one day they had a negative reaction to all of the work that we had done with them. And very sadly this was like the same day that we didn't attend the community meeting because one person who tended the community meeting was in open heart surgery, which like, like, talk about messy. This is, yeah, it was like, Jay Faulkner Literally the most human thing that can happen. Alex Scammon Yeah. And, you know, without much if any warning, all of the work that we had done with them got reverted and, you know, and, and even worse than that we felt like cut out from the conversation that we didn't know why this was happening. And, you know, if we had done wrong, somehow like what was the reason for all this thing. You know, it's sort of sad and we had, we figured out other ways around the problem. And then I ran into one of the YuniKorn developers who we're friends with Wilfred at Kubecon, I don't know, it was last year. And, yeah, we chatted about it. We came to terms and then I needed to ask him to jump onto a panel for me because one of my panelists had not been able to make it to the conference and so, you know, we're still all compatriots and work together. It's not like, Jay Faulkner Was this why an underlying technical thing like what did anyone ever figure out a why there. Alex Scammon I think it was. There was some rationale for it, technically. But it was one of those ones where it was like, you can understand that, but I thought we talked about it with you and if you, if you didn't want it, then why did you accept the PR in the first place like, you know, cool not to have done it as well. I think that it felt like the emotion was that they they thought that we were trying to like steal their project out from under them, or do some nefarious thing. And, like, it's honestly not all we were trying to do we were just trying to make some improvements with their guidance. So, you know, that's a, it's just a prime example of how, you know, I wish better communication there. I wish, you know, we had not been an open heart surgery, when these decisions were being made. You know, there's a human story to all of this. I feel like a lot of feelings were happening without those feelings being communicated. And, and the, the ramification was that our code got pulled and we had to go find another solution to our problem. And that's that, you know. Jay Faulkner It's, it's interesting. I think there's a couple of things we can pull from that. And the first one is that you can kind of do everything right, right? Like, we had professionals people have been working in open source for decades. People who quite frankly, like the number of conversations I have where someone's just like, how do I open source? And we are like some of the people who helped teach folks that. And yet, you know, you can do all the things right you can check all the boxes and you can still not succeed and that's okay it happens sometimes. But the other thing is that there is an organizational reputation that travels open source software, because I, I would be willing to bet that, you know. I'm not going to speak for GR-OSS. I'll speak for Jay. If I spent a lot of time writing a feature for an open source software I submitted it it got merged, and then it got reverted later, without any input or even a conversation with me. I'm going to be honest, that would be like a friendship enders that would be a I don't want anything else to do with that project and like that may not be a professional reaction, but it would be my human reaction because that the idea of spending all that time investing all that time and having it flip back. It would be emotionally hurtful and it's okay to be emotionally hurt of stuff like that happens. Alex Scammon Yeah, I'd be honest I think we were a little emotionally heard I think we went through the five stages of open source reversion. You know, Jay Faulkner I don't remember getting a link to that Jira ticket from my direct boss with basically a what happened sort of idea of like take a look at this and see if you can help us figure out what's going on. Alex Scammon Yeah, and I think, you know, after we went through grief and anger and and all the rest. You know it. We were professional about it and it's like look okay if they're uncomfortable with this, you know, we don't like how this all went down we would rather have not spent the time. But rather to know this now, then to get even deeper into into making them feel uncomfortable about our presence or about our contributions. Fine. This is for the best and I feel like the journey that we've been on since is actually yielded a better solution. So it's, it is fine. Jay Faulkner The adversarial nature of open source leading to a better technical solution over the long term. Even if it hurts, maybe so that's, that's a, that's an interesting story. And I said I remember when it happened and I personally, I'm glad that didn't happen to me. It would have been incredibly difficult for me to wake up and go to work the next day if I was the person in that shoe I do not like throwing code away. And I especially don't like when, when I feel like folks have done stuff without talking to me so I, I know that had to have been a big one that's definitely, I would say like if I'm going to give out a blue ribbon for gracious moment that's up there that's pretty high. Alex Scammon The best one though was like the, the Zen blue ribbon, like if you're going to give it one out is to Headphone James when he was implementing the thing for OpenStack was it the secure networks thing way way back where I think he implemented it three times and twice the community after reviewing it we're like, oh, we needed to reimplement it in a different way. And the, the amount that Jay Faulkner We specifically worked on and resolved those issues for anyone hearing the story five years after the fact Alex Scammon Just the amount of self control that that man exhibited when faced with, you mean they're not going to accept this a second time, and having to redo the whole thing a third time. So that's the grossest moment I think. Jay Faulkner Yeah, that stuff happens to be an OpenStack my reaction becomes something more akin to "Oh hell no" exercise zero self control and we usually find out real quick what's actually going on. Alex Scammon But you said, you said. Jay Faulkner So, I mean, it's always worse that's kind of that's the thing is like if it wasn't hard it wouldn't be work right? Like so you're going to have the rough moments. You're going to have the hard times. You're going to have people who just have straight up philosophical disagreements but that's nice and hey I appreciate you taking the time to come on here. Alex took this with short notice. We needed a guest. I really do appreciate you being here talking a lot about Arctos Alliance. We'll make sure to have links to that in the description below. Honestly there's a graph we have on a blog post about it that I'm particularly a fan of where you can just straight up see that the project was quite literally cratering and then when Arctos stepped in, it hit the bounce and it started coming back up so it's impressive. That is the open source success story but on behalf of those of us here at GR-OSS and with the podcast. I'm Jay Faulkner got Alex Scammon here. Thanks to Ben for producing. We appreciate that. And, always, thank to G-Research for giving us time to go present these thoughts, these ideas, talk about what's going on in open source. If you're looking for a job, you can always go to gresearch.com/vacancies and apply. They are looking for folks all in the UK. Also, if you're just in that area, if you're in London and you want to hear about the coolest, latest in Fintech or in open source tech or whatever, go to gresearch.com. They've got evvents going on all the time. I can't possibly tell you what's specifically coming up next. Just go check it out, give them a look. But for now, this is Jay Faulkner. This has been the GR-OSS Out Podcast. Have a good day. Thank you. Alex Scammon Thanky you very much.