Episode 295 - Shurui Zhou and Jiayi Sun
[00:00:09] Richard: Hello and welcome to Sustain, the podcast where we talk about sustaining open source for the long haul. Who are we? Where did we come from? Where are we going? Who put a fork in this thing? Very excited to talk to our guests today. I'm Richard Littauer, your solo host today. Hello, everyone. That's still me.
[00:00:25] I don't know why I used a different voice. And I'm very excited to be joined today by two guests, both calling from Toronto, which is a pretty cool city if you haven't been. It's in Canada, which is a pretty cool country if you haven't been. I'm joined by Shuaie Zhou and Jiayi Sun. Shuaie and Jiayi are also co-authors of a paper that we have written together.
[00:00:47] So this is a very self-serving podcast in that we all just get to talk about how cool we are for publishing a really awesome paper about open source and sustainability and codes of conduct. Just so you know who the voices are on this podcast, I'm [00:01:00] gonna introduce them individually. So I'm joined today by Jiayi Sun, PhD candidate.
[00:01:04] Jiayi, how you doing? 
[00:01:06] Jiayi Sun: I'm doing good. I'm Jiayi Sun. I'm a PhD candidate at the University of Toronto, working with Professor Xuerui Zhou, who is also joining us here today. And my research focus on using the empirical methods, such as quantitative and mixed method, to study the collaboration and the long-term sustainability in the open source software ecosystems, and particularly within the scientific open source software communities.
[00:01:33] And so I'm interested in understanding the challenges the community face, and also designing tools, better support the contributors and maintainers. 
[00:01:43] Richard: That sounds pretty awesome. Thank you so much. Xuerui Zhou, you lead the ForCo lab. I believe that's how you pronounce it. Can you tell us about your work in general?
[00:01:52] Is that the main thing that you do? 
[00:01:53] Shurui Zhou: Yeah. Hi, I'm Shurui, an assistant professor at Univ- University of Toronto. So my research is [00:02:00] focusing on helping software teams to better collaborate and to build high quality software systems. Specifically, we look into different format of software teams such as co-located teams, distributed teams, interdisciplinary teams, and also teams working on different domain specific software systems.
[00:02:19] Uh, for example, we look into research software or scientific software, or recently we also looked into medical healthcare AI software systems. Thanks for having me here. 
[00:02:31] Richard: Thanks for coming on. We had you on back in 2020 to talk about your PhD thesis, which I believe was called What the Fork? Um- 
[00:02:38] Shurui Zhou: That was one of the paper titles, yeah.
[00:02:40] Richard: One of the papers. It was the best one. I really liked that title. About forking open source, you had come out of CMU, where you worked with Bogdan Stanisescu, I think I pronounced that correctly, who also focuses on the entire open source ecosystem. Xuerui, when you say that you focus on how we understand open [00:03:00] source and how teams work together, you're not doing that from a proprietor's perspective.
[00:03:03] There's no company which has paid you to work on it, and you're not working on particular teams. You're more focused on the entire ecosystem. Is that right? 
[00:03:10] Shurui Zhou: Yes. 
[00:03:11] Richard: Awesome. And Jiayi as well, when you're building tools, you're also focused on everything. So you said that you used quantitative methods. Just as an example, in this recent paper we're going to talk about- How many different repos did you look at?
[00:03:26] Jiayi Sun: I think our data set is at the scale of 150,000 repositories. So this is the scale of what we are talking about for now. 
[00:03:36] Richard: Okay. Listeners, strap in then. This is gonna be the entire ecosystem going on. That's a lot of data coming at you. This is really exciting. Just to narrow it down, let's focus in on this particular paper, which was published at ICSE.
[00:03:48] I actually forget what ICSE stands for. Do either of you know what ICSE stands for? 
[00:03:52] Jiayi Sun: International Conference on Software Engineering. 
[00:03:55] Richard: Cool. Last year was in Rio as well, which I believe you were able to go, Jiayi? 
[00:03:59] Jiayi Sun: Yeah. [00:04:00] 
[00:04:00] Richard: That sounds really fun. I'm glad you were able to go. Tell us about the title of the paper and what was going on in it.
[00:04:05] What's the high level overview? 
[00:04:08] Jiayi Sun: Right. So the title of the paper is Beyond Adoption: Examining the Evolution and Impact of Codes of Conduct on Open Source Communities. So in this paper, as the title suggests, we study beyond whether an open source project has adopted code of conduct and ask what happens afterwards, and we study how the, uh, communities maintain and update their codes of conduct over time, and how the adoption relates to the community engagement changes.
[00:04:39] So we find that most project rarely updates their code of conduct after adoption. At the same time, the adoption is associated with attracting more newcomers, but also a short, modest, uh, short-term increase in disengaging the contributors. So overall, I think our findings suggest that the code of conduct can be an [00:05:00] important signal for the comer, but the documentation alone might not be enough for solving the retention problem in open source.
[00:05:09] Richard: Awesome. Okay, there's a lot to go into there. Let's start at the very front. I feel like we already knew that codes of conducts were effective and that sometimes starting them makes some people go away, but the people who leave were kinda people you don't want around anyway. So it seems like we just underlined that.
[00:05:23] What was the point of making the paper? 
[00:05:25] Jiayi Sun: So I think the motivation come from codes of conduct has become quite common in open source communities. A lot of projects have adopted it from the very beginning. But simply knowing, like, whether a project has one tell us very little about, like, how is it used in practice.
[00:05:44] And prior research that we have looked at mostly I think focused on why the communities adopt these policies and what these policies contain. But we wanna look beyond the moment of the adoption, whether the communities [00:06:00] continue to maintain them and how they evolve as the community projects grow, and whether adoption itself is followed by any changes in the contributors, uh, participations.
[00:06:12] Richard: As part of this work, you would have had to do a lit review. And listeners, I know I'm a co-author. Yeah. I didn't read the paper as in-depth as either of the other two people on this podcast, so I'm going to continue to pretend that I've never read it, which may actually be accurate. You'll never know. Jiayi, as part of this, you had to write a lit review.
[00:06:28] Have other people done this sort of work before and looked at the long-term repository health after a code of conduct? 
[00:06:34] Jiayi Sun: So we haven't found any large scale or comprehensive of data on what happens afterwards, but in the field of empirical study on this topic, many people have studied what kind of content people put into code of conduct, like what it covers, the topic it covers, and also there have been some preliminary exploration on the correlation [00:07:00] between the code of conduct existence with the changes in the, for example, the people, uh, the community's diversity change as such.
[00:07:10] So I think that's what we know so far, but we haven't been able to identify a comprehensive data set that can support the direction we wanna, uh, further investigate. 
[00:07:22] Richard: You mentioned you looked at 150,000 repositories- Yeah ... just right at the beginning. How did you find those repositories? How did you narrow them down, and how did you find out whether there were codes of conduct in them?
[00:07:34] Jiayi Sun: So in terms of compiling this large-scale data set, I think we mainly focus on the platform of GitHub, which is a lot of project hosting their code repositories and their community discussion happens. So we began by searching the GitHub issues and pull requests, also the commit messages, for reference of the keyword code of conduct, and then we [00:08:00] further expanded this candidate set to other known fork repositories that owned by the same account, because the organizations can often use the same code of conduct across multiple repositories without discussing the adoption separately in every project.
[00:08:18] And then with this candidate set, we further validate the, um, code of presence with the help of the GitHub platform feature, where the repository usually have a badge to signify if they have a code of conduct or not. And so by checking this community profile metadata, and also check if the repository root folder have the file name contains code of conduct, and we were able to exclude the projects with no code of conduct in their repository or the project itself might not be a software project.
[00:08:53] For example, like they are a weekend fun project, so that then we can end up with a, uh, [00:09:00] focus set of the repositories that have shown some public interest and they are active communities in the open source field. 
[00:09:10] Richard: Awesome. Okay, so using the GitHub API, you're able to look at the metadata and see- Mm-hmm
[00:09:13] which ones have code of conduct and then use those. From my general understanding, most code of conducts are just the contributor covenant just thrown in, 'cause the contributor covenant is probably the most well-known- Mm-hmm ... code of conduct. How many different codes of conducts did you find, and how many different types were there?
[00:09:31] Jiayi Sun: Um, so I think that's one interesting question in terms of the actual content of a code of conduct. So for the scope of the study, we did not differentiate between what types of template they are using, adopting, but we do identify, say, the length of the code of conduct and how many code of conduct has been reused in the same repository So that this is [00:10:00] one aspect we're looking at this data set was more from the added history perspective.
[00:10:06] Would that make sense? 
[00:10:07] Richard: It does make sense, yeah. Yeah. Okay. Given that, how did you go through that many repositories? Did you single-handedly click through each one and just judge whether, like, it looked like it was a healthy community afterwards? 
[00:10:18] Jiayi Sun: In terms of the healthy community afterwards, so I think with this data set, it gave us a starting point to answer our different questions for the study.
[00:10:30] I think I can briefly go through what type of question that we want answered. So in this study, it contains mainly more connected parts. So initially, we construct this large scale data set of GitHub repositories of over 150,000 with the CoC and their change histories. And then we further ask the two questions: What type of changes are they making, and how does the adoption have an [00:11:00] impact on the community engagement or not?
[00:11:02] So this initial set provide us a pool of candidates to select from. First, if we want to answer the question about the, uh, change type or intentions, we would focus on a subset of the projects that have more intensive added history, which is one of the more surprising finding for us because when we look at the history itself, over 75% of the projects never touches their code of conduct after initial adoption.
[00:11:35] So that significantly reduced the number of projects that we can further use for or to answer our research questions. 
[00:11:45] Richard: So that means they just took a code of conduct- Yeah ... probably a template, threw it in, and then never looked at it again. 
[00:11:50] Jiayi Sun: Exactly. So it's like the rubber stamp. Poof, we have a code of conduct, and then just leave it there and never touch this again.
[00:11:58] Richard: Did you look and see if there was [00:12:00] any way of seeing if any code of conducts were referenced at any point in the repository history? Did people ever, like, open issues and say, "This violates the code of conduct," or did they log, "We had a complaint, and we resolved it"? Um, how often does that sort of thing happen?
[00:12:17] Jiayi Sun: In terms of the issue discussions, we did not particularly check those aspect. Okay. But we, I think our focus was more, like, say if from the maintainer's perspective, do they actually follow up with any sort of updates and changes if anything regarding code of conducts happens and the maintenance team haven't made a decision about code of conduct related stuff to add it into the file?
[00:12:48] So I think that's we're looking at with this data. 
[00:12:52] Richard: From a practitioner's perspective, it's interesting to me that 70% of the repos didn't change their code of conduct, the code of [00:13:00] conduct didn't grow. Is there anything that open source projects should be doing? Should they be changing them? Did you find that projects that did change, the other 30%, were somehow healthier?
[00:13:09] Jiayi Sun: So I think that question in terms of healthier, it's actually very challenging for a research study to identify because there is a causal implication that the adoption of code of conduct or the follow-up maintenance code of conduct can actually lead to a healthier change in the community. I think two parts of our study, uh, was focusing on exploring this relationship, if we can infer any causal relationship.
[00:13:42] And one part of it we look at, we cluster, or so one of the-- our study was to cluster the different patterns of code of conduct maintenance, the projects of different maturity to see if there are a common pattern among the [00:14:00] long-living projects versus the, uh, projects that showed a short-lived. But in terms of the results, we find that the, uh, pattern are very diverse.
[00:14:10] So there can be long projects that have minimal updates to their CoC, but have a large growing contributor base. But there are also projects that have frequent updates to their Co-code of conduct, but their contributor base was not as diverse or evolving. So in terms of the direct conclusion we can draw on the health impact, I think that would be something more individual to the project itself.
[00:14:42] And, uh, one thing we listed could be in the future work was to have more longitudinal data that we can draw that conclusion on for individual projects. I hope that answers question. 
[00:14:56] Richard: Yeah, it does. It kind of puts an asterisk and says, "See future work." [00:15:00] It's okay. Yeah. It's like the point of all studies is to have a really small glimpse at what you're doing, not to- Mm
[00:15:05] say, "Here's how you should do everything in the world." So I get it. You used a fun word, clustering, and I actually wanna dive into that a bit more because I know you didn't do this by hand for 150,000 repos. What tools did you use to automatically cluster and understand these different objects that you were looking at?
[00:15:21] Jiayi Sun: So I think that was one main part where we look at the code of conduct update changes, like what type of changes are made. So remember we talked about the over 70% of the approach have never touched their code of conduct. So within the small set of code of conduct that have pretty much intensive commit history, we actually look at the, um, commit diff, the get diff of each commit to compare the code of conduct file changes.
[00:15:51] And because of the number of commits we ended up with was around 8,000 commits of the [00:16:00] repository have the CoC changes on, so we want to know what kind of changes are made there, what type of changes are made. And we used LLM at the time, I think it was around two years ago. We used LLM GPT-4 Turbo to help us scale up the analysis because if with just me and another author looking at the changes it will take Ages, months to go through all the commits and advise changes.
[00:16:29] So in this method, we mainly initially did on a smaller set and to derive a code book categorizing the different type of changes, and then we use LLM to scale up the study and identify all the change types for these commits not made to the COC file. And with that, we were able to identify the type of changes made over time for each repository.
[00:16:55] And then when we normalize the changes made through the [00:17:00] lifespan of one repository, we were able to categorize the, the repository updates, maintenance history into a pattern fitting into the repository age and group them by this normalized vector so that we can compare the re- projects of different lifetime and to see if there is a pattern in the sense of how they update their code of conduct.
[00:17:26] Richard: Awesome. Okay. That sounds like a really useful tool. Some of that got to question one, which was understanding how do we understand what the clustering is? Yeah. And how do we understand how people adjust and move their code of conducts if they ever do? I feel like it's now probably a good time to switch on to question two, which was the impact of COCs on your repo.
[00:17:44] Mm-hmm. Can you talk a bit about what that found? 
[00:17:47] Jiayi Sun: So for the question of understanding the causal impact of COC adoption with the community engagement, we are particularly looking at the number of the contributors changes [00:18:00] in terms of the new contributors joining and the existing contributors leaving pre and post adoption of code of conduct.
[00:18:08] And we kind of split that into different observation window to look at the long term and short term effect. And through our analysis, we find out in the short term, the adoption of code of conduct can have a causal relationship with the increase in newcomers joining and both the existing contributors disengaging.
[00:18:30] So the sort of, it's kind of, uh, implying a paradox in the sense of what's the function of a code of conduct? What the impact of a code of conduct have on the community? It can attracts people and potentially in a way that also have people leaving the community. 
[00:18:51] Richard: Did you look at what kind of people leave and whether they're jerks?
[00:18:55] Jiayi Sun: That's an interesting question. We did not. That's a straightforward answer, [00:19:00] but so, yeah, future work. So that could be one fut- uh, one limitation to our study and also one future direction we wanna look into. So currently we don't discriminate against different type of contributors leaving or joining, but I think one of our assumption and suspicion was that it can have different impact on whether the contributor is a core contributor or is a peripheral contributor who make one-time changes to the community.
[00:19:34] They do not have the intention of staying longer or versus the, uh, contributor who are actually affected by the, uh, CoC policies that driving them away. So that's what we leave for future work in terms of the contributors' backgrounds and their relationship with the adoption of code of conduct. 
[00:19:56] Richard: Cool. So broadly speaking, [00:20:00] CoCs are useful.
[00:20:01] A lot of people have them. Not enough people look at editing them for their own particular needs. The ones that have put them in drive some people away, but then other people may join, and we don't know who those people are. We just know that there's a weird paradox where there's kind of this forcing function of change, and that's what this paper can say.
[00:20:19] Jiayi Sun: Right. I think it's very hard to draw a, uh, conclusion on saying if CoC cause people to leave or CoC cause people to join. From our analysis so far, I think our interpretation was more like CoC can be an active signal for the communities that there is a policy in place and the contributors might join.
[00:20:46] But when we look at the long term, there has been minimal changes in terms of if people are joining or leaving. So that's, might suggest that they, in the sense of fostering more [00:21:00] healthy community culture or retaining existing contributors, that needs more structural change and maintenance, active maintenance to the community rather than just having one policy in place.
[00:21:15] That's what we can tell from the, uh, current analysis. 
[00:21:19] Richard: At the very end of the paper, you have this really fun section called 5.2, which is implications for OSS communities. I'm gonna add implications for OSS community sustainability because it's basically the same, let's face it. These are really directive, declarative statements of things you can do today based upon this study to sort of help out.
[00:21:37] One of them I think you just got to, Jiayi, when you... So the first one says, "Adopt CoCs as a meaningful signal, then treat them as living documents." I think that's pretty well borne out by this study, right? CoCs are useful for some people to know how you feel about at least CoCs, if not about your code and about your contributors.
[00:21:55] So when you adopt them, think of them that way. Think of them as this is a flag in the sand. Do you wanna talk [00:22:00] about the other four? Are there any that particularly stick out for you? 
[00:22:03] Jiayi Sun: So I think I briefly touched upon the future work. We wanna look at, see, like, for i- individual level, for the repositories and the different background of the contributors, how the CoC might affect them, uh, differently.
[00:22:17] And this is more, requires more longitudinal data of the repository itself to see. So we kind of demo- democratized that if the repository can leave reports, transparency reports to help document the incidents, anonymize the incidents and what happened, how the conflict is resolved. That could help provide more data for the research at the bigger scale to see if the CoC system can vary differently for different contributors.
[00:22:49] I think, so the study was done two years ago, and a lot have changed recently in the scale of open source communities. [00:23:00] And I think this kind of cultural, uh, the interaction between contributors and people has become, this question have become even more important with now the emergence of agentic software engineering and the, uh, growing volume of AI assistant contributions.
[00:23:19] So AI tools now have lowered the barrier of contributing to open source, but they also increase the submission volume and changes how contributors collaborate. And that created additional coordination work for maintainers and existing contributors. And we've seen posts and discussions that community members have very different views on whether the AI should be used in their repository, in their community, and they're having pretty heated argument and conflicts happening.
[00:23:57] So One thing that we've [00:24:00] seen a lot of community adopted was to add sections about how AI should be used in their community, in their code of conduct, or having totally separate policy files for how the community members can have conversations around this kind of topic. So I think in the sense of studying the, uh, maintainers of community policy, uh, which this topic has become more relevant as the field of software engineering or open source community has been changing, uh, so much recently 
[00:24:37] Richard: I would agree with that.
[00:24:39] It sounds like there's a lot of work left to do, which is okay, because that's kind of how it works. You make a paper, then you make another one. The prize for making a paper is you get to make another one. So yeah, good work with this. This was really cool, and thanks for coming on to talk about it. I really enjoyed this paper.
[00:24:53] It was really fun to help out. I probably did the least amount of effort out of all the other contributors. There are other people who are also on this [00:25:00] paper, Hongbo Fang, Jianming Zhang, Jiaqi Shi, Ryotira Lai, and Anita Ihuman. And I apologize to any of you if I mispronounced your names. I am not yet fluent in Mandarin, and I will probably never be.
[00:25:12] But I'm just really excited to have these other guests on, and also the last author, Shirey Zhou, who did the least amount of work. That's not true. The last author does a lot of work. Well done, Shirey, for getting this out as well. For the listeners who don't know, in computer science, sometimes conferences are actually more important than journals in the sense of that's where all the really cool papers are.
[00:25:34] So ICSE is at the top of the field, and this was really a groundbreaking paper for the scale of how it worked and the fact that you now have something to point to if you ever say, "Do code of conducts actually do anything?" Yes, they are a useful signal to contributors about the health of your community.
[00:25:51] Yes, they affect change. Do we know exactly what that change is? No, because scientists always have to add an asterisk on the end. But yes, they are meaningful and you [00:26:00] should use them. So Jiayi, thank you so much. And Shirey, thank you so much for coming on. Is there anything I just misinterpreted in my short summary?
[00:26:08] Is there anything else that you would add? 
[00:26:10] Shurui Zhou: No, there's nothing you misinterpreted, but I wanna mention that this project was supported, funded by Nonfocus, and we really appreciate the support. The idea was also orig- originally actually from the working group, I think Richard was one of the organizers, talking about EDI, and then we start to brainstorm and then talk about, okay, community managers, they already have a code of conduct, but they were still spending quite some time and effort trying to resolve conflicts offline.
[00:26:43] And so that was actually the beginning of the idea of, since we all have code of conduct, then why or why not it doesn't work as we expected? And then after that, we turned that into a research topic to really ask what happens after they [00:27:00] adopt it and are they really useful? And yeah, so this is something I wanna add.
[00:27:05] And I wanna say, although Richard and Anita, you didn't help much on really writing the paper, but you actually gave us a lot of resources, help us to really reach out to practitioners and to talk to those. Although, actually, we conducted bunch of interviews with practitioners who were focusing on writing or contributing revision of this code of conduct.
[00:27:27] Interestingly, we also talked to a few, I don't know whether this is Appropriate to mention, maybe feel free to cut it off. We also talked to people who were actually banned by some of the communities, and they share their opinions. And we saw actually both sides of the world, and it was very, very valuable for us to learn from the practitioners.
[00:27:50] And also in August, I presented this work in the yearly conference of open source community, FOSSY, in Vancouver and [00:28:00] in front of a group of open source practitioners. That was also very, very inspiring to hear from their feedback, and they were also very happy to see actually we provided an overview of what's going on, especially in the code of conduct activities in open source.
[00:28:18] And they also provided us a lot of great ideas that we could explore in the future. Yeah, this is something I wanna add. 
[00:28:26] Richard: I completely forgotten about the Sustain EDI working group. So thanks for bringing that up, and I'm glad something came of it. That's really, really awesome. It's good to know that sometimes it takes a while, and things just slowly move.
[00:28:37] And also, a huge shout-out to FOSSY. FOSSY is the best. That's the free and open source software yearly conference run by SFC, Software Freedom Conservancy, my favorite people, happening again next year. I believe the CFP is open at the moment, or it will be soon. You should totally check it out if you are a listener.
[00:28:56] FOSSY is a great conference to go to. [00:29:00] Jiayi and Sheree, I hope that the work continues and you have a great time. Do keep in touch. Do keep us updated on what comes out of your lab. Jiayi, best of luck with doing your PhD as a fellow PhD student. Thank you. Thank you. Ha ha, ha ha. Yeah, no, you, you got this.
[00:29:13] You got this. You're doing great. And yeah, thank you so much. Thank you. You can follow along with both of our guests by going to the show notes at podcast@sustainoss.org and looking at this episode where we will have links to their profiles where you can see what their current work is and their socials and so on.
[00:29:29] Don't leave yet. At the end of every episode, we have a section called Spotlight where we highlight other people, projects, or things that we think are particularly great or that we just think should have some light shed on them. Traditionally, the host goes first, so I'm going to go first. Today, I wanna spotlight Seldon.
[00:29:46] Seldon is the best person ever, also known as Lori Voss. And Lori Voss has recently written a wonderful, wonderful blog post about open source sustainability and how [00:30:00] registries can right now change the entire nature of the game by just funding maintainers directly Laurie is one of the smartest people I know working on sustainability.
[00:30:08] They ran NPM for five years, and I highly suggest this blog post. It is rare that at 9:00 PM at night I get really excited by open source sustainability. Normally, I get really depressed by open source sustainability. Like every good war reporter, you know, it's just like, everything is horrible. This blog post made me so excited that I sat down and read it in my chair and sort of ignored anything else going on for the entire time.
[00:30:33] So do check out seldo.com and the blog post, Nobody Pays for Open Source. We Can Force Them To. Thank you, Laurie. Shaoke, do you have something that you wanna highlight today? 
[00:30:44] Shurui Zhou: Yeah, I would like to spotlight Arles from, uh, Nonfocus. She was the point of contact for us to actually be connected to many of the open source community practitioners and help us to spread out [00:31:00] the recruitment for our user studies and help us to collect a lot of valuable opinions from practitioners and really wanna just say thank you to Arles.
[00:31:10] Richard: That's Arles Commons. Arles is amazing. Thank you so much. I've dropped her links in the show notes. Jiayi, is there anything you would like to spotlight today? 
[00:31:19] Jiayi Sun: Okay, I don't have a link to share about right now, but I wish there could be more discussion on particularly the scientific open source software communities.
[00:31:31] So this is my, one of my main research interest. So it is the, about the specific group of people, which is namely the scientist who work on developing open source software that outside of their main research work, but more like when, when people talk about it's like the byproduct of their research where scientist main goal was about publishing papers, getting publication done by [00:32:00] developing this community use software to support as a way of altruism to support the broader scientific community.
[00:32:09] I think that was one aspect or, or particular group I wish people could pay more attention to in a sense of no matter it was about the funding the project or the, it's the tools that can be developed to support their collaborative work I wish I could find some link about it, but I couldn't right now 
[00:32:34] Richard: I don't think you need to have a link.
[00:32:35] That's an excellent spotlight, and I also would like to spotlight the entire software, scientific software community. Thank you so much. That's great. That's perfect. And thank you listeners for joining us on this podcast. It was really fun to have you even though you got to say absolutely nothing. Uh, if you would like to say something, you can totally do so.
[00:32:55] Yes. You can go to discourse.sustainoss.org, and you can comment on our forum, or you can email me at [00:33:00] richard@sustainoss.org and be like, "You never invited me on. Of course I couldn't say anything, you jerk." That's totally fine. I will read that email. You can also go on social media and say, "This podcast is so good, I wish I could go on.
[00:33:10] It would be really, really nice." Feel free, or you could just say, "This podcast is really good. You should listen to Sheree and Jiayi talk about this work." That would also be really sweet. You can go on ACM and actually download this paper, or you can just find it in our show notes at podcast@sustainoss.org, where you will also find links to other things that we've mentioned today.
[00:33:29] If you have liked this podcast, please like it. Please share it. Please hug it. Please send it in a postcard to your grandmother so she can also listen. That would be the best. And if you really, really liked it, you should donate because this podcast is not free, and it's always nice to continue to be able to pay our editors and software providers for this service.
[00:33:48] You can do that at opencollective/sustainoss. Thank you so much once again for coming on, Jiayi and Sheree. This was really great. Best of luck with future work, which is always at the end of [00:34:00] every single paper. I hope one day there's no future work and you just solve everything. 
[00:34:04] Shurui Zhou: Thank you, Richard. Thank you for having us