Sean: Hi, and welcome to Pod Rocket. I'm Sean, and with me today is Kadi Kraman, head of mobile dev at Formidable Labs here to go through her recent talk at the GraphQL Galaxy Conference handling Breaking Changes in GraphQL. Welcome to the Podcast, Kadi. Kadi Kraman: Thank you very much, Sean. It's a pleasure to be here. Sean: Yeah, we're glad to have you. And before we get into the talk, do you mind just giving us a bit of background about yourself and how you got into software and what you do at Formidable Labs? Kadi Kraman: Yeah, sure. Well, right now I am a director of engineering for mobile services at Formidable. So I kind of do a little bit of everything. I guess my specialty is a React native, but to be honest, I've kind of worked with all parts of the development stack, so from infrastructure to APIs to front ends on the web and mobile. So a little bit of everything. Mostly in JavaScript though, that's where I'm happiest as for how I got into development. Well, to be honest, I was one of those people that never intended to be a developer. I'm always envious of people who were like from when they were six, they were already coding. I was not that, I think I got my first computer when I was a teenager, but so the actually started coding at university, so I did a math degree and one of the modules was C++. So that was the first code that I wrote and I wrote some codes and I was like, oh my God, why have I been doing all this maths by hand when I can write a couple lines of code and have the computer do it for me? And I was kind of sold from that point onwards. And then, yeah, my current career has taken me here. Sean: Here. Yeah, no, I had a similar experience just getting started where I think one of the first programs I made was a calculator because oh, why don't we just speed up this arithmetic? So I definitely relate to you there. And in terms of the GraphQL talk, I guess, yeah, why did you think it was something that other people could learn from or did you experience those breaking changes in your day job? Kadi Kraman: Very much so. It was one of those talks that come out of a problem that I am constantly fighting and coming across and it's kind of raising awareness because I feel like if you are working on just one part of your application stack, so for example, in my current project we have a website and we have an API, we have a CMS and we have a mobile app. So there's a few moving parts and if you are only working on one half of it or maybe just on the API or just on the API and the website, you don't really consider how your changes affect other consumers of your API. So for me, I'm leading the mobile project, so I work on the mobile app mostly and on the API and on the CMS. So I don't really work on the website. And then there's a lot of web developers that just work on the website or they work on the website and the API. And because we are both editing the API, we are both in danger of introducing changes that the other can't consume. And in GraphQL especially, these changes can have more hard hitting results to the end user than you would sometimes in REST APIs. So you would actually get errors that users wouldn't be able to use a site, you wouldn't be able to use your app just because you made a change. And a lot of the time developers don't even realize that the thing that they are doing is causing a breaking change. So it's more raising an awareness that hey, this is a thing you shouldn't do and also there are tools to prevent you from doing it. Sean: Yeah, no, it's really interesting that it, because like you mentioned, there's kind of an organizational component to it as well as the team grows, and certain folks are working on the API side of things and some people are just working on consuming it might not know how it's changing. It's interesting that you mentioned the REST first GraphQL side of things. And I guess that makes me think maybe we should back up a bit because GraphQL is really common now and it's been a thing for years, but just in case there's listeners who might not know about it yet. Yeah. What is, I guess GraphQL? Kadi Kraman: Yeah, GraphQL so I mean it stands for graph query language. So the clues kind of in a name and essentially what it is an alternate way that you can build your API to be queried. So when I say API, I'm talking about web APIs. So REST is often said GraphQL versus REST because REST is the main big player on the stage. And then GraphQL's kind of the new kid on the block. So people tend to know about REST and they go, okay, how does this compare to scale? But they are both just specs. So there's a graph scale spec that says this is how you could build an API to be queried. And then lots of clear people have gone away and built a graph scale server in various languages according to the spec. And the difference is mostly that... So say that you're building a website and on this website you have a page where you display a user profile. So with a REST API, what you would do is obviously you need to get the user's data, so you would send a get request to API slash users slash one or whatever the idea is, and then this gives you back a bunch of JSON or XML, but hopefully JSON with the user's data, their name, first name, last name, whatever else there is, and then you can use that to render your page then. But with a GraphQL API, you would send this query, usually the convention is API slash graph gel, and rather than using the URL path to describe which date, what data you need, you would send a query parameter with the format is the graph gel the format. So it's like a string that describes exactly what data you want. So it's easiest to think about GraphQL the problems it was trying to solve. So the two problems were over fetching. So with GraphQL I can say, okay, I just want the user's name and maybe their age and the profile picture and I want just that data don't give me anything else. And then my graph query would come back with just that data. Whereas with REST I would just go, you know, API slash user slash one and I will get everything about that user, which could be a huge data object. So that was one of the things that it was trying to help solve. Sean: Nice. Yeah, we definitely love it at Log Rocket just because let's us still model our data on the backend in our actual entity relationships and then kind of have that decoupled query, later query layer to choose exactly what data we want. Kadi Kraman: Yeah, exactly. But it's one of those things that when you start using it at scale, you come up with other problems because obviously one of the good parts is that from a server point of view, you can have Subresolvers for all of your expensive routes. So if the user has, let's say they have, I don't know, friends that would return an array of friend types, but then this Subresolver for some reason was really expensive, so you had to do another API request or maybe you had to do several API requests, or maybe you just have a really inefficient database query, then you could have a more efficient front end query if you don't need, the friends don't just don't request that Subresolver. But then the problem is that it now creates a situation in which the consumer of the API needs to know which Subresolves are the expensive ones and then maybe try not to query them. So it's kind of pushing some responsibility to the front end that shouldn't be there. Sean: True. And especially in that case where the front end team is totally different from the team maintaining the API, now you've, that burden is on people who might not have been familiar with the different sub solvers and which ones are expensive. Kadi Kraman: Yeah, exactly. I mean, most projects that I've worked on, I tend to have access and actively contribute to both the front end and the graph scale API. So I kind of can see what's going on both sides. But that's not the case for, I would say most people. You are either working on the API or you're working on the front end and then you just wouldn't know. Sean: Yeah, so there's definitely a trade-off there. And in terms of the breaking change aspects of it, I guess what would be an example of a breaking change? Kadi Kraman: I mean the easiest example would be removing a field. So removing a field from a graph API type would be a breaking change. As an example, if you go back to that user type, so say that your user type had an ID and a name, and so the name is a string and that was the original version of the API, and then you publish it and you get business requirements saying that actually we want to have the first name and the last name separate. So rather than having one name which has first name space, last name, you would have the first name and last name separate. So it would be tempting for someone to add those two fields and then go, oh, actually I don't need this name anymore. And then just remove it from the schema and that's instantly a breaking change. Sean: Is there a way to solve for that? I know you kind of went over it in your talk, but for our listeners, what might the solution be? Kadi Kraman: The easier solution is to never remove anything from your API, never make any breaking changes, always do everything correctly the first time. Wouldn't that be great? Sean: If only. Kadi Kraman: Yeah, so I would say there's a way to do it that's straightforward, but it has a few steps and it's basically add deprecate, migrate, remove. So basically what you need to do is you add the field that you want to add. So in this case, we want to replace the name with first name and last name. Then we deprecate the fields that we don't want our API consumers to use anymore. So GraphQL comes with a deprecated directive, which is a way that you can denote a field in is schema to be deprecated. So there are most tools that he used to query. The APIs would also notify you that, oh, by the way, you can use this field for now, but it's going to go away. So you kind of notify your consumers that this is about to go. And then the next step is migrate. So all of the consumers of your API, so all of your websites that are using it, you would rewrite the code to stop using this deprecated field. You'd also need to release your website, you need to publish them. And then when you're confident that none of the clients are using this deprecated field anymore, then you can remove it from your API. So there's actually like two API releases in this process. So you add the new field, you deprecate the old field and you release the API, then you go to all of your clients that are using the API, you make the change to stop depending on this deprecated field, you release that, you release your front end, then you go back to the API, then you can remove the field and then you can release it. So there's lots of steps to it, but it's the only safe way to do it. And it's something to note that if you are building a public GraphQL API, so say that you are building the Twitter API in GraphQL, then there is no way to make such a change without having a V2 because it's a public API, anyone could be using it. Sean: In that deprecation stage the first time you released the API in the backwards compatible way. Let's say that the API is something that's user facing. Do you have suggestions for ways to present in the app or site to users that hey, time to update so that eventually in the future that breaking change can be made or when you have users who might not ever update an app or might not ever refresh their website, although that's harder to believe, is there just no fully foolproof way to do it? Kadi Kraman: This is kind of a problem that you have a lot less on the web and you have more on platforms where you have an app bundle that's deployed. So on a mobile app on iOS and Android or if you were doing a TV app or if you were doing a console app or anything, that's not a website because in a website we can basically, you are going to deploy a new version and then anyone who hits that URL is just going to get that new version as soon as your cash are hit. So it's kind of a problem that I personally get a lot because I do a lot of mobile development. To reiterate the problem that we're having is that we have an API and then this API has had some kind of breaking change and then we have a version of the app that we're working on, which is still depending on this field that no longer exists. And then from a user's point of view, what they're going to experience is just, well, I guess it's depending on how we're using it, but when you get breaking GraphQL change, then you don't get any data back. So with a REST API, say that you had a REST API for the user that was meant to return the user's name, but now it doesn't, right. What's the effect of that? It depends on how you use it I guess. But generally you would just have a page with missing data. However, what would happen on the GraphQL app is if you do a request to schema, but there's a breaking change so that the fields that I'm querying does no longer exist, rather than getting back some data, I actually get back no data. So rather than showing this page say with the user, but without their name, I would instead just see an error page. So that's why having a break in changing GraphQL is a more significant annoyance than we used to having. To go back to your original question, which I think was can you make people upgrade away from versions that are using breaking changes in the mobile world? There is no way to force people to upgrade because I mean, how would that work? Apple doesn't have this built-in, Google doesn't have this built-in. What if the person no longer has access to the internet or what if they don't want to spend 60 megabytes of their data updating your app? Because not everyone has access to unlimited internet. So usually what developers do is we build in an app upgrade prompt. So it can be if your app is older than three months or you'll do an API request and then the API will say what the minimum version is you're allowed to have. So you can remotely trigger the next time the user opens the app, you can say, sorry, we've got some updates. You need to update. And then just not let them in until they do. And so it's possible, but to be honest, users hate it. So it's one of those use with caution things, but it's good to have it for our last resort. Sean: Yeah, that's a good point about people might not want to spend the data to download something and you can't have the same expectations for all of your users about what they want out of your app and whether they're willing to download the update. So I guess at Formidable Labs, how do you tend to approach that issue? Is it just to never make fully breaking changes on mobile? Kadi Kraman: To be honest, I mean breaking changes come in many flavors. We talked about very straightforward one, which is removing a field, but a breaking change can really sneak in. And even when you are look watching out for them, it's actually quite easy to miss them. So another example of a breaking change would be if you change a data type. So if you change something from a string to a number or if you make in arguments to a mutation or a query, if you make arguments mandatory or if you said in schema, if you said that your return type it was going to be mandatory, it was definitely 100% always going to return something, but then you make it optional, then that's also a breaking change. Yeah, there's a lot of different types of breaking changes. So obviously we try not to make them, but they will sneak in whether you like to or not. And then, I mean the best option is just to handle them. You could trigger an automatic update if you're working on a mobile app, but usually on the web you would just need to update your website. But there's tools that help you guard against it. So the one that we use is called GraphQL Inspector. It does a bunch of things, but what it really does is it helps you explore your graph scale schema and you can set it up on GitHub or whatever CI you use to actually alert you if you're about to merge a pool request that has breaking changes in and it goes like, "Hey, you are removing a field, are you sure you want to do this?" And then that draws everyone's attention to it and you can make an informed decision if whether it's actually being used or not. Sean: That sounds super handy. So I guess it's kind of built into the CI process in an automated way? Kadi Kraman: Yeah, exactly. I mean we use it with GitHub actions and the way you do it is you have your GraphQL schema on your main branch and obviously you have a GraphQL schema on your pool requests branch, and then it does a between them and it's clearly defined what is a breaking change and what isn't. Yeah, it will just alert you, which is very convenient. And also it's very configurable as well, so you can set it to fail, recover check fail if you have a breaking change. So that's what we do. Our process is that if there is any breaking change, that's an immediate fail. So you can't merge that pool request unless you add a specific label that says approved breaking change it. This kind of signifies that, yeah, we know that there is a breaking change, we've looked at it and then a tech lead has determined that it's fine and then it's on their head. Sean: Gotcha. Yes, someone has to be very intentional about stepping over the guardrail. Kadi Kraman: Exactly, yeah. Sean: Are there any kind of rules, I guess that GraphQL inspector sounds like would be pretty foolproof, but are there any kind of rules of thumb to know if you're making a breaking change? It sounds like the big one is if something becomes required that wasn't or you're changing something about a required field. Kadi Kraman: All right, so it's removing fields, always breaking, changing any data types, always breaking, changing a return type from mandatory to optional and changing an input type from optional to mandatory. So those are the four schema changes. And the return type one is actually, it's easy to, it's not obvious why it's a breaking change. But a good example is if you had this user API, again, if you had a user's name which returns a string and say that originally I'm telling in my schemer that it's always returning a string. And so then on the client side you might do something like user.name and call split on it. So split on a space and because I know that the name is always available, then I can always call split on it. But then if you change the schema, so that's the name is optional, means that you can return null or undefined from the name field. And that in itself isn't actually breaking in that it's not going to cause your query itself to fail. However, if you return null from it and then your client was expecting that it's not null and call splits on it, then you're going to get a JavaScript error on your front end. Sean: So it sounds like a manageable list of things to watch out for. Kadi Kraman: Yeah, definitely. But I will say that's one thing I forgot to mention earlier when you asked about what GraphQL is and what makes it special, and I did say that it was built to solve two things and one of those things is over fetching so that you only query what you actually need. But then the other thing is knowing what you're going to get from an API. So an API being to an extent. So from a REST API, I mean I'm sure you have spent a lot of your life trying to look at API documentation to figure out what the hell this API is going to return? What arguments, what query parameters do I need, what formats are there? And with scale, you don't have this problem or you have it to a lesser extent because GraphQL APIs ship with a contract. So it's like a declaration of here's everything I have, here are all the queries, here are all the mutations, here are all the input types, and here are all the return types. So it tells you with every GraphQL API you can see at a glance what is available to query in a API, and then that's the contract. And when we talk about breaking changes, then they are breaches to this contract. So when you integrate with a graph API, you have a contract, it's like they come to you and go like, "Hey, here's everything you're going to get." And you're like, "Cool, I'm taking this thing, I'm implementing my front end and I'm leaving my front end there." But then if your API changes, the contract breaches it like Rooster Field, it makes some changes, some data types, then this contract is no longer valid. And then that's why your front end is going to fail. And then that's why we're getting errors and that's what the breaking change is. So it is a result of this really good thing that we get. And one thing that I've actually been thinking about as well is it seems like such a pain that we get so severely punished in GraphQL with breaking changes compared to REST. However, the good thing is that we are aware that these changes are happening because with a REST API it's equally easy to make these breaking changes and there's still breaking changes. However, it will be very difficult to track them at all to discover them to communicate that they exist. So there's pros and cons, I think. Sean: At first it seems annoying to have to deal with breaking changes, but it really is failing as fast as possible so that people know to look out for them. And I know in REST APIs there's that open API spec, which I guess is sort of an analog to the contract that you're talking about, but not all APIs necessarily follow them and it's not built in to the code. So I guess it's not something that we can depend on for all APIs. Kadi Kraman: I mean that's the thing, a lot of things that we as developers just agree that we're going to follow because I mean OAuth, Open ID is just spec, REST is just spec, GraphQL is just spec. So if the folks at Apollo server decide that they no longer want to have a spec compliant GraphQL server, then they could do that. They could just add their own things in there or choose not to implement new things that get added to the spec. Right. So it depends on people's understanding of the spec. So I actually built an open source library that handles OAuth and there is a very large spec document obviously for OAuth too, and it's quite difficult to read, but there's a lot in there. And it's very clear that all of these services that have implemented OAuth have, I've had bits of it, but there's a lot of spec non-compliant or with clients out there. There's something you can do about it. It's like, I mean that's the spec, but they made a mistake or they chose not to do something. Sean: Do you think that's most of that is unintentional, that there's just a lot of little details that not everyone gets? Right? Kadi Kraman: I don't think it's intentional at all. I think you are right. I think it's like a accidental, I mean, okay, so here's a very small spec, I'm going to say in compliance, is that the right word? It's actually in the GitHub or API. So by default, so the whole OAuth spec says that all the responses should come back in JSON. So you have the content type application JSON. In GitHub, they actually come back in XML by default and you need to add a header for application type, sorry, quantification JSON in order to get it in JSON. And that's not really actually a big deal. You'll add that header and then you get the data you need. However, technically that's not spec compliant. Sean: That's interesting. Yeah. And one other spec, this is recently came up at our work recently is the YAML one is notoriously long and I think hard for people who have implemented it to get it 100% right. Kadi Kraman: Oh yeah, definitely. And I mean there's also part of it could be personal preference. You can choose to interpret something in a way that you prefer. Sean: Yeah, well the English language is ambiguous at times. So I guess it also depends on whoever wrote the spec to be as exact as they can. Kadi Kraman: Very true. Sean: I think that this is definitely gave people a lot of great tips and pointers for making graphical changes. But before we wrap, I'm just curious if there's anything else you want to plug, any more resources you want to point people towards? Kadi Kraman: I mean, for GraphQL, honestly, the main thing is GraphQL inspector. It's very handy, I like it a lot. If breaking changes is something that you need to be wary of, honestly take the time. It only will take a day tops to integrate it into your workflow and it's going to save you a ton of time and efforts in debugging. Sean: Yeah, I plan on looking into it after this. I hadn't heard of it yet. Well, thank you for coming on. We really appreciate it. It was great to have you. Kadi Kraman: Thank you very much, Sean. It's been a pleasure.