AMOS: Woah! So, Zoom has two windows that open full screen, because I have
multiple monitors, and one of them is just me, so I can see my video on a separate screen.


CHRIS: Oh, fancy!


AMOS: Yeah, that’s weird.


CHRIS: That’s called the vanity monitor.


AMOS: I don’t know that we’re going to be able to record. 


CHRIS: I’m so distracted.


AMOS: I’m just sort of… infatuated with myself here.


*intro music*
AMOS: Welcome to Elixir Outlaws, the hallway track of the Elixir community.
*more intro music*


AMOS: So, what’ve you been up to lately?


CHRIS: Oh, man, all kinds of stuff. I feel like it’s become part of my job - self imposed in, an arrogant way - to contribute to the other libraries that are out there. Because there are certain things in the Elixir world that I get uppity about, and so whenever I see this stuff I feel duty bound to go ping people about it. I’m not going to say who or what the library was, because I don’t wanna be a jerk, you know what I mean?


AMOS: Yeah, you’re nicer than me.  


CHRIS: They were sort of saying ‘oh we built this thing that gives you a singleton process across your cluster’. So immediately my Spidey sense - my Elixir sense - started tingling, and I went and looked at it; they were using Global, and Global doesn’t give you that guarantee. It's funny, because in the read me, it says ‘have you used Global and had these problems? Well guess what! We’ve solved them!’. I pointed out, ‘well actually, it doesn’t solve the problem that you think it solves, like, for instance, if you get split brain, or rather, if you have a partition in your cluster, you’ll easily get split brain, meaning you’ll have two processes running, because that’s just how Global works. And so, I’ve been doing that on a bunch of libraries recently.


AMOS: Did you paste your talk from Elixir days?


CHRIS: No… that felt really really self-promoting.


AMOS: Well, I mean, you you are wearing the shirt that has your name on the back right now, so…


CHRIS: I love these shirts. You know it’s funny - these shirts from Elixir days, they’re this stark white shirt with the big purple Jimi Hendrix. It’s funny, I wore this and my wife saw me, and she goes ‘I think that’s the first time I’ve seen you wear anything but black or gray.


AMOS: My wife had a very different response. She’s like ‘why are you wearing a shirt of a guy with a joint in his mouth around our children?’. I said, ‘I think its a cigarette’. So that’s one way you can get over the need to fix every library. Just have lots of children like me. Then you have the need, you just don’t have the time, so it doesn’t matter.


CHRIS: Right, yeah.


AMOS: I can’t argue with people on the internet.


CHRIS: I try not to argue. In fact, I often will write the stuff up, and then send it to other people and say ‘am I being a jerk? Does this sound like me being a jerk? Or, is this fair to comment?’ So I have to fact check it…. Tone on the internet is so hard. I have definitely fallen prey to being sarcastic, and it totally doesn’t come across that way. If people don’t know me, they may not realize I’m a sarcastic person, and then it comes across as being a jerk, or being really condescending, and you never want that. That’s not a good way to have a conversation. There’s so much fidelity that’s lost when you enter into a sort of text based medium. So you have to be really careful about that stuff, I think.


AMOS: So, I work with people all over the world quite a bit - and I’ve noticed that different cultures too. We’re becoming more of a one world culture, but people approach things differently from everywhere. Some people come across as crass, and then when you talk to them, you’re like, ‘oh, no, that’s just how their culture is’, orr you deal with somebody else from the same culture. That can be really difficult too. 


CHRIS: Yeah, no doubt. All of it’s tricky. And you have to give everyone the benefit of the doubt, and just say ‘I bet they really didn’t mean to be offensive, or to be rude to me or to hurt my feelings. They’re probably just trying to be funny and and it’s just ill timed’.


AMOS: Yeah


CHRIS: What’s been going on with you?


AMOS: Oh, not much. Uh, just the other day, I released a new - we call them packages, libraries… 


CHRIS: I say library.


AMOS: Hex package…. I work with embedded a lot, so ‘library’ is overloaded. Everything is a library - the DLLs are libraries and things like that. SO’s… and there’s not a nice word like gems for Ruby. We need a name just for Elixir. Like, glasses, liquids.


CHRIS: Yeah, there’s some sort of cute word.


AMOS: Flask.


CHRIS: Yeah… But its… a limbic? There’s probably some alchemy related thing we could all use.


AMOS: So, it’s bamboo_config_adapter… You ever use Bamboo for email?


CHRIS: No, I make a point of not dealing with email. I make other people do that.


AMOS: The system I’m working on right now has alarms that go off and people need to know about them, because data centers could burn down if they don’t. So there’s lots of ways that the alarms are published everywhere, and one of the ways is to send out emails to people. The problem is that you have to be able to configure on the fly what SMTP server to use because some companies have one person has to use this SMTP server. Maybe they have an internal one. But then some other people have to use an external one. So every email you send, you have to use a different one. Bamboo’s not really set up for that. So I created an adapter that allows you to have your regular config that Bamboo uses, but you can pass in a config at run time and it’ll merge the two and override anything in the original. Which helps, but now I’ve been reading the new library guide ones, which is the new thing, right? And so, I have a Global config because I was following the rest of what Bamboo does, and I was reading the config part of the guidelines, and I was like, ‘you know what, that does piss me off, sometimes, that it’s a Global config overridden across the system, and maybe I don’t want it to be’. And I kinda wanna change my library, even though it won’t work like the rest of the Bamboo adapters for its configuration. It might be better in the long run. 


CHRIS: Configuration, man. I’m really happy there are things to talk about with the library guidelines, but the one that I can sort of stand behind and say ‘yes, please do this’, is the configuration stuff. And I’m of the opinion that libraries should just stop using it. Like, stop using configuration for all this stuff. Because it is Global. And it totally… configuration making me put stuff in mixed configs and relying on application.gitm to get it all out, because it totally invalidates the ability to spin things up dynamically, or to change things, or to change things at run time, or any of these things that you need in a real system. That’s the other half of the thing with the libraries I’ve been looking at. Half the problems are just because I can’t just pass you arguments. Like, just let me pass you arguments into your function. I don’t have any problem doing that, and if I want to manage my stuff with MixConfig or with Application.get_env, I’ll do that on my own. I don’t need the library to do that for me. Just let me pass you arguments.


AMOS: Right. Well, and then they’re not great with compile time, too, so they’re hard to change after.


CHRIS: Right, so there’s all these crazy hacks to get around it. And I know they’re trying to fix that. Presumably the next release with 1.7 or 1.8… I don’t know when. I don’t think anyone knows what the timelines are for anything outside the core team. So config in 1.7 or 1.8 is going to allow you to do things like System.get_env file. Which obviously you can’t do right now, unless you’re willing to bake it all in at compile time, which, obviously that’s not what you want to do. So that’s going to be fixed. But even so. Even with those enhancements, I still think its a bad idea to use mixed config, for libraries specifically. It's great for me when I’m building an application to be able to store stuff in there - that’s a great opportunity for me to be able to manage that - but trust me to manage it, library. You don’t need to do all this for me. Just let me start you in my supervision tree the way I need to start you, and pass you orderance.


AMOS: I think this is ‘lesson learned’, too, you know. If you look at the original packages, the original libraries that are out there, they’re all configured that way - Phoenix, Echo… I mean, Phoenix is the way a lot of people end up in Elixir from other places. So they’re like ‘oh, this is where you put configuration’.


CHRIS: Right, yeah, totally. And it definitely is a byproduct of the community and where we’ve come from. I’m happy that they’re making strides to fix it, but it's something that we as a community need to get away from. Because it's totally painful. When we were building systems a lot of time at work, we sort of start from this idea that - we start a lot more things dynamically than people have you realize, than the docs would have your realise or than it seems like people in that library encourage. We started stuff on demand when we need it. Or we start stuff once we’ve pulled environment variables out. Our application start is not simple a lot of times.


AMOS: Are you starting gen surfers or multiple application on the fly?


CHRIS: In some cases, full supervision trees for our app. It might be other applications, and we just start that stuff in our application starter manually - a lot of times. Just to avoid the mixed configuration problem. And to control how they start up and what happens when they crash and stuff like that.


AMOS: Like that whole startup thing - I deal with things a lot differently when I’m doing development that is web versus hardware.  You’re mostly doing what kind of development?


CHRIS: I don’t really work on any of the hardware stuff. I most work on services - typically GraphQL services or… more and more these days, its actually Kafka consumers. We use Kafka really heavily for our communication layer, but we’re really rebound. We don’t write that much to it. We use it for reads in a lot of places. And so a lot of that stuff is like Kafka consumers as well.


AMOS: That you’re starting on the fly?


CHRIS: Yeah, well, we’ll either start them dynamically - and a lot of times, the client library will handle ‘I need to spin out more workers’ or whatever, just based on how Kafka works. But even so, there are things that we start. The Kafka application - the client application - we start that ourselves. I’m a big subscriber to - Fred talked about this at Elixir Days, and he’s talked about it at a bunch of other conferences, and then Jesper Lewis also talked about this which is sort of this idea of layering. You know, you’re layering your system and you have sorta like stable system at layer zero, which means the thing is up and it’s running it all. And then there’s a layer above that, which is, it’s up and running, and then also doing somewhat useful things. The first layer may just be, ‘it comes up’, and if there’s dependencies, like you have to read from a file to start that application, then that file has to be there in order to achieve stable layer zero. But then there might be other things that you don’t care if they’re up or not. Like do you need a Postgres connection when you boot your application? Maybe, maybe not. And if you don’t need it - well, let's say that you don’t have it, well then you return an error, and you’re like ‘well, I’m not connected to the database right now, and I can’t return you or resolve’. That might be a way to think about your stem design.


AMOS: Right, why should I wait on that application.


CHRIS: And that way, if it goes away, which it inevitably will, on a long enough timeline with a big enough system...  I mean, now here’s the thing. If you’re building just another web app, you probably won’t experience that problem that often, but if you’re at any sort of a reasonable scale, you will notice when your database goes down, or the network drops out, or whatever the case may be. And at that point, you can drop back safely from layer one back down to layer zero, where you’re still running and you can still service real error messages, but you’re not… you haven’t crashed. You haven’t taken the site down. 


AMOS: Yeah, and I’ve been dealing with that a lot lately. And part of my problem is applications, right, so they’re not configurable supervision-wise like a GenServer is, so since I’m in embedded we use SQLite a lot, and sometimes Ecto won’t come up, because it takes too long, so it times out. So on our bootup, it takes down our system if we set Ecto to be permanent started because it’s an application, or it we set it to temporary, it goes away and we have to try to track that and try to start it again, but there’s not a supervisor so you have to write some way to watch it. Also seems like when doing firmware, there’s a very specific order that you need applications to start, and there’s not a good way to do that built into the system either, so what we ended up doing is Shoehorn and Nerves, that allows you to pick the order that things start, and there’s a poll request out there where you can restart that we worked on with Greg, and we’re using that branch, and it seems to be working well. It allows people to write hand orders to restart applications when they fail. But the bad thing is it feels very hacky, because you’re like taking over the application controller…


CHRIS: I mean, you literally named the library Shoehorn.


AMOS: Right, that’s… I didn’t name it that, Greg did. I’m sorry! Not Greg, Justin. Sorry, Greg, for making you look like Justin. Justin, sorry for calling you Greg.


CHRIS: So Shoehorn basically - if I’m understanding this correctly - sits in the middle of the boot process. Or, between your stuff and the boot process, so you can control how your disperate OTP apps can be started, or then restarted if they experience failure. 


AMOS: Yeah. I mean, its a super basic supervisor of applications, sort of. You don’t have configurations like you do with a supervisor. Instead, you write a handler, which is a module, so that you can do things way more complicated than you would at a GenServer level. But sometimes you need that. We have a DDoS, if you get DDoS on SSH, you can take down an SSH application. And if you set a temporary, that’s great, because you don’t want it to take down the whole system necessarily, but you may want to restart and put a circuit breaker on it. If you continue to be DDoSed, you might not want to restart it for five minutes or a half an hour, or whatever, so it allows you a little more control. In some ways it’s good, and it some ways, it just feels… terrible. Like, why do I have to do this, I must be doing something wrong. But I went with it. 


CHRIS: Yeah, when I start fighting against things, against the system, as it were, I do have to step back and wonder, like, am I the one who’s wrong here? Have I just fundamentally misunderstood something? I felt that way about configs for a long time, because I hadn’t talked to anybody else about it. I stood on a rock and would just skip using mixed configs, but it was a long time before I met other people who were like, ‘no, yeah, you’re totally right’. And since then, everyone I’ve known for a while is like ‘yeah, don’t do that. You should just skip that’. But it does; it feels like you’re fighting the system, to some degree. It almost feels like you’re being counter-revolutionary, trying to do these things, and that feels weird. It puts you at unease. The Beam ecosystem is stable, in a lot of ways. OTP gives you good guarantees for this stuff, so whenever you fight it, I think it adds to that feeling of ‘I really might be doing something really wrong here, and I just don’t understand why’.


AMOS: Yeah, but the good thing is, it almost slows down the progress and makes you think about what you’re doing and how you’re doing it rather than being like ‘I’m right, let’s go’. And maybe come up with a better solution. I know the whole time we were working on that Shoehorn, I was like ‘I think we’re doing something wrong here, but I’m going to keep going at it. Lets see what happens. But I feel dirty, guys’. And they were like, ‘yeah, me too!’. But I’m glad I’m not alone. But, I mean, now with the config thing, like you said earlier, the library guidelines are now saying ‘don’t use mixed config unless you have to’. And really think about it. And there were some other anti-patterns in here - uh, don’t use exceptions, for flow control. I would say avoid using exceptions almost entirely. Unless you’re talking to a low-system level, you probably shouldn’t be raising any exceptions.


CHRIS: The only place that I use them in a library is in Wallaby. And we do it in order to throw specific exceptions for tests, so that we can crash the test process with a meaningful error. We have a handful of exceptions that we’ll throw to kill the test process, and format the errors nicely, and provide context for the errors.


AMOS: Yeah, it gives you a lot more than a bad pattern match error, right?


CHRIS: Yeah, or just a failed assertion. You don’t know why. Because you’re just like ‘I want the page to have these CSS selectors and it’s just like this’, and then it’s like ‘nope, false’. And you just go, ‘okay, why? Why did that happen?’. So we use exceptions for that. But I do think that generally speaking, especially for flow control like what they’re describing, I’d totally agree with that. Its… you don’t want to do that. You want to just let things crash, most of the time, by having strict pattern matches, and if it doesn’t match, you just blow up.


AMOS: I find that that’s hard for people around me to deal with, that haven’t been doing Elixir or anything. I’ve kinda embraced it from day one - like, let it fall apart and see if we can make it restart okay, and then after that, then we fix what made it fall apart, if we can. Because if we can restart okay, we can overcome a whole host of other things that could happen other than this specific instance. But that really… seems to be hard for people to let go of.


CHRIS: It’s one of those things - I’ve noticed this problem as well, because there are libraries out there - and again, I don’t want to point fingers or anything - but there are libraries that we’ve tried to use before that don’t handle failure at all. But they also don’t handle supervision in a way that’s meaningful. We’ve had problems where we just let stuff crash, and it’ll take the app down. Because, like, the supervision stuff that they’re doing doesn’t accommodate these things not working. It doesn’t handle crashing and stuff like that.


AMOS: Like Ecto timeouts on startup.


CHRIS: Yeah, there’s just certain things in the community that we haven’t fully come around to embracing yet, and we need to do to level that up a little bit. And I’m not here to say - ‘let it crash’ is not like a panacea. You can’t do it for everything. Certain pieces of your application you need to be more fault tolerant than that. But, it’s a pretty good way to solve a lot of problems - a lot more problems than I think people are letting it solve for them.


AMOS: Yeah. I think that’s one of those things that comes with experience from designing programs with OTP. I mean, from day one, I embraced it, but I failed at it horribly, because I didn’t really understand how to deal with it.


CHRIS: I still get it wrong all the time.


AMOS: So one of the things with just letting it fail - they suggest using the okay error tuples as returns instead of exceptions, which will throw, like, the bad match exception, but one thing that I’ll get really frustrated with is the error tuple is usually like, its error in term, and a lot of times in the second term, somebody puts like, string, or an atom, and I’ve started putting another tuple there, and putting an atom, and putting a string that explains what’s going on. That way, if I do want to handle it, I can match against the atom pretty easily. But I have a string that gets displayed instead of like :eaddressnotfound. 


CHRIS: And so this is another place where I feel like exceptions are really good. Because what I’ll do a lot of times these days is I’ll return the error tuple, but the second piece of the tuple is an exception. And so the exception has - exceptions are just a data structure, and so what it’ll have on it is all the context that you need, and it has a nice message if you do want to raise it. If you decide you want to raise the thing later on, then you just raise the exception, and you're done. I typically like that, these days.


AMOS: So you’re just creating exception data structure but not raising it, just passing it in. That’s a good idea. I hadn’t thought about that. That gives the best of both worlds for me.


CHRIS: And then the other thing is, you can get it to - one of the things I try to avoid doing is using an error type that is error term, because then dialyzer can’t help you as much. So I try to make those errors useful for dialyzer as well, so at least dialyzer can tell me if I’ve overspec something, or if I’ve got my specs slightly wrong.


AMOS: That’s something I wish was in the library guidelines - is use dialyzer. It gets more important in a library than in an application. You do whatever you want in your application but library operators not providing that gets kinda frustrating sometimes, so I end up wrapping libraries in a layer, just to put some specs on something, instead of just trying to infer it.


CHRIS: Dialyzer is one of those things that I get why people don’t use it, because, if you’re a type nerd - or, if you really like ML types, or you’re really into Haskell or something, then dialyzer does feel less powerful than that, because of the success typing thing. And if you’re just a practitioner that’s trying to come into types, you don’t have enough to go on to understand what dialyzer is doing for you. Like, for me, when I came across dialyzer, I just had to go read the paper, and then I read the paper enough times that I was like ‘okay, I get what its doing and I get why these error messages are like, phrased the way they are’, and that kind of stuff. But it can be a bit cumbersome. Luckily Dialyxer has gotten better, and the documentation for it has gotten better, but it’s still cumbersome. Like, you have to build the PLT, you have to know what a PLT is, you have to know why you have to build it, you have to learn a little bit about these transitive depths, and stuff like that, in order to get it all to build correctly. It can be cumbersome, for sure. And it takes a non-trivial amount of time to run if you’re running it cold. Like, if you wanna add this to your CI step, it’s going to add a non-trivial amount of time unless you’re going to cash PLTs and stuff.


AMOS: That’s where I get complaints from people. They’ll run it once, and they’ll be like ‘it took an hour!’ Or, I had some C developers that put the types in the specs and didn’t realise you had to have an extra tool for it to really be anything other than informational, and they’re like ‘I could still pass in a string’. And I was like, ‘well, yeah…’ It’s not a protection against bad code, unless you’re running dialyzer, then you can't ignore it either. Then the other complaint is that some of the messages out of dialyzer are so hard to follow.


CHRIS: Yeah, they’re completely opaque if you don’t know. If you haven’t been taught, or you haven’t read the paper, they make no sense, because they don’t describe a real problem, typically speaking. A real problem in people’s minds, anyway. Like, ‘what is a no-local return? What does that mean?’, well I can kinda start to piece together what that probably means, but why is that a type error? It’s hard to sort of join these desperate concepts together, and I think because of that, there’s a non-trivial amount of cognitive overhead to learn dialyzer. Which I guess maybe is part of why they didn’t add it to the library guidelines, because it is difficult, but I also think we’d generally be better off if everybody was doing it, because, number one, the libraries would be checked, and number two, if the guidelines were like use the thing, and it wasn’t fun to use the thing, then I suspect we would get to the point where it would be good to use the thing. Because people would be angry enough or frustrated enough to go work on it, and to go make the error messages better.


AMOS: And we might get good docs around it too. I think that one of the frustrating things for a lot of people, and was for me when I first started looking at it, was that the syntax that it puts in the error messages is really Erlang syntax, and, although I might have a decent understanding of Erlang syntax, I might have to translate it when I’m looking at the Elixir code side by side to figure out what’s going on, and sometimes that's frustrating. I’m like, ‘okay, if they’re both in Elixir syntax or both in Erlang syntax, but when I’m trying to talk between the two, it’s just a lot of cognitive overhead for me. 


CHRIS: No, I completely agree. And apparently there’s a way for us to get in the middle of what dialyzer returns, versus what we would like to see from an Elixir perspective, and to change the error messages. I think that’s, like, a supported thing. So it probably just means that someone needs to go and do the work to make it happen, and I just suspect that people have lives, it turns out, and just haven’t done that work. But, yeah, I think that could be a super useful tool.


AMOS: I know that there’s this guy who sees wrong things in the community and just wants to fix every library, so… Maybe you could talk to him. 


CHRIS: But specifically about distributed systems stuff, as it turns out. 


AMOS: Ah, man, oh well.


CHRIS: The one library guideline on here that I think is really cool and a really useful thing - because it leads into something I’ve been really happy with - is the ‘avoid spawning unsupervised processes’ thing.


AMOS: Yeah, see I haven’t run into somebody doing that very much. I can hop right on board with that, I just felt like ‘oh, yeah. I don’t ever do that’.


CHRIS: I have definitely done that. And, in fact - this is the one that I’m the most guilty of, because inside of the raft implementation I’ve been working on, the RPC stuff is done by just spawning a new process. Instead of using like a task or something - although the original implementation was just like it spawned a process and away it went. The problem with doing that is just - if you need to shut down, you have all these processes that aren’t linked to anything and can’t be shut down gracefully, they just get orphaned, basically. That’s why you should avoid that. And the guidelines talk about it even more. But the thing that I’ve been super happy with - and I’ve been using it a lot, but I haven’t talked to other people who’ve used it that much, but I use it constantly, is the new dynamic supervisor thing. Its great! It totally solves all the problems that I had with simple one for one.


AMOS: What are the problems you’ve had, because I’ve been kind of… I’m kinda torn by it. So I used it for the first time the other day, and I actually liked that simple one for one was set up to run a specific GenServer. I have one module that it will start link on. And then dynamic is like ‘just throw the server at me. I’ll take care of it. I’ll watch it.’ Which is like a love-hate for me. Its nice because its quick and easy, but I really like the explicitness of the simple one for one.


CHRIS: You can set up a dynamic supervisor to do… with like, a start child, kinda thing. And it’ll do what you’re talking about doing, where you can sorta say, I wanna start this, all the time, so when I start it, just start it for me please. And you can do that. Because of that, it works really really well with child spec, which is also a thing that I’m a big fan of, and so I think - I mean, I don’t know. I’m not on the core team, don’t @ me - but I think that was one of the reasons that they built dynamic supervisor - is to get it to play well with the new child spec stuff. So you should be able to do what you’re talking about doing. But now it’s actually more explicit, because you’re intentionally starting a thing that is… where you know the children are going to be dynamic children, and it has some convenience functionality around that as well


AMOS: Oh, okay, I didn’t notice it in the documentation before - the children thing. I just went and looked when you said that.


CHRIS: Yeah, its really cool! It’s really nice. And this gets back to this idea of like, to me, the thing we were talking about earlier with configs and stuff and sort of layering systems is a lot of the stuff I start working on, I want to be able to start dynamic amounts of children, like I want to be able to control that stuff, and this is a really useful tool for that stuff.


AMOS: Now I wanna go change the code I wrote the other day. That’s what I always… yesterday me was an idiot. Today me is better. Tomorrow me is going to think today me is an idiot. If you’re not growing and learning like that, I feel like you might be in the wrong field, because there’s so much to learn. You shouldn’t look at code you wrote three months ago and think ‘that is awesome’.


CHRIS: I tend to - the way I tend to approach it is I write stuff and I think ‘you know who loves fixing things? Future Chris! Future Chris loves fixing these things. It’s like taking out the trash. You know who loves taking out the trash? Future me. That guy loves trash. He loves taking out the trash.


AMOS: That almost sounds like procrastination when you do it that way, though. Like, tomorrow me can take out the trash.


CHRIS: It’s - you know, its a growing thing.


AMOS: Yeah, yeah. I like it, though. I might use that. Just make sure you leave yourself enough bread crumbs to actually remember what you’re doing. 


CHRIS: Yeah, I actually try to… see, so it’s funny, I worked with a guy for a while, one of the smartest people I’ve ever really worked with - he told me one time, because we were trying to dig back through some code that he’d written like forever ago, and neither of us understood what was going on, and I was like ‘what? What were you… what was going through your mind when you did this?’ And he was like, ‘actually, I kind of enjoy the process of digging through all these things, and I tend to kind of… wander around. Because it’s kind of fun for me to try to relearn how I got to these… how I made these decisions’. I was like ‘okay, man. That’s cool.’ 


AMOS: I mean, maybe it works.


CHRIS: I mean, yeah, no, if you like constructing your own sort of like, cathedral of madness, that’s great.


AMOS: Ohhhhh….. What else do they have in here? Anti-patterns, invalid data, up front, top of the library, be defensive at the entrance to the library, so you don’t get way down in the weeds before they have an exception. And the example that they give about at least making sure that the file to write to is binary at the top level is pretty good. I kinda feel like if you can’t do it with a guard clause, it’s going to get hard to do because what happens when that - maybe you don’t read that file for a while, and… you can’t check the file exists at the top. I guess you could, if the file exists when you check it, but by the time you actually try to read it, it doesn’t. I feel like deciding what to check at that top level is tough. Or how deep you go in checking it.


CHRIS: Yeah, totally. I have this problem a lot, actually. Because I think about it in terms of, like, you sort of wanna do… I have a tendency to think about designing systems from the point of view of doing all of the .io side-effect-y stuff, either up front, gathering all the data from all these different places, whether its accepting data from a web request or whatever - all that kinda gnarly .io stuff. And then you provide validations around it immediately. But yeah, because what you wanna do is protect your stateless inner core, as it were, so you do all the gnarly .io stuff in one big block, and then pass that into stateless things that you can sort of check, and you don’t have to assert ‘oh well does the file exist in the middle of this what should be stateless operation. And at the end you do more stateful stuff, where you’re just like ‘alright now that we’ve got answers now we write that back out to disk, or we send it to a database, or we respond to the web request’.  That’s how I’d like to approach it, but honestly, that’s hard. That’s hard to do. Because the majority of what we work on, it tends to be side-effect-y. It tends to be stateful stuff.


AMOS: A program with no side effects is a pretty boring program.


CHRIS: Right, you know. It doesn’t do a lot. So, I dunno. It’s tricky.


AMOS: So, are you doing - are you having a, I guess... let’s say you’re going to read a file, and then write a file. So, you read the file, then go through a bunch of chains, the write the file down at the other end? Or does that data come back up to the top, so you’re reading and writing at the same level. Like, is all of your .io at the top, or do you have it like, sandwiched in the middle? Do you know what I’m saying?


CHRIS: Yeah, I get what you’re saying.


AMOS: Like, read file, pass it to something, get returned something, then write file in the same function or is it opposite ends?


CHRIS: I think that’s typically how I would look at it. Is like, you have one function that is sort of, its sort of the glue, ya know. It’s the canonical controller in like an MVC kinda way, or ya know, it might be like your Phoenix controller or something. You have data coming in, you need to handle. Let’s take your file example. So we have some process that kicks off that reading from a file thing. So we then go and we read from the file and that’s all side-effect-y and it could break. We don’t know. So you read from the file, and then you need to do some transformation on it, which is a stateless thing, and then you need to write contents back out. So what I would typically do is I’d have like a module that is typically the stateless stuff. It’s just data, it takes data, and it returns data. And it doesn’t even have to be a process. It’s just some bag of functions that does that transformation step. And it doesn’t hold state on its own, it doesn’t do anything. It just takes data in and gives you data back. And then, I would have a process that actually does this sort of like, the reading part, ya know, this is in a vacuum where we don’t need to read like, this is assuming you don’t need to read gigabytes of data, or you don’t need to use gen_stage or any other stuff. So you’d have some piece of state somewhere that knows how - like, some process somewhere, that knows how to read the file, and then it passes into the stateless stuff, gets it resolved, then puts it in back out to the file and then writes it back to disk or whatever. So all that happens in a single place, and it can all be supervised and it can crash and it can do all the things it needs to do, on its own, isolated away from the rest of the world. I think there’s a bunch of benefit in doing that, because if you do that, then all the real business logic - the stateless stuff - then that can all be, you can really thoroughly test all that. It’s really really easy to test that. Even better, you can property test it all, to really sort of assert that you got it all right. Then you can sort of do one big integration that’s sort of like ‘hey, can you read from the file, can it put things to the file, does it look more or less like what I think it should look like…’


AMOS: Cool. I think there’s a tendency I’ve seen to not put that impure - I’m going to use the term ‘impure functions’, so functions that have side effects - in the same place. I see a lot that you have an impure function that passes to what I would say - the function itself is pure, but it passes to another one and they keep passing down until the very task function, is in a chain, is impure also. So instead of it returning all the way back up, it continues to pass down. And when its returned all the back up, the thing that you get returned is like ‘yes, I’ve brought that to the database for you’. So you, like, turned all those semi-pure functions into impure functions, because the very end of the chain is impure, so now the whole thing is. Making it hard to test.


CHRIS: Well, so, the sort of functional programming way I would think about that is you… you’re taking a set of impure things, and you’ve got some pure things - presumably pure things - and what you’re doing is you’re lifting the pure things into the context of file.io or whatever. You’re basically into the context of doing impure things.


AMOS: So we might need to define lifting. I mean, I’ve been reading a lot about monads, so I’ve heard it enough times, but I think I need a little more.


CHRIS: So lifting just means, in this context, in the way I’m using it right now, I would say lifting is taking a program, or a set of functions that can form like a program, and running that program in the context of impure side effects. And so, that’s kind of the beauty of doing these kind of… I don’t know, I don’t want to talk about monads that much, but its kinda the beauty of this design, I would say. You can build a whole thing that is purely stateless that just takes data in and gives you data back out. Except, as we said, without side effects, it doesn’t run, it isn’t useful. So you have to use side effects. But once you have that pure stateless program, that takes data in and puts data back out, you take it, and then you run it in the context of doing file.io, and that process of taking it and running it in that context is referred to as lifting. Like, we’re lifting it into the context of .io, or something like that.’


AMOS: Okay, so it's… one thing that I've seen that I’ve worked with and I’ve done was thinking that I’ve done something pure, when I’ve sent a message to another GenServer. That’s not a pure thing. It just fell through. Because I was like, ‘oh, I’m still inside of Elixir’ and then I was like ‘oh, no it wasn’t’. If you’re changing state of a GenServer or even just sending it a message, once it leaves you, that act of leaving is just like writing to the screen. Bringing that back up to the same impure level you were saying. It’s like an impure shell around your application.


CHRIS: I mean, yeah, totally, that’s one of the funny things about Elixir and Erlang, in their history as languages. There’s really not a lot about… you have to go out of your way to write pure functions in Elixir. By default, that’s not what we’re going to do. And the thing is we do impure things all the time, like sending message to processes or any of this kind of stuff… but it turns out that’s okay because we… there’s a lot of benefit in doing that, and it’s just part of the way… we guard against that, using things like supervision trees, and using these other constructs that OTP gives us, and I think that’s like a really cool pattern as well, like I’m happy using Elixir. As much as I love Hascal, I’m happy using Elixir for production because it lets me do these useful things and doesn’t get in my way when I need to do this stuff. But it’s a funny trade off. 


AMOS: Yeah. And not so fun. Sometimes. I think it gets… it just gets hard to recognize until you’ve practiced it a while. Not the benefits of it. I think that, for me was easy, but recognizing that ‘oh, shoot this is impure. I should probably put this somewhere else’. I figure it out when I’m testing.


CHRIS: Testing is a great way to figure that stuff out. Testing is a really good way to prove, ‘oh yeah, this not… this isn’t going to do what I actually wanted it to do’ or ‘this is obnoxious to actually test all this stuff, how do we decompose those things’. And side effects will typically be the thing that indicates that. Whether you’re sending a message or writing to a file, its like ‘its a pain to test that. We should isolate that or decouple these pieces’.


AMOS: I wonder if this kind of stuff should end up in the library guidelines. That was one thing I remember missing in there is that there was nothing about testing that I remember seeing. Under publishing, it says write tests, but it's brushed over. That’s my issue. 


CHRIS: Oh, gotcha. There were a couple things that I thought were missing from this, and one of them was writing tests and talking about… not the art of writing tests, but some guidelines on how you’d want to do that, et cetera. But the other one that I was surprised about, but not necessarily disappointed that there wasn’t anything in there, was that it doesn’t say anything about the formatted. Like, it doesn’t say anything about like you need to use it or whatever. I guess they assume that people are using it, which, that’s fine.


AMOS: Yeah, I’m actually happy that I’ve seen less whole requests and stuff taking time working on the formatted… I’m not trying to bash the formatted. I’m feeling that there’s actual movement on Elixir itself, where the formatter just didn’t feel the same. I didn’t get the same satisfaction from seeing the poll request for the formatter, because I read a lot of the Elixir poll requests - that’s my OCD thing - there’s also so many of them that I can’t read them all - but it’s nice to not see every day a new formatter poll request or complaint.


CHRIS: There were so many issues for a while that were like ‘why does it not do this thing’ or ‘why did it do this or that’. I mean, for a while there was legit bugs, like it broke - so for one of our apps at work, we’re still on Elixir 1.4, and we ran the formatter on that code base, and it broke it. It wouldn’t compile any more with 1.4. So they fixed that. That got fixed in a patch release. But for a while there was legit bugs. Like, it just broke. There was one thing where we ran the formatter on the file, and then ran it again on the file, and got different results.


AMOS: I had that happen too. I was like, uh oh. If you format something that you just ran the formatter on, it shouldn’t change anything.


CHRIS: Yeah, that was our thinking as well. So that was concerning. But, yeah, it seems like they’ve fixed most of that stuff. And it seems like the formatting, in the opinions part, it's like everybody sat their opinions down. Those issues have been closed, as ‘will not fix’ or whatever they’re going to be. Maybe some of those opinions made it into the formatter, maybe they didn’t. I don’t know. I don’t use the formatter.


AMOS: I noticed some of them did. Some things changed, but a lot of them are like ‘sorry. This is what we decided’. And I get it. I get what their goal is. And it does make it nice for them, working on core, is they can just be like ‘formatt. Now all of core is consistent. If you don’t like the way core looks, you don’t have to use the formatter’. But it took me awhile to get there. Like, at first, I was angry. Like, ‘why are we spending time on this’ sort of thing. But I can’t complain, because I wasn’t spending my time on it either. I might have been reading poll requests, but I wasn’t doing it.


CHRIS: Yeah, I mean, kudos to them. I’m glad its a thing and people seem to be really excited about it. I remember when they announced it at Elixirconf, a lot of people were really excited about it. Which, I found surprising, because that stuff is not… it’s a nice thing to have, but its not compelling to me. It’s not like a thing that I was really burning for. But I mean, people seem to like it. So that’s fine. Like I said, we don’t use it at work. It’s like one of the only universal decision my team made, where we didn’t have any arguments. We just chose not to use it. Which is pretty funny. But yeah, if people are liking it and its useful for them, that’s good.


AMOS: Yeah, I was surprised that there was so much excitement over it, mainly because style guides and things like that seem to be this emotional thing for a lot of people. So I was like, they’re going to feel like they’re losing it. That’s not what I like. And, we’ve used it on a couple files, just because they were so bad, and certain people on the team get super uptight about the way things go, and I was like, well, that’s a really old, giant file. Go change it to however you want it to look. Just make it consistent. At least in that file. I like that it keeps things consistent. I think it would be a cool tool. I think it would be really hard to write… if you could create a style guide and then have the formatter read that style guide and do the thing.


CHRIS: Yeah, and it defeats their goal, which is, they don’t want it to be configurable. They don’t want you to be able to change it, because then nothing is consistent. Which I totally buy as a design goal. So, that seems reasonable.


AMOS: Yeah. And I think its good for open source, because it gets rid of the bikeshedding argument, for them. Well, but then there’s arguments about the formatting itself, but if I’m a library maintainer and I use the formatter I can be like, ‘hey, not my problem. Talk to them’. So I don’t know overall how you feel about the library guidelines. There’s nothing in there I'm against, I don’t think.


CHRIS: Overall, I really think these are good. Actually, I don’t think there’s anything in there that I disagree with. These are definitely all problems that we’ve encountered through work and whatever else. They’re really good. It’s definitely stuff that, if you’ve been doing this a while, none of it is going to be shocking, but all of its good to read. And, I hope that people start taking this stuff into account and start using these ideas because I think I’ve encountered every single one of these in a library, at some point in time. And os, the more that we avoid these kinds of pitfalls, the better we’re all going to be. So, yeah. I like it. I’m happy that they put this together. That’s good.


AMOS: I just want more about testing. Maybe we need testing guidelines also.


CHRIS: Yeah! Maybe someone should work on that. I’m sure no one is going to turn that poll request down.


AMOS: No, probably not. Although they may say ‘just put it in the next unit of documentation’, but it feels… but the act of testing feels different than how to use specific  parts. It’s tough. Actually, I’ve been tracking a lot lately, you know, proper testing, proper check, and your discussion on stateful generators, and I’m totally going to mess this up - I’m just going to start out completely messing up somebody’s name on purpose, because otherwise I’m going to mess people’s names up later, and they’re going to feel bad. So I’m just going to mess em all up. So, uh, Clahsss Alfrek - so it’s Claus Alfred, maybe? - he had some pretty good stuff in those PRs too. We should link those.


CHRIS: Yeah, I feel like we should do a whole discussion at some point about the state of property testing, and - I’ve had this discussion with a bunch of people, and I want to get more people involved in it. I think it might be a good topic for the future, to go through and talk about where we’re at with property testing and Elixir and talk about how that might be changing, and what we might be able to do with it in the future too. 


AMOS: Yeah, and… what was it? 2007? Elixir days, that you talked about property testing, using what? Quixir?


CHRIS: Yeah, Dave Thomas’s library.


AMOS: Yeah, that was a pretty good - I really liked it. I think it’s good instruction for people. And then maybe they can move on to stateful generation and maybe generating events instead of just data to pass into a single function. Because I think that’s where the real power come it.


CHRIS: Yeah, absolutely. Like the model checking. The stateless properties are fun, they’re nice, but the magic is in model checking, its in the stateful stuff. And its… it totally… it’s humbling, but it feels so good to have that stuff.


AMOS: Anyone that I’ve talked to that has done property testing for any amount of time brings up the humbling fact, and then… It’s humbling in two ways. It’s humbling when you find out the code you thought was well tested and really well written has these crazy horror cases that you’ve never thought of. And then the second thing is that just coming up in the property test and the actual properties and the amount of thinking that takes. You’re like, man, I could write tests all day before. Now it takes all day to write a test. 


CHRIS: No, it's so hard. The property tests are so much harder to write. Especially if you want them to be useful, to be good, and to string properly and to give you real results for stuff. They’re so much harder to write. But that one test. The mileage on that one test is so good. So so good.


AMOS: Well, I can’t wait for us to talk about that again. I know that you’ve got stuff going on today. I need to go and my intern - I wrote on my door to my office ‘on air, leave me alone’. And I moved his laptop out because he wasn’t here when we started. So I moved his laptop out of the room and was like ‘you can work in the break room’. Because I’m recording. So we should probably…


CHRIS: We should button this up. We’ve gone all over the place, generally speaking, library guidelines. We’re for ‘em. 


AMOS: Yeah, I think they’re pretty good. And, ya know, this is our first episode. And we’re missing somebody. I think we should let that person introduce themselves. 


CHRIS: That sounds good to me.


AMOS: So, if anybody actually listens before that other person shows up, they can make guesses and wonder. I always listened to Ruby Rogues back in the day, and then when I was on This Agile Life, we had picks on the end of the episode of something that has your attention. Doesn’t have to be technology-wise or Elixir-wise.


CHRIS: Yeah, I mean, honestly, if you want to learn more about property testing, I think the best resource out there is Fred’s book, Proper Testing. It’s available at propertesting.com. We can add a link to that as well. Go read through that, because it’s great, and it’s written in that sort of style that he has. And it’s great. You’ll learn a ton, just by going through that. It’s in Erlang, so if you’re coming from Elixir, you gotta do the translation, but the translation is pretty easy. And I think that generally speaking, the principles you’ll learn from going through that are fantastic.They’ll really give you a good sense of what to do.


AMOS: Maybe somebody should go translate all the coding samples in there.


CHRIS: I think that’s… I think that's a thing.


AMOS: And its funny, because it seems like everything Fred does I’m like ‘hey, you should look at that’. So I had to teach somebody about tracing, and because we’re running into too many log statements just killing the system and its hard to read the code because there’s log statements in everywhere. And so I was going to show the dbg and sys, and the things you can do with that, but dbg is cryptic and really hard. It kinda… all the methods are like mpc, nlp, is this stuff. Recon is a library by Fred. It’s also in Erlang, but since we can use that anyway, it has a much nicer user interface. And dbg is super powerful, and you can take down your system if you mess something up while you’re using it. Recon tries to protect you from that a little bit, like, its prod safe-ish. Because it probably is not going to shut down your system entirely. But you can’t trace everything. If you try to, it just tells you no. So, check out Recon.


*outro music*


AMOS: Hey, everyone! Amos here. Thank you for listening to the first-ever episode of Elixir Outlaws. Thank you to my son, Noah, for the music for the podcast. I just wanted to let everyone know you can find us on twitter @ElixirOutlaws or on our website, https://elixiroutlaws.com. Again, thank you for listening.