AMOS: Welcome to Elixir Outlaws: the hallway track of the Elixir community. Episode 2: Is this the show?


AMOS: Three… Two… One… Mark


CHRIS: Close enough.


AMOS: I’ll be able to find it ‘cause there’ll be enough silence in there. It’ll be alright. Oh man. So, when I got up this morning, I started down my driveway taking my daughter to school, and then suddenly had my - my whole dash lit up, all the lights on it. I was like, ‘what the hell is going on with my car?’ … The back tire was completely flat. So I spent this morning pulled over in the mud trying to jack my car up with a screw jack. Worst invention ever.


CHRIS: You should have pulled over not in the mud.


ANNA: Yeah, that sounds terrible.


AMOS: Well, I didn’t have a choice.


CHRIS: It was a bad choice. Of all the places to pull over, that’s probably the worst place to pull over.


AMOS: … I had no choice. 


ANNA: Well, you got it though. Everything is okay now.


AMOS: Yeah, we’re good now. I ran my car over - that’s the reason I was late - to have them patch the tire.


CHRIS: Well, hi, everyone.


AMOS: Hellooooooo…


CHRIS: We’re all here.


ANNA: Hellooo!


CHRIS: This is good!


AMOS: And Anna’s here!


ANNA: Yeah! Heyyy! 


CHRIS: So the biggest - Is this the show? Are we in the show now?


ANNA: Yes, we’re in the show now.


AMOS: I thought we were before this.


CHRIS: Oh, okay, yeah, yeah, okay, excellent. I’m just making sure. That we’re in the show now. So… first of all, I just wanted to say thanks to everyone who listened - a surprising number of people listened. So thanks, for that. And, the biggest piece of feedback that we got is that we didn’t actually introduce any of ourselves. And I will go ahead and say, the real excuse is because we weren’t all here, and that would have been rude. The actual excuse is that I forgot. I dunno about you, Amos.


AMOS: I just assume everybody knows who I am.


CHRIS: Yeah, see, I’m not at that level of arrogant yet.


ANNA: I think that’s a safe assumption, Amos. Totally safe assumption.


AMOS: Well, when you have a name like Amos, sometimes, when people meet you, they’re like ‘your name is what?’... Yeah, it’s an ‘m’, not an ‘n’. 


*laughter*


AMOS: You’re welcome.


CHRIS: Alright, we’ll start… You wanna start, Amos?


ANNA: Yeah, do you wanna start?


AMOS: Yeah… Well, I dunno. Um. I’m Amos. I did a podcast called This Agile Life for a really long time. Audacity is stupid. I think it tries to up my recording volume based on how loud I’m talking. So, anyway, I did This Agile Life for about five years, I do freelance consulting. I love Elixir - or the last year I’ve been doing Elixir full time on embedded devices using Nerves. 


CHRIS: You have to pick someone to go next.


AMOS: Oh, is this… Is this popcorn?


ANNA: Chris, you go.


AMOS: Anna picked for us. She’s the leader. 


CHRIS: She’s got places to be.


AMOS: She’s awesome. 


CHRIS: Yeah, I’m Chris Keathley. I’ve been hanging around the Elixir community for a while now. I’ve done a couple talks and those kinds of things. I help maintain a browser testing library called Wallaby, and recently I’ve been working on a lot of distributing systems stuff and just some other things like that.


ANNA: That’s rad.


CHRIS: Anna, I think by process of elimination, you’re the last one, so you have to go.


ANNA: So it’s my turn? Yeah, I’m Anna. I’m a developer at a consulting shop called Carbon 5. I’ve been doing Elixir for a couple years. I started Elixir Bridge. We try to put on workshops - free workshops - for unrepresented folks in tech to learn Elixir in Phoenix. Yeah, very fun. 


CHRIS: Very recently gave a talk about building a blockchain in Elixir.


ANNA: Oh, yeah, I’ve done some speaking. Yeah, I was talking a little bit about building a blockchain in Elixir.


CHRIS: And… Erlang factory now… code… beam… I don’t know what the name of it is anymore… Its Codebeam, right?


AMOS: They keep changing it.


ANNA: Yeah, it’s Codebeam now. I’m going to Stockholm, actually, to give it one more time.


CHRIS: Oh, nice.


AMOS: Man… I need to start applying for overseas conferences. So that’s actually where we all started talking about making a podcast, was a little bit at Elixir days. So I met Chris, what, two year ago?


CHRIS: Yeah, back in Florida.


AMOS: And then, this year, I met Anna, who gave a fantastic talk - you say it’s about blockchain, I say it’s about GenServers. 


ANNA: It’s kinda both.


AMOS: It was fantastic. I enjoyed it.


ANNA: Almost as fantastic as you creating lightning on stage. That was rad.


AMOS: Well, when you have no real content, you have to do something crazy.


CHRIS: I dunno. You did bring a Van de Graaff generator on stage and make lightning. That feels like content.


ANNA: Right?


AMOS: I loved it. My whole talk was just gratuitous BS so I could make lightning. I really wanted to shock somebody, and it didn’t work out for me.


CHRIS: The talk was a vehicle for making lightning on stage.


AMOS: Exactly!


CHRIS: I think that’s a good excuse.


ANNA: No, totally.


AMOS: I’ll take it.


CHRIS: We did have some pieces of feedback that I wanted to get to before we go into any sort of main topic here, that we got after the first show - they just reached out to me and let me know some of the things that we got wrong, and also helps to provide a little more context for stuff. So, first of all, its not Ekto’s fault that your SQLite app is crashing when it can’t talk to SQLite fast enough. I guess it’s apparently because the SQLite3 driver doesn’t actually support async connect. And when I say driver, I don’t actually know if that means the Elixir diver can’t do that, or if there’s some underlying reason that that’s happening, but that’s the root cause.


AMOS: I’m not sure that SQLite can do anything Async.


CHRIS: Yeah, right, that was actually one thing that I was wondering. I don’t know where the root of the problem is, or if it’s fixable in Elixir land or not, but that is… that’s where that’s at.


AMOS: Yeah, I’m not sure either. We attempted to fix it… with adding a timeout of infinity.


CHRIS: The best timeout.


AMOS: Right.


ANNA: Did that work?


AMOS: Well, it’s working. I don’t know for how long, ya know? I kinda fought against it, but at the same time, if it’s not working, the system might as well shut down - the whole thing. Also, they’re like, ‘well what if we pick something arbitrary’. You know, if you pick 30 seconds, someone is going to have something that takes 31 seconds and they’re gonna be like ‘well, why can’t we just make it 31?’


CHRIS: I like five years. I pick five years as my timeouts. Five years is the maximum amount of time I’m willing to wait for this database to come up.


ANNA: Sounds totally reasonable, guys.


CHRIS: So, that’s that.


AMOS: Yeah, that was great feedback. And I went and talked to the person you got that feedback from - I’m not sure we should say who it was. But maybe we should.


CHRIS: I did already. Did we compromise his security?


AMOS: Yeah…


CHRIS: Yeah, no, sorry. I thought I did…


AMOS: It was Jose. He let us know that we were wrong. And we appreciate it a lot, actually. No, I’m not trying to ping on Jose. It was amazing that he reached out and let us know.


CHRIS: The other thing he informed us about is that the core team is working on dialyser improvements. Specifically, I guess, on messaging, and making sure that the errors look nice and stuff. But. Its probably - it’s a bit trickier that I originally thought, it’s a bit trickier than anyone originally thought. They have to change some stuff in OTP to make it happen. But it is on their to-do list of things. I don’t think there was a specified release date, or what version it’ll be in, but it’s something that they’re definitely working on. So that was cool to hear too, ‘cause I think that will be really beneficial to the community, so it’s cool to hear that they’re working on that.


ANNA: Yeah, that is really cool. Any ideas when? Did he mention when that was gonna happen?


CHRIS: No, but based on the extent of the work, like kinda what he was describing will have to happen, I suspect… it’s not even tied to Elixir at that point, because they have to change stuff in OTP to make it happen… So they’re gonna be beholden to that, even once the work is done.


ANNA: That makes sense.


CHRIS: So it’ll probably be a bit, but the fact that it’s happening at all is pretty rad.


ANNA: No, that’s totally rad.


CHRIS: Yeah, so if those things are interesting to people and they want to get involved, they would loooove contributors to help out with that effort.


ANNA: Totally


CHRIS: So, should we dive into our main topic for today?


ANNA: Yeah, let’s do it!


AMOS: Sounds good!


CHRIS: I think last time we said we wanted to talk a little bit about property testing.


AMOS: That shouldn't take us very long. It’s not like that’s a complex topic or anything.


CHRIS: No, definitely.


ANNA: People don’t have feelings, right? Yeah?


CHRIS: There’s not just a multitude of papers that have been written on the subject… no, it’s good.


ANNA: Well… what brought - I wasn’t here last week - what brought up property testing? Or, what drove you to that part of the conversation.


CHRIS: I think we were talking about testing in relation to the library guidelines.


AMOS: Oh, right.


CHRIS: And it kinda naturally came out of that, just because… well, it’s one of the things I like to bring up a lot. And because there’s some movement in the community towards that right now, I think it just sort of came up. I don’t remember.


AMOS: When we bring up testing, I think I’m kind of into it, too, and so - and I know Chris is, so its immediately like ‘hey, let’s talk about property testing’.


ANNA: Oh, that’s rad.


AMOS: I’m trying to learn more and Chris knows more than I do, so I’ll bring it up anytime I can when he’s around.


CHRIS: So, what is y’all’s experience been with property testing.


AMOS: Anna?


ANNA: Well, I was gonna say Amos, but, yeah. I’ve only actually… I’m curious to learn more. I’ve only actually done a little bit of it. It’s been really positive. The stuff I have done has been really helpful. But I currently have somewhat limited experience. I’ve only done a little bit on personal projects that I’ve worked on. I’m curious to hear what your all’s thoughts are about it.


AMOS: I think I’ve experimented more with it than actually used it on production systems, but I have used it on a few. I’ve used it for very basic properties, like a single function you would pass, stuff into, and I’ve also used it to generate a set of commands and run a full system through a random-ish set of commands - a logical ordered set of commands. And actually found some bugs with it that way. It was pretty neat. I didn’t go as far as to use propcheck and preconditions and postconditions, which made my tests very complicated, and… I actually used property testing on the lightning sensor. 


ANNA: Oh, that’s awesome!


AMOS: And actually got feedback from Chris on that which is kind of fantastic.


ANNA: Oh, yeah, what did Chris say?


AMOS: ‘Amos, you suck at property testing. You should give up.’


CHRIS: This is the worst thing I’ve ever seen.


AMOS: Yeah, I would say he kinda pushed towards the pre-condition, post-condition part of the command. He said to generate what should be the output of the command along with the command.


ANNA: Oh, I think you had mentioned that. 


AMOS: Yeah, so the input to the system already has the assertion to what the output should be. My test had a bunch of ‘ifs’. It was like ‘if the last command was this, then this is what you should assert’. It cleaned up the test quite a bit. Although one of the things that has been tough for me sometimes has been reading some property tests when they’re built that way. I don’t know if its just that I need more time, but they read very differently than a traditional unit test. Or even integration tests.


CHRIS: I guess before we go too much farther, we should define a little bit about how property testing works, in case there are people out there that aren’t familiar. So with property testing, what you’re really doing is - its a combination of a bunch of techniques. Let’s say you have an addition function, and you want to prove that addition works for all numbers. You could do that simply by writing a bunch of examples, which is typically what we would do with unit testing, but a more convenient way to do that, and a more powerful way to do that, and a more robust way to do that is to generate numbers just randomly - all numbers - then add them together and check their output. But the problem when you do that is that, because you don’t know what numbers you’re going to get when you generate them, is that you can’t make an assertion like 2+2=4. Because you don’t know you’re going to get two and two. So you don't know the outcome, then. So, instead what you have to do, is you have to use what we could call properties about addition and assert that those are valid. So, for addition, it’s commutative. So, lets say I had to add 2 and 3 together. Well, I can add 2 and 3, and I can add 3 and 2, and they both give me the same answer, so I can validate that is a property that holds, and I don’t need to know the outcome in order to validate that property. And you could do the same thing with the identity property, which is like if I add 2 and 0 - if I add anything to zero, I get the original back. So you would write that as a property. And there’s other properties for addition, but once you’ve written these pretty straightforward tests for them, you’ve proven that addition works because it follows these mathematical properties. So the different technologies that come together are the generation of data - so, you need to generate all numbers, or you need to generate strings. Elixir has atoms or maps or lists or whatever, and you have the property itself, which is effectively the test. And then you have this third piece of technology that sometimes gets forgotten, but it’s the true magic - its one of the pieces of magic of property testing - is shrinking. So what the shrinker does is it - when it finds a set of data that invalidates the property, or when your test fails, it will take that data and shrink it to the smallest possible test case. So then you can just say ‘well, my addition didn’t work for one and zero. Oh, well, I wonder why that is.’ And then you can go and look at your actual notation. So, that’s kind of property testing in a nutshell. And what you were saying, Amos, you needed to generate data - and if I remember correctly, the problem you were solving was like ‘I have this binary protocol and I want to validate that I’m parsing and coding correctly’. It was something along those lines, right.


AMOS: Yeah, yeah, I can’t remember if I really was gonna test it, it’d be like 2^64 tests before I could actually test every little bit of the binary.


CHRIS: Right. And so instead what you did, is you generated all the data that you need using a handful of generators, and based on what you generated, you were able to check its outcome in a handful of tests.


AMOS: Now, one of the things that you suggested that I didn’t take advantage of, too, so like, addition has some pretty simple properties, and they’re pretty straightforward, and even the reducing process there isn’t much. I don’t know why - that was a tangent. The big thing that you had suggested that I ended up not doing was, I had a decoder and you said write the encoder to ‘cause a good property is a symmetric property where if you decode something you encoded it should equal the original. Or encode something you decoded, it should equal the decoded version. Which is a great property. I think that symmetric properties also need to be used with other properties. ‘Cause otherwise I could just have them return hardcoded things and it’s not really testing anything.


CHRIS: Yeah, for sure. I mean, I think with everything you have to layer this stuff together in a way that helps you build a robust system. When it comes to testing, I don’t want to be an advocate for ‘throw away all your base tests’. Those are really useful if you’re checking error values or checking for specific things that might happen. You still want those things. Or you might want other properties. You need that stuff if you want to build resilience and reliability around your system. Or around your function, or whatever it is that you’re trying to test. You need that multilayer approach to really make sure that you’re handling all those different edge cases. Properties are nice ‘cause they’re big, in the sense that you can handle - you can start to test a lot of things. But when it comes down to practical considerations, you want it to shrink really well. You want it to generate failures on a regular basis. If that’s the case, you actually end up trying to find properties that are a little tighter. You try to find properties that hone in a little bit more, because then you can accommodate these real world situations.


ANNA: Chris, for people who are listening who are maybe less familiar with property based testing, can you give an example of something that’s a little more honed in as a property versus something that’s a little more expansive, or is it really case-dependent so it's hard to come up with on the fly.


CHRIS: So, that’s really - I think this is - explaining property testing and explaining the benefits of it is a little bit like explaining monads. You can say why these things are good, but it's so hard to put it into practice because it is so - especially with property testing - it’s so case sensitive, about the thing that you’re actually working on. And it’s hard. The addition thing is cheating in so many ways. You may not know what those properties are, but you know that they exist, so you can go on Wikipedia and you can find out what they are. You don’t have that, a lot of times, for other problems. I think, where you see it the most, is when you have a data structure, and you know you’re implementing a certain type of data structure. Well, that data structure has properties and you can look up what those properties would be, and you can start to use those to guide your tests.


AMOS:  So, in building properties and coming up with them, sometimes - this might go with the narrowing it down thing - if you were gonna build a min that takes in a list of integers and finds the smallest one, and people are like, ‘well the property is it’s the smallest one in the list’. And that’s… kinda what you’re after. So it gets hard. You might have to take a step back and think about really what you want and maybe compare it to the input. So maybe… the smallest item is the one that is less than or equal to anything that’s already in the list. But that’s still not enough because it could be the smallest one in the list minus one, would still fit that property. So you also have to say that it is equal to something that is in the list. So it has to be in the list, but it’s also smaller than everything else in the list.


CHRIS: Right, so, a property for that could be like, it returns one thing. Or - because that already forces you to think through the rest of your design, and it’s gonna find an edge case immediately, which is what happens when the list is empty. And what do you wanna do when the list is empty? ‘Cause you can’t return one. And that’s the first thing that your property test is gonna determine. It’s gonna find that failure. And now it’s up to you to determine what you’re going to do about that. Maybe it’s going to return bill, or return an error. It depends on the guarantees of your function at that point. I think that was a real problem that was in Elixir at that point. It might not have been min, but there was something like that that changed some antics a couple times because they didn’t know what the quote-on-quote right thing was to return. 


AMOS: And sometimes the right answer is to change the data that you’re generating, too. Maybe you find out that your generated space of data coming in isn’t even valid to come in. Well, and you said it might be to raise an exception for that.


CHRIS: Well, and I would say, too, about properties, is you do want some errors - like, you want to throw bad data at it and make sure for this class of data, it’s going to raise an exception. You throw some strings at the min function, let’s say, and what happens? Well, what it ought to do is throw an exception. And maybe that’s the right thing to do. And what you wanna do when you do that is say, ‘well, we get this exception for all this garbage data’. And you write a property that generates things that aren’t in your valid data set will still do that. Are you all still using mostly stream data when you’re playing with that stuff?


ANNA: Thus far, yeah. 


AMOS: I use stream data and just last night was trying out propcheck ‘cause I’ve been following and reading all the propcheck stuff. It might need some more documentation… I’m trying to figure out how it work so I can give some documentation back. But I wrote a small test in it last night. That was fun.


ANNA: For the folks listening, can you talk a little bit more about what propcheck is?


AMOS: So, there’s a propertesting.com is where I found out about it. And there’s proper and then there’s propcheck. And propcheck, from what I gather, is the Elixir version of proper. And proper is from Fred Herbert, right?


CHRIS: Proper the library?


AMOS: Yeah.


CHRIS: Uh, no. He uses it really heavily, and he’s worked on it, but actually… so proper is a tool that came out of Kostis’ lab - same guy who worked on dialyzer and other Erlang tools. But that’s - I think, he and whatever post-docs and other graduate students he’s got - they’re mostly working on - they’re the ones who are maintaining and still doing active research with proper. So that’s where that comes from. Proper is an Erlang library. Propcheck is the Elixir wrap around it. So a lot of times, the documentation in propcheck is pretty lacking, but that’s because it assumes you’re going to go read the proper documentation, and use that and get your information from there. There are docs. I don’t want to say there’s not docs, but there’s no sort of - if you don’t know anything about it, it’s really hard to dive in and just use it because, there’s no guidelines on ‘here’s how to write property test’. So if you’re just getting started, it’s a hard one to get started on. That said, proper is a pretty robust tool. And it has a bunch of features that are really nice. And, in fact I have kind of gone back to using proper a lot more these days. I was using StreamData, and still do use stream data, you know, our production tests suites for some of our services and stuff like that. We still really enjoy using steam data. But there are certain features in proper that are really really attractive that stream data currently doesn’t have. They’re being worked on or discussed, but right now, they don’t have them. 


AMOS: So, proper and propcheck have the command test, right? Like, the stateful testing?


CHRIS: Yeah, and that’s really the big reason why I continue to use proper. And honestly, I would even say that, where property testing really comes into its own is model checking. And that’s sort of stateful checking. And the way that that works is - you were kind of alluding to this earlier, Amos - but the way that that works is you build a state machine - which is just a model of the way that you want your system, or some part of your system, to work. And you build the idealized version of that. Like ‘hey, when I get these commands, I transition from this state to this state and this should be the outcome of that, and now my current state should be this instead’. And you build an idealized perfect model of that, and the way that you transition states is you generate commands. So your generator, just based on the previous state you were in, can generate commands. And those commands can be valid state transitions, they could cause errors, they could do whatever you want them to do. Like, for instance, we use this in like the raft stuff I've been working on, and we actually kill different processes and we shut down networks and stuff, based on these commands. Then we assert that these outcomes happen. So we generate this big list of commands. It uses the model to do that. And then once it has the big list of commands, it actually runs it against your real system. And after every command, you check your idealized state versus the real world, and you say ‘does the thing that I thought should happen match against the real world?’ And if it doesn’t, you get an error, and the magic of shrinking kicks back in and actually shrinks down the command list until you end up with the three or four or however many commands that it takes to execute this problem. It’s awesome. It’s such an amazing tool once you have it, ‘cause it adds this giant layer of robustness over what you’re working on. That said, it’s nontrivial to write command tests, you know. It’s hard. I wanna say that our - the big high level system test around raft is something around four hundred lines of code? Like, it’s a big model. It’s a big thing.


AMOS: That’s the thing that has been the hardest for me when jumping into that is coming up with a model that isn’t rewriting the actual system that I’m testing. I think really coming up with the pre and post conditions first help me to go back and write a model. Even though I’ve done very little of this so this is just trial and error at a very small level, but writing pre and post conditions for each command really help me flesh out what I need my model to do and be compared to the real system. 


CHRIS: So, I totally agree with you. Those preconditions and postconditions are basically required. Like, every condition almost always has to have a pre or post condition. And the post condition is where you actually do the checking to make sure your idealized state matches the new state. And the precondition is used for shrinking, ‘cause the way that shrinking works is that, once you have this big list of commands, it just starts removing commands from the list. And because of that, it could attempt to execute commands that are no longer a valid sequence. So, before, when you generated them, it was all perfect. It was based on the idealized state that you were in and that were valid in that state, et cetera, but once it starts removing them, all bets are off. So you end up having to put real preconditions around it to say ‘you can only call this in this state’.


AMOS: Right, so, if you’re writing a system that has log in, your failing test might say ‘log in, log out, log in’, so when its shrinking, it’s gonna make sure it can’t do ‘log in, log in’. ‘Cause the precondition for log in is ‘I’m not logged in’.


CHRIS: Exactly. And you end up having to put a bunch of those preconditions in there, just to get the tests to not fail because of precondition issues. You have to have a lot more guards around it than you think. That said, once you’ve put the effort into that, you start to find so many real problems, because, at the end of the day, testing functions is good, and testing in coders and decoders is good, and stateless testing is useful, but most of what we do involves some state and that’s a big part of why we use Elixir, and it makes it easy to sort of, manage state, different processes, and processes can talk to each other and all that. So once you have all that, you need these higher level systems tests that are really fine - the really really gross edge cases around it all.


AMOS: And it’ll find stuff that you never expected.


CHRIS: Right, yeah, it’ll just generate things like this that you would never ever think to do on your own. Yeah. It’s magical, in that way.


ANNA: That’s awesome.


CHRIS: So, it’s worth pointing out that, right now, stream data doesn’t support stateful generators, and that’s really the big thing that you need - you need the ability to generate things based on previous state. And stream data, sort of by its nature as a stream, doesn’t support that yet. That’s one of the reasons I’ve been using proper more - there is an ongoing discussion on the stream data repub. We’ve talked about it in IRC and in a bunch of other places. In order to add stateful generation, and what that would look like in steam data, and how can we best provide that to the community, and what do we have to change to make it work, et cetera. You know, we’ve talked a lot about, like, do we wanna adopt the library - sorry, the API design from proper and that kind of stuff. Do we think we can improve? That’s an ongoing discussion, and I’m pretty hopeful that that’s gonna end up in a good place. Especially because stream data seems like it’s poised to be added to the core language. Or, at least to the next unit. I think that would be a really beneficial thing to have in the community, and would help show people who are new to this how to go about doing it. Plus, the real big issue with proper - or, at least, the thing that a lot of people get hung up, or at least scared about, is that it’s GBL licensed, and so, that’ can be… I am not a lawyer. I am not here to tell you - I know people use it for production systems - you know, check with your local lawyer that is on the clock for that, ya know, I’m not here to tell you to do that one way or the other.


ANNA: But that’s a good thing for people to keep in mind.


CHRIS: Well, and it’s also one of the reasons why it wasn’t ever considered for Elixir itself. Because if you want to distribute it with Elixir, Elixir has - the licensing for Elixir has to change. Which, I understand. That’s a big reason why they built around, and I think that makes total sense.


AMOS: There was somebody I was talking to - I can’t remember who - that said they were using it, but they kept their tests in a completely separate repo, because they were worried about licensing. 


ANNA: Yeah!


CHRIS: That’s a thing. That’s definitely a thing…. You know, there’s - I think somebody, again, not a lawyer, don’t @ me - somebody was telling me it’s fine as long as you don’t shift your test code to production. And, ya know, if you’re doing releases in Elixir, you wouldn’t be doing that. But if you were deploying with GIT or something, then you are. So, be careful. Watch what you’re doing. But yeah, so, that would be another really great reason why, for the inclusion of stateful checking inside of stream data as well.


AMOS: Should we talk to the rich audience too, and tell them about the one library that we haven’t talked about yet - Quick Check.


CHRIS: Well, yeah, so that’s the thing. Quick Check is the best. I don’t think anybody is going to debate that. If you need the most robust, most reliable tool out there for this kind of stuff, just pay for Quick Check. It’s not an unreasonable amount for a company to pay for, but I would say it’s pretty far outside the reach of most individuals.


AMOS: Do you remember what the licensing is? Is it per seat?


CHRIS: Yeah, I think it’s per seat, and it’s many thousands of dollars. Like, it’s totally something that a company could and should pay for. Like, I honestly think it’s worth that.


AMOS: They’ll help you write tests, too, right? Somebody was saying. You can call them up and say ‘hey, I got a license for this saying that… uhh… I don’t know how to test this thing at all’.


ANNA: That’s interesting.


AMOS: They’ll help you out. At least, that’s what I’ve been told. I’ve never put the money up for Quick Check. Have you used it, Chris?


CHRIS: No, I’ve never - I don’t have a license for it. Part of it is because the stuff that I would want to use it for is a lot of my open source things. And some work stuff, but like, I feel bad asking a company to buy that for me, ya know?


AMOS: You know, if we can get anybody from Quick Check it listen, if they’ll give us three licenses for Quick Check, we promise to do a whole episode on how amazing it is.


CHRIS: I’m here to tell you right now, it’s awesome. It’s - I mean, what’s cool is that they also release their stuff as paper. A lot of the interesting stuff they write papers about it. So I’ll try to link all the different papers, if you wanna go check those out. And they’re great. And they’re written in a really readable way. They make a lot of sense. They helped me understand what was happening at the core of all this and gave me a bunch of insight, which was useful. I mean, they have stuff in Quick Check Pro, or whatever it’s called. They have their own runtime scheduler that helps you do find race conditions. Like, they have all kinds of really really smart engineering going to that thing. If that’s what you need, you can just spend the money. I highly recommend it.


AMOS: They also put out some good conference talks if you’re not the kind of person who likes to read papers.


CHRIS: So, yeah, that’s the big one. I think most people will be really well served. Well, I’ll say this. If what you want to do is just start getting into this stuff, I think stream data is the most approachable, ‘cause it’s gonna look the most like an Elixir library, ‘cause it’s not beholden to an API and stuff. But if you wanna do the stateful stuff, your option is proper, although I think that might change in the future. But, yeah, all of these tools… you’re gonna be really well served picking up any one of these and playing around with it, ‘cause they’re really useful. When you go back to normal unit testing, you feel like all your power has been stripped away from you.


ANNA: That’s fair. That makes sense.


AMOS: So, now that Chris and I have talked way too much… How about you, Anna? Have you done any property testing?


ANNA: No, I mean, this is why it was interesting for me to listen. Most of the stuff I’ve done has really just been with stream data. Mostly function testing and generating random data. So hearing you all talk about proper… I’ve looked into it a little. I haven’t used it yet, but it does sound really interesting. I’m excited about digging into it a little bit more. It’s been interesting hearing your perspectives because my perspective is a little bit limited on this particular subject.


CHRIS: It’s also one of those things that, depending on the kind of work that you’re doing, or the specific problem you’re working on, it may not warrant doing property tests all the time. Like, if you’re trying to discover an API or you’re trying to get something out the door, and in order to test it, in order to get it in front of a customer, in order to run through an API or something, it’s a lot of extra work to write property tests. It may not be worth it ‘cause once you add enough of them - especially once you go into sort of this stateful model checking stuff, there’s a nontrivial amount of code there. And you don’t necessarily even want to bake that API in yet. So I don’t know that it’s worth rushing into. There are definitely places that you’ll see ‘oh this would be a really good place to property test something’ and you can add that layer of robustness, you know, I don’t think it’s bad to not convert all of a sudden, everything that you do, into a property test.


ANNA: That’s a good question. When does that become appropriate? ‘Cause when your test driving things, or you’re trying to figure out the format, or how your API is going to look - given that there is some overhead to creating property tests, especially when you do it with escape, at what point does it feel appropriate.


AMOS: So what I do is I still unit test things, and when you get far enough into that, you start to have an understanding of ‘here is how this thing behaves, here’s what the inputs could possibly be. So now I know how to generate those, I can start to look at what I think the outputs should be’ and then start to write property tests based on the assumptions that you have so far, and let that property test then guide out the edge cases that maybe you missed with your example testing. Or, if you’re replacing another system, I’d just hop in with a property test and use the other system as your model. Just call out to the other system and say that your new one has to match the same output.


CHRIS: Yeah, I agree with that. I think a lot of it just really comes down to intuition. And you know, having done it enough times to know what to concentrate on, you know, you have the rule of three where, once you have three big tests that are testing different inputs and trying to check their outputs and stuff, that’s a really good time to sort of evaluate, ‘oh, I think I can consolidate this down to one thing’. Other great places are like, if you have a well-defined set of data that could be passed into a function, then that’s a really good time to do that as well. That’s more common than we think. A lot of times, it’s, ‘I’m gonna pass this very specific stuct in here, this very specific type of data’. But if you do have a function that’s like ‘I need to take inputs 0-255 ‘cause I’m building an RGB hex converter or something, then that’s a really good time to reach for these other tools.


ANNA: And as you said, it’s intuition, so as you play with it more, you get a better sense of when and how to test things. 


AMOS: And if you’re already using it to generate data for a test, it’s not a big hop to a property. I mean, your test is probably some sort of property-ish thing that you’re asserting, anyway


CHRIS: The other really cool thing that I wanted to throw out there about the benefits of having these generators - is that once you have them, you can do anything. Once you have stream data, it’s just streams. It's just literally streams of data. It’s incredibly well named. And now, if you just wanna generate fake data, whether you’re in IEX or wanna like throw fake data into your database in order to do development work, well now you have this infinite stream of fake things, and you can reuse them for all those things, too. They’re real code. There’s no reason you can’t use those, and we do that a bunch. We seeed our database with fake data to test stuff out. There’s a lot of use cases for those generators once you have them. 


AMOS: So put your generators in a place where you can get to them from other test files. Just start that way. 


CHRIS: Is there anything else anybody wants to throw out there? I think we’ve covered a lot of ground here.


AMOS: I do notice when property testing gets brought up, there are often a lot of questions of ‘how is that different than fuzzing?’ So… I think it would be good to address that. My understanding of fuzzing is less than my understanding of property testing, but from what I get of it, fuzzing is just throwing output at something and kinda seeing what happens. The property part and the shrinking part really aren’t as heavily pushed at. What do you guys thing? Either of you done fuzzing?


CHRIS: I’ve never done fuzzing in any sort of anger. I think for me the big difference is a lot of times in property testing, you’re explicitly passing real data. You’re passing data that you know is meaningful in some way. You might be throwing data that’s quote-on-quote bad. Like, for instance, with addition, if you throw strings into it, what’s supposed to happen? Well, it’s supposed to throw an exception. Well, for that - for the property of, when I get data I don’t understand I should throw an exception, bad data is in the correct set of data that you care about. And that, I feel, is one of the big differences. You’re not throwing garbage at it. You’re throwing data you expect to cause an outcome. And the other big difference for me is the shrinking aspect of it. It feels really really focused when I can sort of say like ‘alright. I expected addition to work with one and zero and it doesn’t and I only discovered that because my shirking was able to collapse all the big values that I had, and say ‘well, the base values are one and zero and that’ll kill it every time. I think focused properties is the big difference between property testing and fuzzing, although I think there are a lot of similarities. 


AMOS: I’d never done fuzzing as testing, but I had been told it’s more about finding when your system will crash and saying ‘well is this what we should have done?’


CHRIS: I think the ven diagram between those two things is pretty close together, and a lot of what makes them unique is just what you do with the outcomes. Like I said, property testing just just a little mor focused.


ANNA: Right, but it seems they’re achieveing similar things. Property testing just does it in a different way.


CHRIS: So, that’s all I have. I’ve said all the words that I have to say about property testing.


AMOS: That’s all you have to say on property testing? Wow… I didn’t know that we’d ever get to the end of it.


CHRIS: The problem is, if we keep going, I’m gonna loop back around on myself, like a snake eating its own tail.


AMOS: If you eat yourself… do you disappear or get twice as big.


ANNA: I think we’ve found our stopping point. 


CHRIS: This is like what happens when you turn a bag of holding inside out. What happens when you turn a bag of holding inside out. Does it destroy the universe? What happens?


AMOS: I think that we’ve reached ultimate nerd. And I know that everyone has stuff to do today, except for me.


CHRIS: Final thoughts before we put this up?


ANNA: I think it’s awesome that there’s this much conversation in the community about testing, and that it’s continuing to grow and that people do think it’s important. I think that says something about the community and about the language.


AMOS: I just encourage everybody to try property testing. Just go play with it. Go write a list function and use properties to, and sorry if we have lots of clipping on my audio because Audacity keeps automatically turning up my volume. We’ll try to fix that for next time if it’s bad. 


CHRIS: We’re a work in progress here.


ANNA: Work in progress, yep.


AMOS: We always will be.