Amos: Welcome to Elixir Outlaws, the hallway track of the Elixir community. Amos: So this isn't the show yet. Anna: It's always the show. What are you talking about? Keathley? How is your, uh, paternity leave? Are you going nuts? Chris: No, it's good. I'm back to work as of today. So I've spent my entire morning reading back through slack chat logs to try to catch back up a little bit, which was reasonable. And then I spent the remaining time trying to get my inbox back down to zero. And, um, that was also less reasonable, but it was actually really fast. Cause I just like select all archive back to zero. Done. Amos: I have an email rule that if, uh, if it's more than 30 days old, I just archive it and assume I didn't care enough to actually do whatever it's asking me to. New Speaker: The important things will rise to the top and you will do the important things. If it's worth doing, you probably have already done it. Amos: Or forgotten about it. New Speaker: That's how I feel about it. Anna: Then it's not that important if you've forgotten about it. New Speaker: Exactly. Amos: Or it wasn't that entertaining. Anna: Or that. One of the above. Amos: Like right now I have a piece of hardware that, uh, I'm working on that the last three or four times I've tried to deploy to it, it's just restarts over and over instead of going into the software. So I'm trying to figure that out. And I'm really glad that today is my day off. New Speaker: What is it? It's a board of some sort? Amos: Yeah. Yeah. So I'm working on network management controller for uninterrupted power splice and I needed to debug some stuff on the actual board. And when I deployed new software to it, for some reason, it compiled locally, everything built. And when I got it on the board, it would get through partway and say, you need cowboy plug in the boot up process. And then it would just restart. But, well, that's fantastic. But since it's restarting it doesn't end up with an SSH connection. So redeploying to it is, is a huge pain that takes like an hour because I have to go into u-boot and reset everything and I have to set up an FTP server for it to connect to. And yeah, it's awesome. Chris: Did you add cowboy plug? Did you try, did you try that step first? Amos: I did try that step and, and now I don't even see an error message. It just says it lists everything that starting under, under, um, Elixir, all the applications it's like starting this application, starting that one. And when it gets down to the end, it just restarts. Chris: Nice. Did you take the restart application out of there? Just let it start? Amos: It's a really good one. New Speaker: I found your problem. Anna: Good job, Chris. Chris: You're welcome. I blame a shoe horn. I don't even know what it, anything you're doing? But that's what I blame. Amos: It's probably shoe horn problem. There's something underneath that Shoehorn might be hiding, uh, right before this, it was running out of memory all the time. So it's like one problem after another. And when you're working on embedded hardware, uh everybody's like, oh, you should just buy some more Ram. It's cheap. No, I'm sorry. It's built in and I can't do anything about it if I only had some type specs. Chris: That was a, that was a gross segue workshop that, and come back here in a few minutes and I'll see your case then. Amos: It's like a bejeweled, hot pink segue. Chris: So you been found guilty of terrible segues. You're going to need to come back here and try that again here in 20 minutes, we try your case. Amos: He's got that Southern accent down. Chris: It's born and raised where you going to do? It's just bred in. Amos: So Andrew, did we get, did we get your recording working? Andrew: My name is Andrew. I live in Chicago. I worked for a company called Albert. Uh, we are in the education space, um, making tools for teachers and students to help them study for their PA tests and within the classroom as well. And we've been working with elixir for the last several years. I'm also the maintainer of the Dialyxer, their project, which is the Elixir wrapper around the Erlang tool, dialyzer um, and then it's new offshoot, Erlax. Amos: I'm a big fan of those tools. Almost every time I'm digging through a project and, uh, or, or looking through a code review and I'm like, Hey, I think you're, you're missing some logic there. I usually tell them, you know, if you put in some type specs, you might at least recognize what you're looking at and maybe think through it. And you might realize that you're not handling certain cases. Chris: But only if you also use Dialyzer, it turns out. Amos: That's true. I have seen quite a few projects with specs and no Dialyzer or anything. And like, so you're commenting. hat's that's fantastic. Yeah. Andrew: Even that I find is better than nothing. Sometimes, you know, when you're, when you're digging into some random project that you've, you know, that, that has some bug in it, uh, having at least the author's intent there in the type spec is usually pretty helpful. Um, even if the tool is not actually verifying it, um, having the authors, uh, original thought there is, I don't know. I think that can be a good thing. Amos: What brought you into working with Dialyxer or Dialyzer and, and like, why, why do you keep doing it? Andrew: So my background is, is in the Java space. Uh, so I'm used to having type system I'm used to having that crutch to kind of lean on to say, Hey, you're not doing that. Right. You're trying to put this square peg in this round hole, um, and started reading up on what was available to me in the Elixir space and found that it was pretty much dialyzer, um, started reading into that and get trying to get it working am sure as every single person that has ever newly introduced Dialyzer into their project, sees the initial wall of output is very large and very, uh, frightening as, as it were. Um, and I just wanted to make that better because I don't like mentally putting new lines into walls of texts that that's not how I like to spend my time. So I wanted to see if I could write a tool to help me do that. Amos: So that, that initial start with the big, big wall of text, if you know, somebody who's working on a project or, or you hop onto a project and it has no Dialyzer, Dialyxer usage, how do you, how do you take that first step to not just be completely overwhelmed by, by what's happening? Andrew: I think there's really two phases to that. So one is being familiar with Dialyzer and, um, I guess, Dialyxer itself, um, and, and the things that it can do and can't do for you. Um, and then, you know, if you're, if you're kind of familiar with that, then from there, just carving off a little bit of a little bit of your application that you know, how that works and then just adding some facts around it. Um, I kind of look at Dialyzer as just, you expressing some fact about your function or some fact about a segment of your, of your application. There are areas of your application that, you know, better than others and, and starting from there and expanding outward, I think is a good way to go. Amos: Can you run Dialyzer Dialyxer on just a part of your application? Andrew: You can't, but you can, there's a filtering options now in dialects, so you can ignore certain parts of your output. Um, so if you have your code reasonably structured, then you're going to have certain files in certain folders that are going to be the good parts and the bad parts. And so you can kind of go that way. Um, and you can say that you don't care about particular types of warnings, and there's a bunch of different ways that you can go about trying to make that a little bit more manageable. Amos: You mean like...That's not Dialyxer. Nevermind. I was thinking Credo where it complaints about Logger all the time. Chris: Yeah, I've discovered has been, um, tricky is, uh, trying to add, Dialyzer sort of after the fact, uh, with like an established project and you want to add it in order to start like verifying these things, but you may not have time to go in and, uh, actually fix all the, all the specs or whatever, but you also want to add it to CI and ensure that it doesn't like get worse, if that makes any sense, like, so you want to catch like new things. Uh, I'm curious to know, um, if you have a recommended way of starting to gradually introduce dialyzer into an existing project, because the way that I've done it, it's just like the hack shell scripts together with like some of the filtering that, that is already provided and make it kind of work through that. Andrew: I mean, that's kind of where, where I was going with that. Just picking, picking a corner of your application that, you know, pretty well and then saying, ignore everything else, or ignore explicitly these particular areas, just using the built-in dialects or filtering mechanisms. And then, then from there you can, if you're adding new files or you're adding new warnings that you're see, I should just reject that at that point. Um, that's probably how I would go about that, but there, I don't know, there might, there might be better ways, but that, that to me seems the most reasonable. Chris: Sometimes too, you end up getting stuff where it's like, it's in somebody else's library and they didn't notice it cause they're not running Dialyzer, but like their specs are wrong or whatever. And so you don't, you don't discover these things until late too late in the process. And then it's a lot more work to get in and start fixing everything to make CI pass again and all that kind of stuff. So the filtering is very important. I think. Amos: I was going to say the, the, the biggest pushback that I get from a lot of people is, uh, running Dialyzer is, can be extremely slow, especially in a larger project. So they don't want to do it locally all the time. And then when they push it to CIA and you set up a CI server, there's often the complaint that, well, it works, but in, in the projects that I've gone through and started adding and running Dialyzer I've, I find that I there's a lot of bugs that, uh, end up popping out just from the Dialyzer warnings. You look at like, it'll come up with case statements that you're not handling correctly, just cause you start looking at the types and, oh, well you're not handling these three types that could come in that your type signature doesn't have or, or whatever it's been really helpful. Are there any tools out there to, to help alleviate that time frame that people are putting into it? And that they're complaining about? Andrew: I don't want to say that this is a myth because it's, it's a problem that does exist, but it's a problem that you only have to pay the cost of every now and then, um, the only time that the, the very large time for Dialyzer to build up, it's - actually hold on before I do that - how Dialyzer works is it will go through all of your, uh, applications or all the applications that it start in your, in your Erlang virtual machine. Um, and in particular, it'll do that for all of your dependencies. And then what it'll do is it will build a cache about the facts about it, the, your dependencies type specs. And then what it'll do is it'll then after that's all done, it'll, it'll cache the, the result of that. And building that up can take like 10, 20 minutes. That that's an annoying thing. Um, but then once that's there, the only time that that ever gets thrown away is if it's explicitly discarded or if you bump your Elixir or Erlang versions. Um, and then so from there, it really comes down to how do I cache that appropriately? Um, and on your local machine, it should just stay there just normally. Um, but on CI servers, you can just say to cache, all of the adopt PLT files, and then after it builds its initial initial state, then it's pretty much good to go. Chris: Caching on most things is pretty straight ahead. Like we, we, we have it set up in circle, um, that's, which is the CI, uh, thing that we use. And it works really well. And it's, it's like two lines or three lines or something like that to both restore the cache and to cache PLTs, once you're done, like it's, it's pretty straightforward. And I think Travis has something similar and I haven't used any other CI things outside of Jenkins. So where do you have to do everything yourself? So, uh, generally that stuff is pretty, pretty easy. I will say that 15 to 20 minutes upfront where you just cook your laptop or whatever for awhile. That does take that is that's that's a little bit annoying. Amos: Yeah. I usually do it when I need to send some emails or I'm going to go make some coffee. Chris: I do it when I'm cold, and I just hold on to my laptop. The fans were up. Anna: You're not just mining crypto Keathley. Chris: Yeah, no. Andrew: I mean, after the JavaScript news this week that a, that might be true. Chris: Bitcoin is down below $4,000 again. So it's like kinda not even worth it. I don't know, upshot - GP user like affordable again. Amos: And they've all gotten way better since last time I purchased one. So what was the JavaScript news? I missed that this week. Andrew: There was a dependency that, uh, was used in a couple projects that the owner decided to give away to, uh, one of the contributors and that contributor then went and put obfuscated, uh, Bitcoin mining code in there. And, uh, then just shoved it on MPM, just shoved it up there. And then people proceeded to download it... Chris: Because it's detached. It's just like, you can push whatever you want to MPM as long, like MPM doesn't care, you just shove stuff up to it. I mean, honestly in the same way that Hex does, like Hex doesn't care, like what is on GitHub or wherever, it's just like, give me code and I'll host it for you. So dude just like shoved, uh, uh, a patch release that had like, I think it like, like tried to steal Bitcoin wallets or something like that. Like it was wild. Andrew: It was, it was a particularly targeted attack towards another dependency. And I don't know, it had tried to exploit something in there. Chris: To be clear, when you say it was in a couple of packages, it was getting like 2 million downloads a week or something like that. Wow. Yeah, it was a very highly, because it was like, I think the package that was the, so it was - if I remember correctly, it was two different dependencies. And the actual dependency that the issue got opened up on is not even the dependency that was like, that had problems in it. But the dependency that got the issue opened up was literally called something like flat. It was like Flat Map Producer. It's like, it's like literally like one function, some utility thing that had a dependency. First of all, that had a dependency and secondly was the dependency. It was malicious, it was wild, but it's led to all these conversations about like the nature of open source and the nature of maintainer ship. And what, what do we owe the people using our software and all this, this stuff. Amos: The "what do we owe the people using our software" is not going to stop the person with malicious intent. Chris: There's many medium hot takes on this. If you want to go read them on what this means for, for the future of open source. Amos: Oh man. All right. Sorry about the sidetrack there. Chris: No, no, no. That's good. Amos: Dialyxer. Dialyxer. Chris: So when did y'all decide to, um, to add the explain stuff for Dialyxer. Andrew: Um, so I guess the pretty much all the work that I've done towards that project has been in the vein of trying to just make it more approachable and readable for, for people. Um, and so the first step of that was to actually start parsing the output appropriately and, and putting in the new lines that I was talking about before, um, and actually converting it over to Elixir. Um, but once, once that was all kind of in a good spot, I still realized that I had no idea what it actually wanted me to do. And so, uh, and so I wrote an explained feature. It was actually fully ripped out of Credo. I just borrowed the API straight out of Credo. Um, and just wanted to put in examples of, of how something would produce such an error. Um, because I thought that that might be helpful, you know, for, for me and for others, uh, that also can have the same confusion about what no return means or call without whatever. I don't even know what all the, all the different ones are, but, um, yeah, they, they can be a little bit terse. Amos: I think that will help sell more teams to use it too because outside of the time that people initially spend, because they just see that first build PLTs and they're like, oh my gosh, this takes forever. But then that, when they did get air's back, it was, well, they had no idea what it meant. And so now you guys have made the errors that come back, look nicer, and then you have the explain feature. So like that is going to save, I think a lot of people, a lot of time and headache, if they, if I can jump on the bandwagon. Amos: Yeah. That's definitely the hope. Chris: A lot of times, too, those errors, they stack on each other in weird ways. So you fixing one thing ends up fixing maybe three other things. Like they, they, they're not hierarchical. And like if that's the right term to use in a useful way, a lot of the times you get a lot of like false positives. Andrew: I agree with that. Um, as an example of, of how that could manifest itself, say for example, you were using plug and plug in it's type spec, decided to say that it returned okay. And obviously that that's erroneous. Um, but then everything that now uses the plug behavior would just be incorrect and it would give warnings all over the place. Um, and so I guess when, as another way to, to approach, figuring out how to, how to fix your code base and make that wall of text go away, um, is to try and find the deepest modules that are, that are throwing, um, that are throwing errors, like the utility modules that are shoved into the corner that get imported all over the place. Um, and then just the, the functions that get called in a lot of places. Um, if there's anything wrong with them, then it's going to lead to unexpected areas up the stream. Chris: As you've been working on this stuff, what's the most common, uh, issue that you, you feel like people run into when they're getting started? Andrew: So I guess most of the errors that people open with us are when things aren't working right with the pretty printer, um, and that that's led to, to some insights about the language that I didn't quite know about, um, that we can get back to that, but people, people don't really like have, I don't know, it's kinda hard to like in isolation cause people, when they post a bug report, it's really just like, I have this one function or I have this, like these walls, this wall of, of output. And it's, it's usually just like some simple fix, I don't know. And I don't really know how to answer that question very well. Sorry. Chris: That's okay. Amos: I have, I have one of those bug requests out there and I have no idea what the issue is. And I have, like, I don't know, I have a ton of warnings, so it's really hard to, to figure out exactly if I can find the example of the code that causes that problem, that isn't so ingrained in my customer stuff that I can't show it to you. So that's, that's been my problem in, in coming up with, how do I, how do I give you an example of what's going on? Cause I don't even know. I can't, I can't look at it. I even know where to begin to fix it. Andrew: Yeah, yeah. That that's, that's been an issue. Um, some people have given me access to their private repos, which I didn't generally ask for and they were, they were very gracious to do so. Um, but that that's been interesting and just going in and going, oh, that's why that's misbehaving in this really bizarre way that I couldn't determine, uh, just from the output. Anna: What are some of the insights that you've seen? Andrew: Yeah. So, um, I don't know how many people or how many listeners have worked with Erlang itself and not just elixir. Uh, but Erlang itself is a single static assignment language. And so if I go and I say that X is four, and then I go and say that X is five, that's actually a compilation error. Um, because you just told me that X is four why - what are you, what are you doing now going and redefining that. And so how Elixir gets around it is that all variables get this little wrapper around it and it's a capital V for variable and then whatever the name is. And then at the numerical binding of that variable. So V X at one or VX at two VX at three, um, and that's how elixir gets around that Erlang design choice. Um, and so when I was getting output, uh, from, from people, it would just have these random V app things that I was like, what are those? And then started digging into it and realized that that's what it was. Um, so I thought that that was kind of cool. Chris: That's super interesting. Do you have enough? Um, I'm curious to know, like when you're doing, oh, I guess probably not, but by the time that you're actually wanting to do like, kind of nice errors for people in Elixir, you probably don't have the context anymore to be able to like map those back to Elixir source code and know like it was this variable name or whatever, or do you? Andrew: So yeah, so Dialyzer is, uh, is, is, uh, it's a fun tool. I drops a lot of things on the way out. Um, and so for example, if you have a, if you have a really big struct, like you have a con for example, um, what it'll do when it represents that to you in the error output is it will say it'll, it'll grab a couple of fields from it. So it'll say like assigns and post and stuff, but then it'll just say underscore fat arrow underscore for the rest of the map. Um, and so I can't do any, any nice diffs with that. Um, I have the line number and the file number that the error is purportedly on. Um, but you don't really get too too much all the time sometimes though it tells you exactly everything that you need. And, um, and, and you're able to give a nice error message that tells the user exactly how to fix it. Um, but sometimes you can't unfortunately interesting, right? Chris: Because you're working, you have to basically work off the raw diff or like the raw output, but I should say, uh, right. So you just have to have some parser that can actually determine like what all this is and that's all you get. Andrew: That's what, uh, what Elranx does in particular. So Erlax does the heavy lifting for that. Anna: Do you want to talk a little bit about that for folks who may not know? Andrew: Sure. Sure, sure, sure, absolutely. Um, so Erlax, uh, solves the problem that we were just talking about, um, where it takes the raw Dialyzer output, um, or it's actually meant to be a little bit more general than that, but how it was originally designed was to, um, to, to just take the, the structs and the, uh, the already Erlang-ified output that we get from Erlang based tools, and then be able to convert that back into what it would look like in Elixir. Um, so it parses it and it pre-prints it. Um, and then is meant to be plugged into all sorts of tools that that would be useful. Chris: That's coo. Are you using, um, like leaks or whatever the heck that thing is called yeek? Yak? Andrew: There's a built-in lexer and parser into the, uh, into the Erlang and it's in it's part of the Elixir tool changes in the compilation path. Um, so if you put an XRL and a YRL file in your, uh, top level SRC directory, then it'll produce Erlang modules that you can invoke to run your Elexer and parser for it. Chris: And it's standard enough rules, um, and all that, like you would get out of like Yacker or one of those kinds of tools. Um, I've been known to throw that those tools at problems that may or may not deserve those problems, this or this tool, because those are easy. I mean, they're there in the sense that like, you know, like parcels parser generators are kind of magic. Like they're, they're using some deep math to be able to work all that stuff out. And it's like really interesting, but it's really straightforward when you look at the rules and you're like, there's the parsing rules. Andrew: They're definitely very powerful tools. They are a little bit more low level than I prefer. Um, if I were using parsers and lexers regularly in my day job, I would probably lean a little bit more into the nimble tools and nimble parsec is the nice one. Um, and that's a little bit more at the API that, that most people want, I think, um, but for such a low level tool, like Dialyzer, we can't really be bringing in libraries all over the place that, that seems inappropriate. Chris: I think, I think too, the parser, it generates is faster than the one that the base tools will, will generate as well. Cause I think it's a different parser technology. I could definitely see that that be wrong. That might be apocryphal. I don't know. But I think that's the case. I might be spreading rumors, nimble parsecs, like supposed to generate a really fast parser though. That's part of its goals, I think. Andrew: All the nibble tools are very well done. There's a CSV one as well. Um, and it's from the platform of tech folks. So they're, um, really well done. Amos: You, you, you talked about line numbers, right. And being able to point people back to a line number at least. Um, I just was curious when you said that, because macro seemed to be a big problem for a lot of people for line numbers and things. Do you run into bug reports and problems that people are having based on the fact that they're using some maps? Andrew: Um, occasionally some of the, some of the funkier ones, um, I guess some of the funkier Dialyzer errors that I've seen, um, are the result of, of the line numbers being shifted around. Um, but usually that means that I'm missing, uh, uh, the, the, what does it bind, quoted or not bind quoted there? There's a, there's an op that you can pass. It says to keep your, keep the line numbers as they were. Um, usually that indicates that at least in my own code, um, but usually usually the areas that people are reporting, like, et cetera, are just the result of my tool just didn't do something properly. It didn't parse some piece of output, particularly it, it didn't anticipate some particular edge case that, that the output would produce. Anna: What has been the most, um, well, two questions. What has been the most like fun, rewarding part of working on dialers and what has been the most challenging portion of working on it? Andrew: Um, so when I first started doing it, so, um, it was actually a José opened an issue and he basically said, all right, guys, I know that this is the result of poor source material. You don't really have very much that you can go off of, but this output just is not, it's not good. We, we can totally do better. Um, and so being able to actually close that issue was, was really fun. Um, and that seemed really cool to me. Um, and finally getting all of the output on example projects to just be parsed appropriately. Um, when I, when I was actually writing it, I had access to a project that just had a whole dearth of Dialyzer errors. And so I was able to get a lot of source material and get through a lot of the initial edge cases based off of that. Um, yeah. Yeah. I think, I think closing that, that initial issue though, is, was, was definitely the, the highlights so far. Anna: What has been the hardest part or what has been the most challenging? Maybe not the hardest though. Andrew: I think framing the problem and kind of figuring out how to, like how, how to even do this was, was kind of difficult, um, because kind of writing the tool, anticipating that I would cover all the edge cases and trying to figure out how to do that in a defensive way, such that people's computers wouldn't crash and they would want to just roll back to the previous versions. Um, that was, that was kinda tough. Um, and then doing like the, so the Erlang and Elixir are very simple languages under the hood. They don't have a lot of, a lot of, um, fancy constructs that need to be put into the parsing grammar. Um, but some of them are trickier to express than others. So, so there, there are definitely some, um, that parser has seen a couple of, a couple of revisions let's say, um, and, and cleaning those up and getting down to the heart of how some of these constructs actually are expressed, um, has, has been nice. Um, but that, that was also just a difficult thing to do. Cause I'm, I'm not really a lexer parser person. Like I've done compiler stuff in school, but, um, haven't really needed to do that as part of my career. Um, so picking up those tools was also interesting. So it was like learning, learning the lecturing parser stuff, as well as learning how dialyzer works under the hood, and then like learning how this, this new tool works and trying to make them all sit together. Um, I don't know, it was, it was hard but fun at the same time. Amos: So what did you w when you, when you got into the parser stuff, what things did you do to, to learn it? Or is it just like hard won beat on it until I figured out how it works? Andrew: There isn't a lot of documentation around, uh, the Erla g based tools, uh, leaks and yek. And, um, I managed to find a couple of blog posts, uh, from early devs, uh, probably written back in the nineties, or, you know, maybe, maybe even earlier, who knows, um, just talking about how the tools worked and how, how to write some of the example grammars and how they like to do things, I guess. Um, and so, I'm sorry, what was the question? Amos: How did you go about learning the to use the lexer and parsers and stuff? Andrew: So, yeah, so just kind of just beating, beating on it once I, once I found a least a little bit of framing from, from some of those blog posts, um, and then just kind of refining it and figuring out how could I just more simply express this, this rule it's kind of like working with like regular expressions. Like you can do some really gnarly rag reg Xs, um, and there's probably something more simple that, that you could use instead. Um, and it's really just iterating and trying to come from that more complicated one to the simpler one. Yeah. There, there really isn't a lot of documentation around those. So unfortunately, um, there's also, uh, there's a bunch of warnings that it decides to produce called shift reduce warnings, um, which are apparently just a, uh, thing that's built into, or that, that parsers produce, um, or the particular type of person that this is using that they produce. Um, and usually that means that you're doing something wrong and it's not always apparent why you're doing something wrong. Um, and so figuring out like what that even meant, and then figuring out how to, how to start debugging that, um, was, was tough. Amos: In writing Dialyxer or in working with Dialyzer. You probably see a lot more, uh, you see a lot of Erlang code D had you done Erlang before, and if not, like, how has that experience of coming from Elixir to Erlang? And did you learn? Andrew: I've ctually never been an airline dev. Um, obviously I've, I've read a lot about the Erlang internal is just working with the Elixir, um, and re read lots of stuff. Um, but haven't really worked with the, the language very much itself. Um, and so digging into some, some of the actual, like, source, like Erlang or laying on get hub, uh, source has been interesting trying to find where Dialyzers happening to see if there's some hook that I'm missing, um, or anything like that to, to make this process a little bit easier. Um, but yeah, it usually it's, it's, it's Erlang's representation of your Elixir source code. So if you think about like what Elixir is doing under the hood to, to kind of produce that, Erlang, that's usually the best way to frame it when you're, when you're debugging it. Um, at least to me, I think, um, because it, it might not necessarily be idiomatic Erlang, even though it looks here itself writes really idiomatic, Erlang. Amos: Have you ever utilized anything like, uh, the decompile and looked down at the, um, machine code for the...? Andrew: No. I think Dialyzer itself does that though. I think that's actually how it does its job. If I remember correctly, it'll go through and it'll run across all your, the compiled source code and then figure out if, if it, uh, mismatches with the type specs that it knows about your program. I could be mistaken about that though, but I'm pretty sure that's right. Amos: Seems like it would be a lot simpler for it to parse the, uh, not machine code. That's not what I was thinking, but maybe even machine code over top of whatever thing. The, uh, Assembly not, there you go. That's what I was thinking. Not machine code, it's just one step up. That's fine. Uh, so yeah, I, um, spend some time reading the beam book and, and if you haven't seen that, that is pretty amazing on it's all open and get hub. So anybody out there that has an interest in knowing how this works into the covers, the beam book can, yeah, it's a Andrew: It's discussion about preemptive scheduling and all that fun stuff is very well done. I think, um, and preemptive scheduling for, for listeners that don't know is how the beam does its soft real time guarantees. Um, it basically says, Hey, process, you can go and you can work for a little bit. And then after a little while it says, Hey man, you've, you've been holding up the queue for a little bit. I'm going to just put you in the back and then let somebody else do some work for a little bit. Amos: Have you found that learning, uh, how the beam works underneath changes the way that you, or, or even the Erlang code that spit out from Elixir? Has that changed the way that you write your Elixir code? Andrew: Um, a little bit, I would say that I would say that, uh, you can do a lot of stuff to make Dialyzer's job a little bit easier. Um, and so that doesn't directly answer the question I guess, but it kind of goes towards it, um, because Dialyzer, isn't a perfect tool. Um, we haven't really discussed that, but so it, it will sometimes not tell you that something's wrong when there is something that is wrong. Um, and conversely, if it does tell you that something's wrong, that means that something's wrong. Um, so you at least get half of a yearly skip that side of the guarantee. Um, but as far as like when you're, when you're writing functions, if you reduce the parameter set to as shallow of things, as you actually need to give to that function, dialyzer is often able to do a little bit more insight, um, into, into what's going on. So for example, if you, if you have a function that takes in a con and gives back a con, well, that's cool. I don't really know what's going on in the middle there. Um, whereas if your helper functions are written where they take in explicit maps or strucks or something that you pull off of the con, then dialyzer is able to look at that and say, oh, you're trying to access a field on the struck that doesn't really exist. Um, and so you can, if, if you just, you know, follow love Demeter, and those sorts of traditional software engineering practices, uh, you, you can help Dialyzer along, uh, and be able to get better output and more helpful warnings. Anna: I don't know how much you want to get into this, but can you say a little bit more about, or not being a perfect tool? Andrew: The, the paper based, the, the research paper that, that produced Dialyzer the research group, um, they, they read, uh, a couple of papers around their findings, um, and they effectively made some, some trade-offs, um, to get around, I guess, I think it's to get around some of the, the distributed nature of Erlang and, and the actor model, um, because I believe that that's an open research problem that they're not able to fully type that or something along that effect. Um, and so it makes some trade-offs to be able to do a good enough job most of the time. Um, whereas I think so if you're sending a message between Erlang processes that are on distributed nodes, I believe that you can get into a state where, um, the sender and the receiver might have different versions of the program that are running, um, or things of that nature, and being able to type that message is difficult to do. Um, I think is the crux of the problem. Um, or at least that's one of the problems. I'm not sure if that's the only problem though. Chris: It's definitely one of them. Um, uh, I'm trying to remember all the examples that they cite. Um, one of the, one of the things is like Erlang is all, side-effects basically, you know, concurrency all this kind of stuff, like sending messages, it's all, it's all side effecting things. Um, and yeah, like session types, which is a way to encapsulate those sorts of, um, sending of messages and contracts, like in a type system, uh, amongst other things is, is like definitely still open research. Um, and people are pursuing that. Uh, and at the time, I don't know that I don't know that anybody has started working on that stuff. And surely, like, I mean, lots of people had come up with ways of typing side effects, but, um, the, the other flip side of that was, uh, you could type, you could theoretically create, uh, a whole type algebra around Erlang that, um, you know, did like the Haskell thing. Uh, if you wanted to do that and like work out a type algebra to do all that. Um, but the flip to that was it forced everybody to rewrite all their code, um, to make the type algebra happy, because there's a lot of idioms that happen inside of normal Erlang Elixir code, uh, that don't play well with that sort of type algebra. And so in order, you know, dialyzer like one of the things they opened with is basically the original paper, I mean, is like, they wanted to get that type algebra adopted. They wanted that, you know, they needed it to be adopted by the industry. And, uh, in order to do that, you have to make it conform. You have to make it permissive. Like it's the same way people want, you know, TypeScript or flow to be adopted, right? You couldn't, you can't add a type system to JavaScript that's overly restrictive or else no one will add it to their project after they already have a working product project out in production. So that's how you end up with these like gradual type systems. And Dialyzer is in vein of, we want this to be adopted, so we're not going to make you rewrite on your code. And because of that, we're going to be super permissive - in some cases, way too permissive. But when we tell you an error it's never wrong. That was like the, that was, that was the sales pitch, I think. Amos: It seems like a good sales pitch being permissive - under permissive, over permissive. There there's a lot of options that you can pass into Dialyxer. Uh, I, over spec under spec and all those things is there for, uh, uh, maybe an existing project that wants to, uh, start adopting Dialyxer or what, uh, is there a set of flags that you suggest... Andrew: Actually do? I find the, uh, I think it's under spec. No, the over spec I find over spec to be obnoxious. I think it's, it's, it's a, it's a very strange warning. Um, and I'm not sure in which code base it's, it's been useful, but I'm sure that there's exists out there somewhere. So how it works is that if you, um, say you say that your function returns a map, which is, you know, a useful or useful thing, if you don't say that it's a map of a part that has these particular keys and these particular values, uh, it will yell at you and produce a warning. And I find that that level of verbosity is just unnecessary and most in most code bases. Um, so I always turn that, that went on and in particular to, uh, in all my projects, um, let me take a look or I turn it, turn it off. I turn on warning, turn, I turn on the ignoring of that warning. Chris: That one's the under specked flag. And then the, the over SPECT one is the one that will say the actual success type of a function is, uh, any, but you've said that it only takes booleans, but it actually accepts any. And so that, like, the way you could get into that scenario is like, let's say you have, um, an add function and it takes two integers and it produces do positive integers. Even let's like be really restrictive and you type it as such, you say, ad takes two positive integers and gives me a positive integer out or whatever. Um, the, like if you run it with the over SPECT flag, um, it'll like yell at you unless you're actually using guards and such to assert that, um, that it's actually doing all those things correctly. Like if you don't guard against what the inputs well, then your inputs are actually anything, like you could submit strings to it and it would just be wrong. And it might, you know, under the hood, like, it might catch the error of like you doing one plus one, or you doing variable one plus variable two or whatever. Um, but yeah, it it'll also tell you that like the functions in Nutro is just wrong. That one's useful occasionally. But I think that one is that the one that also breaks the singular year of we're never wrong. I feel like over SPECT is one that will tell you things are wrong when they're not wrong or something like that. Like one of the, one of those two, it might be under spect. I don't know. One of the two will tell you that stuff... Andrew: Is yeah, I believe, I believe that's correct. Um, Amos: I think our confusion here means that those names are not... Andrew: Most of the names are not that great to, to be perfectly truthful. Um, but you know, that's why we have rappers for things. Chris: Well, and if anything, if anything that helps make it easier for people to learn is good. Cause I remember when I learned Dialyzer, I mean, I actually just had to read the paper. Like that was the only thing that made all these errors and stuff makes sense was going and reading the paper and then trying, I had to figure out like why. Andrew: Writing the explanations has been interesting. Um, cause yeah, like, like I was saying, I figuring out how to write code that will even produce some of these areas is interesting. Um, and then some of the ones I don't use, uh, in, in my day-to-day life, like a pig data structures are actually super cool as an aside. Um, but I had no idea how they worked until I started working with this project and had to write documentation about how to produce warnings through a pick data structures. Um, so pick the instructions just for, for those that are unaware is that if you mark a struct as an opaque data structure, um, and I think it's like at opaque or something like that at some annotation, um, and you pass that into a function that function is not allowed to match on the internals of that struct. So you pretty much get a closed box that you can pass around, um, and not have to worry about some, some client of that code, uh, looking, looking and saying, oh, well this has this particular key. You're you're saying I'm no longer giving you any guarantees about the internals of this data structure as the consumer of this code. Amos: I want to use that more often so that people will quit reaching into the data structure. Andrew: It forces orces you to put that in a box Amos: Now I can't change the data structure. What are your suggestions on how much you should spec in a project? Andrew: I spec all the things. There's actually a creative rules that you can turn on that say to, uh, to issue a warning if you do not spec a function. Um, and so that's that exists for Def functions. Um, we took that creative rule and also turned it on for Def P functions cause we are crazy people. Um, and I find that that is really, really good for just human readability. If I'm popping into a file, I'm debugging something. I want to look at the type spec. I want to look at what this thing is supposed to be doing. Um, and I think that any, any knowledge that you can kind of express at the time of writing, um, will just be so much more valuable at the time of debugging. Amos: Awesome. Anna: Thank you so much for coming. Amos: I know the Anna's got to go soon. Chris: Is all the, is all the explained stuff still... Andrew: We're still kinda playing with some of the output a little bit, uh, some, some of the explanations just weren't there or they were wrong. Um, because one of the ways that you can do the filtering is on the output of the short error message. We want to make sure that those are right before we go and issue a one-oh, 'cause that's something that we don't really want to revert back from that that's annoying. We don't want to make people have to fix their, their warnings. Um, and so we're, we're kind of still getting, we're also still fielding some, some bug reports from people. Um, and hopefully, hopefully it'll be in a spot soon where we can actually do the one-oh of that and then feature revisions that are the result of, you know, parser bugs can just be delegated to Erlax and then that can be iterated on independently. Um, that's kind of where we want to get the project to. Um, and yeah, that, that, that's kind of where that that's stuff is at. Chris: Well, that's exciting. Uh, it'll be, it'd be nice to, um, it'd be nice to really take advantage of it. Andrew: I definitely encourage people to run dialyzer and, um, some of the newer release candidates on, on their projects, um, the output is much more sne than, than if you've used it in the past. Um, and I don't know. I, I think, I think using type specs is really important if you maintain a library, um, because it, it it's, it really just makes it easier for people to find bugs and to spot bugs. Um, it does spot improvements and like when you're going through and you're writing when you're writing type specs, it's, it's often illustrative of what your program is actually doing. Um, so I dunno, I, I think they're very good and I wish more, more projects use them. Amos: And it is super helpful whenever they're part of your documentation because it prints them there too. Chris: There's, there's an annotation called type doc. You can, you can actually write docs for all of the specific type. Andrew: One interesting quirk about types is, or the app type, uh, in particular app type P you can write public functions that say in their spec that they returned private types, and that makes it impossible for consumers of your code to type spec. So please don't do that. If you maintain a library, I've had to fix that and did people's libraries in the past that I've like, I think that this should not even be allowed, but, um, you know, here we are. Yeah. Thanks for having me. Amos: I really appreciate it. I look forward to, uh, the explain feature. Cause I think that's gonna, that's gonna be great. Andrew: If you have writer's hands um, please help some of those explanations. They could definitely use an editor too. Chris: All right. Amos: Have a great day, everybody. Chris: Bye.