Daniel: "I celebrated my birthday last Friday outside with social distancing and everything. Everyone was tested and it was not too many people, but I had so much fun. And still every time I have to cough because I have a dry throat, I'm like, ah, but I'm all fine."

dave: "Mm-hmm. Ayyy Well, I'll tell you something, I hope you enjoyed that, I saw some pictures, it sounded lovely."

Daniel: "I did, it was so nice. It was so nice, just like hanging out with friends and everything. Lisa gave me, how do you call these? Like basically, like a thing you put on your head and it looks like an ax has been driven through your head, like it's a plastic ax basically, like a Halloween thing. Because apparently at Halloween, I told her that if I had such a thing, I would open the door for all the children ringing the doorbell, like."

dave: "Yep."

Daniel: "dressed like a zombie and then having a knife or axe or whatever stuck in my head. So she gave me one for my birthday, which I loved and I wore it the whole day. It's just out of reach. I could grab it. I'll grab it later."

dave: "Yeah. Hehehe You could show me other things, you can grab it later. I'm trying to make the connection though, because it's Christmas time, that was a Halloween thing, I know it was for your birthday, but the vision I've got in my head Daniel, is that if you were to get Christmas carols, you might open the door to people singing Christmas carols with that on."

Daniel: "Ha ha ha! The thing is I only say it's Carolas and instead it's like an artist with a lot of play cards just like telling me that I am perfect."

dave: "Oh, okay, yeah, that makes."

Daniel: "The first placard says, say it's careless. And then he has like, you know, one of those boom boxes that just plays like careless from the, yeah."

dave: "That's a love actually reference, isn't it?"

Daniel: "It is! It is. Oh, shall we start the show?"

dave: "Yeah, yeah. Oh. I think we should."

Daniel: "Let's start the show. Hey, welcome to Waiting for Review, a show about the majestic indie developer lifestyle. Join our scintillating hosts, Dave and Daniel, and let's hear about a tiny slice of their thrilling lives. Join us while waiting for review. You're making, are we recording video? You're making faces at me, trying to get me to laugh while I'm reading the intro."

dave: "I was making, yes, we're recording video, but we're not. I know, I know. Um, just cause I could. Um, we're recording video. This, this system we use is records video, but I've got no idea if anybody would really want to watch both of us talking rather than just listening to the part."

Daniel: "Sometimes I feel like we could put it on YouTube or something, just like as a secondary channel or something. But then you probably have to wear something. I don't know. Like, I'm enjoying this, but, you know..."

dave: "Wicked. Yeah, I probably have to get dressed. So for anybody listening, just to be aware, it is currently not winter where I am because I'm in New Zealand. And, you know, southern hemisphere, northern hemisphere, it's all different. It is warm here for, comparatively, for New Zealand. So currently I have no top on. However, I do have... a jacket of sorts around me. So Daniel isn't treated to a complete eyeful of my bare chest. But anyway, that might be a good reason not to put this one on YouTube. I'm not exactly dressed for the occasion. But yeah, another time, another time."

Daniel: "Ah, screw that. Um, yeah, this is, it's actually pretty warm here as well. Like today it hit like 10 degrees or something. That's Celsius. For the people on the other side of the pond, that's about 280 degrees Kelvin."

dave: "Mm-hmm."

Daniel: "ish, plus or minus."

dave: "Wow. You went there. Speaking of SI units, because your brain is probably cooking on that mode, you've been playing the new Kerbal Space Program just before this call, right? And that's all in SI units."

Daniel: "It is finally good. It's finally good. Oh, so it still has like a few tiny bugs. Um, but that game came out in February 23, I think. And with this update, you finally have something to do because it adds like a progression, so you kind of have to unlock all the different parts in the game, like the rocket engine and the tanks and the cockpits and whatever."

dave: "Excellent. Mm-hmm."

Daniel: "and you unlock them with science points. So you have to go to places and then land there and perform science experiments. And now with that also comes a lot of new parts, a lot of new gameplay and a lot of just like bug fixes. And the game is now in a good early access stage. So that's awesome. So, I mean, it was like a frustrating wait for it to get there."

dave: "Oh, that's great."

Daniel: "But I can totally relate to, and I can totally relate to the developers who have been incredibly enthusiastic all the time. And then the player community was always like, where's the thing that you're enthusiastic about? This has bugs because I can very much identify with them because sometimes you build something and you see the potential and you see what is there."

dave: "Mm-hmm. Yes."

Daniel: "brain kind of cuts out the fact that you must not touch this part of the application because otherwise everything crashes. Whereas your users or players, they don't know where the hidden pitfalls are, so they will make it crash."

dave: "Yeah. You get used to, you do get used to the state that things are in and that you're working around when you're in that mode for sure. It's kind of like, I kind of want to say it sort of ends up being a bit like, I don't know, every mother loves their child sort of thing as well, right? Or every, do you know what I mean? Like, it's a horrible analogy, I guess, but like that whole sort of premise of like, you're too close."

Daniel: "I'm out."

dave: "to see anything other than its potential and its beauty. You know, and anybody else takes a step back and that's not always the end."

Daniel: "Or it's the other way around, where you work on something and you only see the flaws, or you only see the difference between what is there and what might be. And so you're always down about your creation. That's more like me with Cinematry Degree. We get a lot of positive feedback, and I'm always like, yeah, but it's kind of shitty because it doesn't have the thing that I envisioned yet, because I always can think of 20 more features that I haven't been working on. And so it's always..."

dave: "Yes. Yeah, yeah. Yes."

Daniel: "unfulfilled, but yeah, KSP2. So KSP2, what did I do? I started a new campaign. I switched it to what's it called? Exploration mode, which is the mode where you have to actually explore and get science points and then I went, I actually unlocked enough parts to get to orbit. So it's like three flights or so. And."

dave: "I think that one's more common. That one's more common for me, to be fair."

Daniel: "I actually managed to get to orbit before the game kind of expected me to get to orbit because I have been playing this game for literally years. So with just the barest minimum of parts, I already got to orbit, but that gave me a slew of science points. I could unlock all kinds of things. I could unlock wings and a jet engine and various like, I don't know. floaty things where I could try to build a boat maybe. And then with those very tiny wings, I tried to build a small jet plane that will just take the new science parts and just fly me somewhere where I can use them. And then I was kind of super completely deep in the zone, just like trying to find the perfect balance for this aircraft, like have the center of lift and the center of... like gravity, like in the right positions and like, oh, now it's, it's tears too sluggishly. Maybe make them put them closer together, but then it's kind of impossible to land. And yeah, I was just like, I got, I got nerd sniped so hard."

dave: "Mm-hmm. I've got this vision of you loading it up and you're like, I know this. It's a Kerbal system."

Daniel: "Yeah, pretty much. That's like from Jurassic Park 2, right? It's the first one."

dave: "Nah, it's the first one. It's the first one. And the original quote is like, I know this. It's a Unix system. Which was it? I thought it was the first, because in the first the computer guy causes all the gates to open and he leaves that really annoying screen saver with him going, on the screen. Yeah, and the girl gets rid of it."

Daniel: "Thought I was from the second one. Right, right. But... My recollection is wrong. I remember it from the book, actually. The book that I read, it had lots of... In between the chapters, there were always chapters of cellular automata with quotes from the fictional Ian Markham. Then it also had screenshots from this 3D user interface that was the Unix system in question. I thought it was the second one. Oh, yeah."

dave: "Ugh. Mm-hmm. Mm-hmm. Okay. Because it was all 3D interfaces for the future. I have made well on that note I've made a reference in our shy notes. I'm going to look up and find out who's right and we'll find out."

Daniel: "We will never know. Literally never. Yeah, write it. If you are watching this on YouTube right now, then write it into the comments, of course. Ha ha ha."

dave: "I'm not putting this on YouTube, we're spanking about it. Nobody needs to see me in my dressing gown on YouTube. Bye."

Daniel: "Like it's very, it gives off a very much like Ken energy, like Ken energy because Ken has this, in the Barbie movie has this like very plushy gown as well."

dave: "Kennedy. Yeah, I'm kind of feeling that, like, living my best Ken life. I am definitely, I am Ken off these days, Daniel. Me. Ah."

Daniel: "Fantastic. I love it. And you had, oh, I saw, I wanted to say, you have like, I never saw, seen your coffee machine before you just, just before we started recording, you sent me a picture. Um, and I think you wanted me to tell like, okay, I'm just getting my coffee and then I'm, I'm ready. But do you have like a proper De Longhi, la specialista? Like this is, this is like a serious machine for serious people."

dave: "Mm-hmm. Yeah, yeah, yeah. Yeah. a serious coffee. Yeah. So, um, we've actually literally just bought that yesterday. It's, it's doing the job. I'm very, very pleased with it actually. Um, yeah, what can I say? It's making good coffee and it can cope with the, um, there's a local coffee I get here."

Daniel: "It's beautiful. Mm-hmm."

dave: "that our previous machine just wouldn't work very well with at all. Like it just didn't get the pressure properly for whatever the grind is. Um, and this thing, this thing's just gone, yep, I can do that. Um, so yeah, I am fully caffeinated this morning, as you can probably tell."

Daniel: "RC very nice and it has like an integrated grinders what I'm seeing I think and also then one of those tempers for tamping down the but it's also but yeah and it's but it's still very manual it's still very like you have to oh god what's the name in English like you have to move the tray with the handle and stuff like that"

dave: "Yes. Mm-hmm temper and it's got like a pressure gauge. Yeah Mm-hmm. Yes. It's not completely automated. If you use the grinder, you've got to move it from the grinding station to the, to the whatever the word is, where the portafilter goes into the mechanism and it pulls the border. I don't know. Like, this is, this is the other thing, Daniel, somebody who really knows their coffee and coffee machines will be like, that machine's all right for a consumer. You know, like it's"

Daniel: "And it's like And it tries to look like a proper machine because it's all made of steel and stuff. But in reality."

dave: "Yeah, yeah, yeah. But it does the job for us pretty well and it's making a good coffee right now, so I'm quite happy with it. But yeah, it's one of the only vices and things that I have these days, Daniel, because I can't really drink, and so alcohol I mean. So coffee, good coffee."

Daniel: "Mm-hmm."

dave: "Yeah, definitely a must. But it's just as well, actually, because I've had this week off work from my day job. And it's, as we record, it's the week before Christmas time. And I think by the time this goes out, it could even be the new year because we've still got an episode ahead of this to put out this week. That's good. We should always record in advance. So then."

Daniel: "Fantastic."

dave: "know, we can keep a bit of a schedule. But, uh, week before Christmas I am all sorted for Christmas presents, this, that and the other. Uh, so I've got nothing to do but please myself at the moment. And this is where the coffee links in because I have been getting caffeinated and working on light beam app stuff, on my the stuff we talk about on this show, my, my apps and everything. Um, so that's what I've been up to. last couple of days and yeah it's been a lot of fun. I've basically been working my side projects as a full-time thing for at least the best party yesterday yeah yeah. So I wanted to talk about this quite deliberately on the show to be honest. I am landing this on purpose onto this point right? Yeah almost crowbarred in."

Daniel: "That's nice. Ah, the perfect segue!"

dave: "So a couple of weeks back, Daniel, and you know this, but nobody on the pod knows this, but you helped me avoid having any excuses. And that looked like this. I have an idea for a new app. I think we've spoken about it briefly on the, at some point on the show before, probably, in that what I'm pulling together is an idea of Go VJ Pro. So like an iPad, Mac OS. version of my video mixing app that is a level above what I've got in the store already. But rather than spend the next year building it and then hoping people love it, I decided I was going to put up a landing page, test the theory, see if I can get anybody to sign up for a mailing list effectively. Kind of road test the idea. So, Couple of weeks back we spoke and I'm like, yeah, but web design sucks. I hate it. Uh, it doesn't suck. I'm just not experienced enough in modern web design. It doesn't suck anywhere near as hard as I whinge about it. I think is what I'm trying to say. Daniel, you, you swooped in and saved me from myself and set up a template with, um, I want to say 11 to, I think it is an NPM templating system. Um,"

Daniel: "I mean, it kinda does suck. Mm-hmm. It's 11T plus tailwind."

dave: "Yes. And you've very kindly stood that up in a Git repo for me, which I mean, we could link that on the show notes if anybody's interested. I think it's a public GitHub repo, right? That it's all in. Yeah."

Daniel: "Sure. But like this, there's better ones. This is kind of clobbered together from the 11T starting project and then modified and stuff like that. And then I kind of pulled in like a few, like a few various other things in from like other, like some of my projects, a bit of like demo code from Tailwind. So yeah."

dave: "Thank you. Mm-hmm. Yeah. OK. But yeah, in principle, you've got that set up for me. And I then went, oh, shit. OK, I can install that, get that running. And it just needs me to spend a bit of time. Like, no excuses. There it is. Anyway, the last couple of days, that's what I've been up to. So I've got a hero image of the app. I've cobbled it together in SwiftUI over the last week, just actually getting a sort of mockup of what I think the app is gonna look like, which is cool, and I can turn that into the real prototype in fairly short order. But so I've got my hero image, I've been sorting all the colors out, all the text out, the rest of it's on the template site that you pulled together for me, and I'm probably... Today, I think it will only take me a little bit of time after this call to sort of start getting that ship shape in terms of actually putting it up and into the world. So yeah, thank you for that. I've got a landing page. I've also figured out, I've got a, like the next thing after a landing page is I need some words for people to sign up. So that means a mailing list system of some sort."

Daniel: "Ha ha."

dave: "So I spent a bit of time yesterday standing there as well. So yeah, because I used to use MailChimp. I don't want to use a service that's going to charge me lots and lots and lots of money when I probably only going to get a few hundred people at best sign up for the thing. So I've gone in on a self-hosted system, which is running on Docker on my own hosting, with a service called Postmark."

Daniel: "Mm-hmm."

dave: "send the emails to people. So that's..."

Daniel: "Are you actually sending any emails right now? I mean, you're just like collecting the addresses, right, for people who are interested."

dave: "I'm collecting them at the moment and yeah, exactly. So again, I can pick and choose later because I'm self-hosting this signup. If I suddenly get, you know, I don't know, 10,000 people out of nowhere signed up and it was gonna cost me extortionate amounts on whatever servers, I could shop around before actually sending emails out. I'm not locked in."

Daniel: "Right. The thing is, I'm not really afraid that you will have to pay incredibly large amounts of money. Because even if you get 10,000 subscribers, and I hope you get 10,000 subscribers, it's not going to cost a lot of money to send 10,000 emails. The thing is though, that sending emails generally from anything you self-host is always going to be a horrible amount of work."

dave: "No one. None. I'm going to go."

Daniel: "Once it's time to really send out those emails, you might actually decide to export your email list and then use any other email service. Like there's a few around."

dave: "Yes. No, yeah, no, but this is the thing. That's probably, I've not explained it so well, but Daniel, I'm self-posting the sign up, but the sign up can talk to any service I want to send the emails. So I don't need to export anything anywhere. Yeah."

Daniel: "Oh, fantastic. Fantastic. Yeah. Because like, don't send, don't like, these days, 2023, do not do not send emails from your own server. It just doesn't work because there's too many layers of spam protection out there, which is all fair and good, but it means that you have to be one of like three companies to actually send the emails."

dave: "It's not worth it. Yeah. Yep. Yep, exactly. So I think that before a couple of years back, I was trying to self-host my email just to see what happened. And exactly that happened is that I couldn't send emails anywhere after a certain point because Google or Apple servers, for example, would say, who the hell are you? We're not dealing with you. But yeah, I mean, this again, this is this is also a good side of self-hosting though, is that the signup system. decoupled from the sending system so I get to choose. At the moment I've got it configured with postmark because they seem reasonable and they work well with the signup listing service that I've got installed. So and I've given it a bit of a test seems to do what I need it to do I think their free tier will probably be enough for me right now and then you know if I have to pay them it will be a nice problem to have."

Daniel: "Mm-hmm."

dave: "I'll have enough people worth paying them to send to. So yeah. Not yet, not yet. It will be by the time this show goes out though, so it should be something like lightbmaps.com slash govjpro. And I'll link that in the show notes. That will be where I put it or I'll put a redirect here if I"

Daniel: "Is that thing live somewhere? Can I try it out? Okay. Because that's what it's for, right? For Govj Pro, which is the pro version of Govj."

dave: "Yes. So yeah, and it's a simple landing page, right? It's a hero image, brief description of features. Be the first to find out when it's available next year. Sign up your email here, boom, Dom. And that should be enough to let me post it on Facebook groups, Reddit groups, and all the other places that potential customers are likely to be. Because, yeah."

Daniel: "Mm-hmm. I love that. I love that idea. We talked about this earlier and also love what you've done with the template that I gave you because mine was all like white background and everything and you posted a screenshot or a screenshot video the other day. And it was all like dark background. It looked very like, like very purpley like it looked like the laser filter in Apple FaceTime."

dave: "Mm-hmm. Yes. Yeah."

Daniel: "Like when you raise like two hands, I don't know if that works in our eye. It does work in our video chat here, but like it just looks awesome."

dave: "Hahaha Yes. Oh, thank you. And yeah, I'll link it in the show notes. People can go and have a look by the time this is out. It will be online. I'm committing to that. Again, this is the thing though, Daniel, I keep saying, oh, I'm going to do a landing page. I'm going to do this. I'm going to do that. And then it flounders because I would much rather be building the app. Right. So, but I'm committed to this. It's nearly ready. It's nearly out of the door."

Daniel: "Yeah, I get that."

dave: "And actually the commitment I then got to myself is that I'm not going to build it, not until I've had a bit of the detraction, a bit of people actually indicating interest. So in my remaining time of this, this break, if you like, rather than piling straight into building the thing, I'm going to actually chill. Uh, and when I'm not actually chilling, I will be adding nodes to my node based video pipeline because that."

Daniel: "Mm-hmm."

dave: "That's going to drive this app and probably several other apps. So it's time well spent in the meantime if you like."

Daniel: "Yeah, that's smart. Yeah, I love it. I love it. And I mean, I get it because I'm kind of the same. I'm finding it also very hard to not work on stuff, it turns out. So my plan for the rest of the year and the beginning for next year was trying to move more into sales and away from programming."

dave: "Yeah."

Daniel: "Whenever I say sales, I mean, whenever I say sales, I always want to be like, oh, let's hoist the sales, my geese. But not those kind of sales, but like selling software. And it's so hard, because every time I send one of those emails or interact with a potential customers or stuff like that, I'm always like, oh, this was hard, but it was nice. Now I should be able to do it. Now I should probably relax by just spending like, I don't know, 15 to 20 hours on the code, but then I will write another email. And it's so hard. And it's working. It's working okay. I'm forcing myself to spend at least half the day on hoisting the sails, but I'm still, I don't know, like the other half of the day, I'm actually still finding myself in the code. But I'm now working more on features that will unlock additional customers, as in... before I was mostly focusing on features that are cool or features that will improve the quality of life for existing customers or features that will just increase the visibility of telemetry deck, which are all incredibly important and they're not like totally gone by the wayside, but I feel like at least for the next month or so."

dave: "Mm-hmm."

Daniel: "I want to be able to, I wanna finish the React Native integration so that I can then go to my list of 10-ish potential customers that I already wrote down. I had conversations with them and I can email them and say, hey, you are using React Native and we now have a React Native SDK, so let's talk more. I'm like, I also, we're also like releasing a..."

dave: "That's cool. Yeah."

Daniel: "I can now I can officially announce that we're also releasing a Flutter SDK, but I'm not going to write that myself. Instead, I actually outsourced it to our friend, Konstantin. And yeah, the code that I'm actually currently writing is performance because the bigger customers are still actually having to wait a bit too long for their query."

dave: "Wow."

Daniel: "like calculations to show up. And the other thing is a huge rework because we need a roles management system because our customers are getting big enough that not everyone should have read and or write access to everything and every single app in their organization. And the thing is though that this is of course, so like in the current status quo on the production server,"

dave: "Mm-hmm."

Daniel: "A user belongs to an organization. So one organization, so a user always has one organization and a organization has multiple users and that's it. And then what I need to do is a organization has multiple roles and then a user belongs to a role. But if I'm already ripping this up and this is basically touching every single API endpoint and everything. And then also there's like"

dave: "Mm-hmm."

Daniel: "permissions that connect to apps or whatever. And I'm like, okay, I'm gonna, if I'm touching this anyway, I'm also gonna include multi-organization support, which is something that kind of comes naturally here. Like if we have the roles, like then, like users can just belong to multiple roles and the multiple roles can belong to different organizations. So this way we can, like I can be administrator of telemetry deck organization and then also read only member of"

dave: "Yep. Yes."

Daniel: "let's say light beam apps. And so I can like look at your data but not like mess anything up. And so it's like, it is really cool. I made a lot of charts. I can actually put one of those charts in the show notes. Just how stuff should look. Then the next thing I did actually is I knew that this would touch most of the"

dave: "Yep. That's cool. That's really cool."

Daniel: "most of the code, just like most of the API code, has some sort of read or write access to data. And so I would have at least to touch a little bit of that data all the time, or like the code. And so I thought about like, how do I wanna do this? And I decided the first thing I need to do is write an incredibly large amount of tests."

dave: "Yeah."

Daniel: "because I won't be able to test all the API functions by hand. So instead, what I want to do is I just want to know where stuff breaks. And so I was actually sick two weeks ago. And I didn't like. And so one day, I was."

dave: "Mm-hmm."

Daniel: "Like my mind was already like very active again, but I was like still tired. So I sat in bed with my laptop and just wrote, wrote like one like API endpoint tests after another. And I have this copilot plugin for Xcode. And especially with the tests, like one, like if you have, like if you write one test for the, I don't know, like read endpoint, and then it will kind of more or less automatically create the ones for the create and update and delete endpoints. Of course, you have to double check them and whatever, but it's still very helpful so that you're writing less. And so now I have test coverage for the 50 most used endpoints for the API, which is basically everything. I know which ones the 50 most used are because of the Metrydeck, of course. And so now I started updating the database. And then of course... because this is all Swift Vapor, nothing works anymore because the models have changed. So nothing compiles anymore because the models are different now. And this also means that I have to touch old API endpoints that I'd rather just not touch, but maybe the iOS app is maybe using an old API endpoint, so I can't really delete it already."

dave: "Oh dear. Uh huh."

Daniel: "So I did write a lot of helper functions that allow me to write all rights management, like checking for, does this user have the right to access this kind of app? Does this user have the right to access this organization? Does the user have the right to write to that data point or whatever? I kind of like extracted all of that out so I wouldn't have to write it multiple times because it turns out that logic was repeated across."

dave: "Oh."

Daniel: "like API endpoints a few times. And yeah, and then after a while, I made it compile."

dave: "That's, um, sorry. Sorry, Daniel, I'm just to interrupt your flow there, but with, with the, was there any option for you to like version the API so you didn't need to migrate old endpoints in any way."

Daniel: "Yeah, maybe I should have versioned my API differently. My API is versioned. We're now on API version three. But the thing is, the versioning doesn't go down to the database level because it's kind of like, at the database level, you have your models and they look a certain way, right? And so because you have your models and they look a certain way, then you have classes in Swift Vapor that also look like the database models more or less. And if you change those classes,"

dave: "Hehehe Yeah."

Daniel: "then even API V1 won't compile anymore because it says like, oh, this line says user.organization, but user does not have an organization property anymore. I won't compile. And so I kind of had, even though I kind of had to, like a typical API endpoint kind of looks like at the top of the function, it will usually check does."

dave: "Yes. Mm-hmm. Yep."

Daniel: "the user who is requesting this have the rights to do whatever it's there about to do. And then it's like a database call. And then it's like convert whatever the database gave back into a specific format and then return that. And so even for V1 and V2 API, which I'm not really using anymore, I kind of had to touch the topmost part of those functions, which deal with, can this user do this? And sometimes like,"

dave: "Mm-hmm."

Daniel: "just updating the database query in a way that will still give me the data that I need. Also, for the very old functions, those didn't use async await yet, because I wrote those before async await arrived in Swift Vapor. And so that was still using a Swift NIO callback style code."

dave: "Is it like, is that promises and futures and that sort of stuff?"

Daniel: "It is promises and futures and also like lots of callback functions. So you have like a kind of the pyramid of death, but it turns out it was very refreshing to actually refactor those two async await code and it was actually like pretty done pretty well. Like they were one of your times where I was just like wrapping stuff because all the like vapor gives you a lot of like comfort functions that will kind of convert from the old style to the new style."

dave: "Hehehe"

Daniel: "And so at one or two times I was like, okay, this is too complicated. If I touch this, I will have to like rewrite everything. So I'm gonna just wrap this into a wrapper and like switch to async await around this. But yeah, and that's where I am. It compiles again. And about half of my tests are green. I'm not sure."

dave: "Yeah. Okay, so you've still got a bit to do but..."

Daniel: "Yeah, so there's still a long way to go because I need to write more tests that make sure that the new rights management actually is in effect everywhere. I don't know how to do that yet. I mean, I could just do it manually and with copilot, I think. So that's the one option and the other is like abstracted somehow because it's always the same thing like, hey,"

dave: "Mm-hmm. Mm-hmm."

Daniel: "just pretend I belong to this organization and just even if I don't, or just pretend I have right access to this item, even though I don't, and then apply that to organizations and apps and insights and groups or dashboards or whatever. Or I could just like write those few lines 50 times. And I'm kind of suspecting that it is less time to just write it 50 times."

dave: "Yeah, yeah. I would say that's a candidate on my, in my vision of yours, the stack that you've got, and this might not gel very well, so correct me if I'm wrong, but like, could you have something where you stand up a version of the backend..."

Daniel: "But yeah, I'm still thinking about it."

dave: "that you're talking to in one form or another, like a mocked back end of sorts, where it's got that data in there. You've got your dummy organization and user and some roles and stuff already written and set. And then you're actually using that as an integration test, in a sense, as well. Yeah, I guess that is what I'm describing. You'd have that in the back end of sorts. backend that's probably running locally to talk to and instead of being anything remote, so it's self-contained. And then you test those cases based against that test data. Right, so rather than testing that the call is responding appropriately, you're testing that, you know, given this data is over here and I try and do this action, can I, can't I?"

Daniel: "Mm-hmm."

dave: "you're asserting that all together. So it's like a test harness, I guess. Would that be a thing? Would that make sense? Or is there a better way?"

Daniel: "Um, it does, it does make sense. So what I'm doing right now is I'm using, um, I'm using just the integrated tools that vapor gives, like, uh, gives you that they have basically various extensions to XCTestCase that will allow you to spin the server up locally. Um, then have like a setup function that will populate the, um, the database. And the database is automatically separate from the, from the, whatever is your development databases, like a test database basically. Um."

dave: "Mm-hmm. Yes."

Daniel: "And then it will also just like populate that, like put the database into exactly the state that you like ask it to be in. And then like call via HTTP, pre-call a specific API endpoint, and then like compare the output with whatever you're expecting. And also like check if the database, like if a second database call basically where like, okay, does the database look the way that I want it to look, you know? So that's what I'm already doing. There's a second way which I'm trying to remember the name of the technology. I think it's called, is it Puppeteer? Basically, Chrome-based browsers allow you to record a set of actions. And even they have a JavaScript-based language that they will translate it to. So I could. You can completely."

dave: "Okay. Yeah."

Daniel: "script a Chrome based browser to say like, okay, navigate to this page, then find the link that says, click it and then so on. And so another option would be to create such an automation and then just have like a complete staging environment set up and then run the automations on there. And then see, okay, does the registrations go through? I haven't done that so far."

dave: "So you'll literally then, so that would take up the position of almost a UI test performing integration test together, right? Yeah."

Daniel: "Right. And that would be a full integration test, I think. And because it would test the UI and the API and the database and everything, the thing is just like, it feels very brittle. Like because as soon as you change your UI, everything just breaks. Like, of course, like if you just visually change it, but then like, I don't know, buttons still have the same identifiers or whatever, that's fine. But as soon as you like, I don't know, imagine the registration has this..."

dave: "Mm-hmm. Well, it will be. Hmm."

Daniel: "another question in there that needs to be answered or whatever, then you always have to update your scripts, which is fine, but I don't think it's the right time for that right now. I think just where I am in the development stage is automated tests for all API endpoints is the right level of safety versus development speed."

dave: "Yes. It's enough. Yeah. Yep, no, that makes sense. That makes perfect sense. I'm at the phase with everything I'm doing at the moment where no tests is the space I need. Yeah."

Daniel: "But yeah. Oh yeah, I've been in that phase for a while as well, especially for the UI. So the web frontend used to have no tests. Now it does actually has a lot of tests for UI components as well. But still, the test coverage is still abysmal. For the API, it always had a significant number of tests for the most important."

dave: "the"

Daniel: "API endpoints and also especially for like units internally. So like, especially a query execution, query compilation, like, because we have our own query language, but the query language is based on Apache Druid query language. So we need to compile it because we have comfort features that need to be compiled down into like basic Apache Druid. And also like various verification stuff where"

dave: "Yep."

Daniel: "where it's like, hey, we need to basically make sure that people also can not write weird queries that access data that they're not supposed to access. So yeah, so after, like once all my tests will have passed, then I need to also update the UI to actually allow customers or to like, you have like switch organizations and also like,"

dave: "Yes."

Daniel: "manage those rights that they have, right? So yeah, that's also something that I really need to do. But I think there's gonna be a longer project that all just needs to be, I kind of have it planned that way where it's very like, I'm trying to be very structured and have like the work split up in very small and tiny tasks because I expect to do this on the side for the next few weeks. Just like in... fits and bursts. And it's of course, it's way more fun to work on this like huge chunks of time that go really deep. But yeah, I don't think that's the right way of working on it right now. But at the same time, it will be finished at one day and then I can also like call up various like medium sized or enterprise sized customers and be like, hey, now you can do the thing that you couldn't do before. And that's really cool. And we should really talk. So hoist the sails."

dave: "Yes. Yes. Wilson."

Daniel: "Um, yeah. Other than I did another thing. I did another thing too. I found during that refactor, we're very early actually, while still writing the tests, I discovered a bug in telemetry decks, query caching algorithm. Um, and it wasn't, huh, it's kind of, it's kind of, okay. So what."

dave: "Mm-hmm."

Daniel: "I do in the API is when you give me a query, and at that level where I'm talking about, all the queries are always in the telemetry query language, even if it's created through a UI or something. Then I will calculate the hash of that query, and then look in my, and then use that as a cache key. And so if that is cached, I will actually just retrieve the cache if it's very new."

dave: "Mm-hmm."

Daniel: "or I will just calculate the query and then store it in the cache. And I had various implementation of generating that hash for the query. Like at first I converted everything to JSON and then just as treated that as a string and then just calculated the hash for that string using the Swift hash value function. And then Swift came out with a hasher or like with the hashable protocol."

dave: "Mm-hmm. Yes. Okay."

Daniel: "And so I can just like say like, okay, this, this struct is now hashable and it will give me various, um, it will give me a, a function that's called. Hasher, I think, where..."

dave: "hasher and then it's got an in and an out variable or something hanging off it."

Daniel: "Right, exactly. Like it will give you like a Hasher object and then you need to append to that object everything that's relevant, like all the different properties and then return the same object."

dave: "Hmm. Which is great because you can choose to exclude or include things from the hash, right?"

Daniel: "Right, so I switch to that and that's kind of how it's been. And the hash value that it will give you is an integer between a minus large number and a plus large number. And that's like used as a hashing key, but also just like to manage the queue during query execution, stuff like that, because we have like a queuing system. And it turns out that"

dave: "Mm-hmm."

Daniel: "The Swift hashing system that I just described to you is specifically made to be stable only during execution. So if I finish my execution, then spin up a new instance of the executable, then all hash values will be different. If I have five different API workers spun up in like,"

dave: "Mm-hmm. Wow."

Daniel: "separate containers, all their hash values will be different. So they won't, even though they share the same hashing server, even though they share the same queuing server, they will just work on their own. They will recalculate stuff five times because they won't realize that they haven't calculated this query before. And I stumbled on this by accident, kind of while writing those tests. And it's not a bug, it's a feature that has always"

dave: "Yep."

Daniel: "been in the Swift documentation and that kind of has been turned on around the time of Swift 5.6 or 5.7. It is, yeah."

dave: "Yeah. And I can imagine there's a security benefit there to some degree, yeah. So how do you resolve this though, Daniel? I've gone straight into engineer mode. I'm thinking, right, okay, what's the fix? This is an issue, what's the fix? Can you give it a parameter or something when it runs the application that gives you like a seed value to pin it?"

Daniel: "There is an environmental variable that some people say will disable this behavior, but it's not really documented and the documentation that I did find was unclear and warned against using it."

dave: "So your next step is going to be to, I guess you're going to have to use some other mechanism right you can't use this to do what you're trying to achieve."

Daniel: "Yeah. So my next idea was, hey, I can just implement like I have, I am in Swift. I'm on an Apple platform. So I have access to crypto kit and I can just use their native hashing function like SHA-256 and implement that. Turns out I can do that on a Mac, but on Linux. But turns out, turns out that Apple has actually read"

dave: "Thank you. Heh. Doesn't exist."

Daniel: "It's out for a while now. They have released an open source version of CryptoKit that I can include as a Swift package, and I can import it. And it has actually a pass through. So if it detects that it's running on a Mac or an iPhone or anything else that has CryptoKit natively available, it will just like bypass everything and just like go through the native functions. But if it's on Linux, then it will actually run on the included functions, which"

dave: "Okay. Mm-hmm."

Daniel: "made me a bit uneasy, so I wrote a lot of tests around that functionality and made sure to run them on various Linux, like on the build server and everything. But yeah, this thing works now. I have stable hash values. In addition, they are now, because I'm returning them now as a string, which as a hex string, so they're a bit shorter and less collision because I can actually..."

dave: "Yep. Yeah."

Daniel: "include a larger range of values. And I have reduced the average query calculation time from 0.4 seconds to 0.2 seconds and the maximum query calculation time from 35 seconds to 17 seconds. This is amazing. Yeah. And I'm like, that was like basically free."

dave: "That's cool. That's fantastic. Yeah. Just wondering something out loud, Daniel, because these two concepts have now combined in my head. after the duration of this call, right? So you've been looking at roles, and now you're talking about caching queries. How does the role stuff interact with that query cache? Right, so for example, I have, I make this query, you know, I'm user, I'm an admin user, and then."

Daniel: "Mm-hmm."

dave: "Ten minutes later somebody has taken that right away from me and I shouldn't be able to see that anymore. What happens at that point? Is there an issue there that could happen?"

Daniel: "Yeah, we cache words after compilation. So compilation always runs. And what compilation does is, one, it compiles down the specific telemetry deck functions and comfort syntactic sugar into this regular druid stuff. What it also does is it modifies your filters in a way that it will generate a"

dave: "Hehe. Yes."

Daniel: "filter on top of all the filters that you defined manually that will always include all the apps that you have read access to and wraps that around the filters. And that's one of the things that makes this query language more safer than SQL, which is very much parsable because this language is very much like..."

dave: "Mm-hmm. Right."

Daniel: "converted to JSON, but in telemetry data, actually I wrote a huge amount of structs and struct definitions that map to that. And I have like a large amount of tests that has the test conversion to and from those data sets. And so here I can very safely say that, okay, if I take this filter object and then I just wrap around like a filter on top of that, even if like it doubles the filter that like"

dave: "Mm-hmm."

Daniel: "the server won't care. But I can always guarantee that the outermost filter is the one that says, hey, this is for apps A, B, C, D, and E. And so because that isn't cached, the compilation will always be. And then the compiled query will be cached. So if the same person with the same permissions runs the same query again, then it will result in the same compiled query. And that also touches pre-compilation, which is if a query actually takes longer than two seconds to execute, what I will do is I will also chuck it on a list of queries to be pre-compiled, which is whenever the server is a bit idle, it will just like take one of those, the one that hasn't been compiled for a while, but and then just like rerun it through the server. So whenever you open your UI, you have"

dave: "That's cool."

Daniel: "still you see still see outdated data, but it's less outdated. And it's nice to look at while you like the newest version of this is calculated."

dave: "That's really neat. That's really, really neat. I love all of that, Daniel. And I'm sorry if that was a bit of a security 101 there with the caching. I just, yeah, I was thinking out loud in the moment. But yes. But no, this is cool. We've probably gone a bit down into the weeds with some of the database."

Daniel: "Oh yeah, of course, of course. Like, oh yeah, probe everything about it."

dave: "end of this stuff, but it's really good to get down there and just hear a bit more about how it's structured. I know you've put a lot of thought into all of this and continue to put a lot of thought into all of it as well. Yes. I look forward to a future where I'm talking about adding tests for my video pipeline because it's actually given me a successful product."

Daniel: "It is just so much fun though. Hahahaha."

dave: "I think at the moment that's the stage I'm at is like, I'm building all this stuff, but, and I had customers, right? I have people subscribe to the original app, to the basic Go VJ. And it's just, it's not big enough to justify me spending a lot of time getting all of the test structure laid out and the rest of it. I kinda know what I wanna do when I get there. You know, like I wanna have some test images and I wanna have it."

Daniel: "Mm-hmm. Mm-hmm."

dave: "kick off filtering them and various different pipelines and then asserting that the output still matches what it was on the last save test. So I do some sort of caching the output and then reading that back in to be the baseline that it's testing again. So I can check everything's still working the same. Is it still filtering the same? Does this pipeline of lots of things linked together still give me the same type of outputs as it did the first time I built it? I think that's probably the direction I'll go for testing something that's quite visual. But at this stage, if I did that, I would be spending all of my time writing tests because I'm also changing stuff every other minutes as well. You know, maybe not like that, but over there."

Daniel: "Yeah. It's the, it's the balance between move fast and break things and, and be testable and whatever, but I think you're on the right side of that, of the balance. I think writing tests, like the TDD fans will probably disagree, but like the writing, but writing tests for me is a good time is when, before you go on a big refactor or, um, when something breaks and you need to find the bug anyway, then you can like, try to find out. Like, but."

dave: "Yes. Mm-hmm. Yeah."

Daniel: "It's a good tool to have, but my personal philosophy is like, I need to feel some kind of pressure to write the tests. Otherwise, I think they're a bit of a useless exercise. And yeah, people will disagree with this. That's just my personal philosophy here."

dave: "Mm-hmm. I feel the same, certainly when you're trying to just get something started up and get something operational. Like, if you've got a... and this is always the argument that I'll come back to, is that if you've got a defined scope, a defined set of acceptance criteria, it's not likely to change very drastically anytime soon. Then, you know, fill your boots, wrap all the tests around up as best as you can. You've got everything you need to know. Boom, you can even do TDD to begin with because all of that is down. If you're in a very fluid, rapid, changeable environment where I don't know what this thing looks like when it's finished, right? I'm still just noodling and building. To me, tests will run counterproductive to my flow. Now, TDD advocate will turn around and say, ah, but you know, if you were to be able to build your test first, you won't do so much noodling because you'll actually be clear about getting your situation first, but that's not actually the way I work. Yeah."

Daniel: "They say that, but yeah, the way I like the way I think we the way we both work. I've never worked with you on software really, but I think we're very similar in that we kind of sketch using code and kind of have like lots of iterations and like tests hurt that process, I think. So, so for me, it's more like a..."

dave: "Yeah. Yep. Yeah, they do. They do."

Daniel: "have like a push with lots of iterations and then something can go out and then that's the time to write tests once a certain area of the application is kind of stable than tests. And also like start from the bottom, start with the most basic but still breakable things like with my like query stuff and then go like."

dave: "Yep."

Daniel: "Work your way up to the, like test the UI at the very latest. Like UI tests."

dave: "Yes. Yeah, I mean, we could go around and around on this topic. I think me and you are both pretty much in alignment. I totally respect and understand this sort of TDD philosophy. I just think that for me, the environments where that really shines and really works are a subset of the broader environments that I've ever worked in."

Daniel: "Hmm."

dave: "And then for how I work personally, like you say, it runs counterproductive to that sketching sort of mentality. Who knows? You know, I mean, there's other ways of approaching this in the future potentially where, you know, automation, copilot, those things perhaps makes that a little easier in some ways. I think my flow would still be the same, but I might have more tests wrapped around if I could just sort of go. Hey, this is the state of it right now. Write me all the tests that you think are relevant here, right? And walk away, and it does its thing."

Daniel: "Yeah, yeah. That does work surprisingly well in the JavaScript environment. Like I have the, I don't know, I think it's still in beta, the copilot two in my Visual Studio Code. And you can actually select a group of functions and be like, hey, just test the status quo for that. And of course you will still have to read through that. And of course you will have to need to correct it. But for me, it feels like an improvement. Davis is writing in our show notes document pissing off the TDD advocates. And I also just pissed off the people who don't like co-pilot for understandable reasons."

dave: "Yeah. Hahaha! Yeah, I was just thinking actually, I'm kind of almost one of those people, but if I can get a local version running, I'll be happy."

Daniel: "HNNN Oh, that would be great. I think someone had, like not GitHub of course, but like someone had actually a version of that running. So like an open source model that kind of replicated the copilot functionality. So it's still too much computing power to like really run it locally properly. But like, I mean, computing power is something that probably will increase."

dave: "Mm-hmm. Yep. Yeah, but I'm very sorry Daniel, but I need to finish up here on this call because my day is about to go from 0 to something else. But I'm not going to say 0 to 60, I'm on holiday, but I've still got a few things that have got to get sorted."

Daniel: "Ha ha Also, I mean like 60 kilometers in hours, not that much. 60 kilometers per hour, that's about 12 parsecs. Yeah, no. I'm gonna launch some more rockets, I think. This has been like so much fun. It's been like one of the nerdier episodes, I think, but still. People, listeners, thank you so much for listening. Rate us on iTunes."

dave: "You've got kerbals to play. No. Yep."

Daniel: "please, because that's really cool. If you have comments or questions, send us emails at contact at waitingforreview.com or write us on the interwebs. Dave, where can people find you and write messages to you?"

dave: "Mm hmm. You can find me on mastodon at soci uh slash at Dave and you can find out about my apps on lightbeamapps.com as well and as we said earlier on by the time this goes out you should be able to take a look at that landing page which will be lightbmets.com slash govjpro. How about yourself, Daniel?"

Daniel: "Fantastic. Yeah, find me at daniel at soci on Mastodon, telemetrydeck.com of course, more about Telemetry Deck. Also, I have created a threads account for Telemetry Deck. It's called telemetry underscore deck. And because Lisa doesn't have access to that account, I'm posting lots of shenanigans there."

dave: "Ha ha."

Daniel: "and not a lot of corporate PR right now. But it's still, I feel like my main home is still Macedon. Yeah, have a great day. See you soon, Dave."

dave: "Yeah. Catch you later, Daniel."

Daniel: "Byeeeee!"