Daniel: "Hello, ladies and gentlemen, those in between, those outside the gender binary, I hope you have your cake ready, I hope you have a nice drink, because it's podding time. Welcome to Waiting for Review."

Dave: "It's podding time indeed. You can have your have your cake and please do eat it. Such"

Daniel: "It's podding time. We're just leaving Poddington station. There was on Mastodon actually a discussion about how people now say pod instead of podcast"

Dave: "a strange phrase. Why would I? Steady, steady. So. Yeah, yeah, I'm kind of down with it,"

Daniel: "and I just love it so much, it is so weirdly cringe, it's fantastic, it just gives me energy. podding."

Dave: "to be fair. Yeah, I'm on the pod. We're podding. And honestly, it makes sense, actually, in"

Daniel: "Welcome to the pod. It does. And it makes me think of like a space capsule, you know, but everything makes me really think about space travel, so I mean, it's just my thing, apparently."

Dave: "a sense of how language shifts and evolves. Yes, yes, we are bringing. You're obsessed. I'm pretty good. Thanks, Daniel. Yeah. Yeah. It feels like it's been ages again since we last spoke. We've been a bit disjointed with with episodes and recording and various other things going on with life. But for the listeners, I think this will come relatively on a usual"

Daniel: "Dave mate, how are you doing?"

Dave: "schedule, to be fair. So anyway, yeah, I'm good. I'm knee deep in 30 versus seems to"

Daniel: "Hello."

Dave: "be as always these days. And my app Topiary has now properly launched since we last spoke that that I was building it. I had some feedback, I think the last time we spoke and I was about to launch. That's that's been off and into the ether and people have actually bought"

Daniel: "Fantastic, amazing. So let me see if I can describe Topiary. From what I understand, and I've tried it out, Topiary is a tiny app that helps you prune and generally manage the people who follow you and the people you follow on the Fediverse."

Dave: "it as well, which has been cool. That's always a good test of a new app. Yeah. That's right. So it's been the proof of making to test decay, if you like, for me. So I wanted something"

Daniel: "It's not specifically Mastodon, but basically everything that Toot SDK supports."

Dave: "that I could just hack away on that was not as involved as building a full client. And then I had an actual use case for this, which is that successive waves of people coming over from Twitter to the 30 verse and mastodon haven't always worked out for everybody. Right."

Daniel: "In this case, it's my home."

Dave: "People have hit it, played around and then decided it's not for them. And that's fair enough. But then, you know, it's been a few months and I realize I'm following a bunch of what are effectively dead accounts. And that just to me, that that feels like something I don't want to be part of. Right. I'm following my following number or following list of people is sort of potholed with accounts that don't really post. It doesn't really matter. Like you could ignore them, you know, or they might come back. But for me, I was looking and going, well, can I see who's inactive and can I filter that? And so before I knew it, I was making an app and that's that's where topiary comes in. So topiary means, if anybody's wondering, it's the process of pruning a bush or a tree into a special shape. Yeah. You know, when"

Daniel: "Or even a bonsai."

Dave: "you see those. Yeah. Yeah. And my surname is Wood, so there's almost a connection there. Yeah. Yeah. I've been been debating, like, do my apps all fall under this theme and is it some sort of cheesy thing that links them all together? That could be the case as I"

Daniel: "Oh, I didn't make the connection before."

Dave: "release future things. But so to describe it from my view to a potential user, it's a case of you can sign in with your Fediverse account, your Mastodon account or whatever. It will then load all the people you follow and all the people who follow you, because it doesn't always work both ways, adds it into its own store and then presents that"

Daniel: "Okay."

Dave: "information back to you in a way that lets you browse all of the accounts. For the first"

Daniel: "Okay."

Dave: "sort of version 1.0, the main thing it focused on was the people you follow and being able to split them based on posting activity. So sort them by either everybody or by people who posted in the last 30 days or by people we have no record of any post for whatsoever. And that lets you then go down, you can look through people, you can click their profiles, see a bit more about them, you know, see their actual profile information. And you can make the choice then as to whether to unfollow them or what. And if you unfollow them, then they get put in a unfollowing list and you can go through and you can refollow people if you've made a mistake or you can clear them out of that and say, hey, I'm done. And"

Daniel: "Okay."

Dave: "that's kind of it for version 1.0. So that's in the app store right now. And I want to talk a bit about the release process in a minute, actually, because I tested myself on a couple of things with that this time around. But my headspace for the app at the moment is very much on that next version. So I shipped it, got it out there. It's had"

Daniel: "Okay."

Dave: "pretty decent reception. I've had a lot of people pay for the in-app purchase to update. If you update to sort of the pro version, then it does a couple of things. It turns off adverts because I display adverts in the feed of people that it shows. And we can we'll talk about that in a second as well, because, yeah, because they're not proper ads. Yeah."

Daniel: "Oh, yeah, I'm super interested in what kind of adverts you're showing."

Dave: "But you've got your list of people there. You can there's adverts in that feed. And if you update to pro, they get removed. You also get some bonuses for being a pro user. You get the ability to change the theme of the app between a few different colors and everything updates really nicely and fluidly when you do that. And it's quite satisfying."

Daniel: "Yeah, it looks really cool too. I very much like it. By the way, did I make that feature request before?"

Dave: "You get a little little haptic feedback as you change the theme as well. So. Thank you."

Daniel: "Filter by people I follow that don't follow me back."

Dave: "Yeah. So that's where my headset for version one point one, actually. So the next release, it's going to show you everything split between either mutuals. So you both follow each other followers. So people who follow you, but you don't follow back and following, which is"

Daniel: "Okay."

Dave: "the inverse of that's people you follow, but don't follow you back. And that's a bit of a mouthful. But when you play with it, it becomes very obvious. And then from there, it's a much more useful split overall. It means it needs to load a bit more data. But"

Daniel: "Okay."

Dave: "that's that's fine. And I have updated it as well to be using a database underneath instead of using an object store through boutique. Boutique is lovely, by the way, boutique is"

Daniel: "Okay."

Dave: "really, really good for a very quick and simple object store. But I'm now in that territory where I need to do some more complicated things with the data. And I reach straight for a"

Daniel: "Hmm."

Dave: "library called GRDB, which, yeah, yeah, we're using it in the company I work for during the day. And the more I come across it, the more I love it. So it was a case of like, okay, if I'm going to go for a database, that's what I'm reaching for. And by the way, boutique has a database layer, it's got SQL like it's underneath, and there's a lot of ways of working with it. But I hit a couple of limits, just in terms of how I was querying the data that"

Daniel: "Oh, cool. All right. Have you have you considered a Marko Amund has a new database layer? I think. Blackbird."

Dave: "made me want something a bit more database-y. Yeah, I wanted something tried and true, to be honest, and GRDB kind of hit me like that. Also, the author of GRDB interacted with me"

Daniel: "Yeah, I get that. Yeah, that's cool."

Dave: "a few weeks ago on the Fettyverse, and I was like, oh, he's seen something I've done. Sorry,"

Daniel: "Also, I kind of dislike the fact that this is named after military hardware. Like the Blackbird library is named after the, is it the B-52? Like the stealth bomber, basically. And I'm like, ah, it feels a bit, you know, like maybe if you're American, maybe it feels different."

Dave: "what is that? Oh, yeah, yeah, yeah. Yes. Yes. Mm-hmm. Yeah. Yeah. Yeah, sometimes these names don't translate that well across. I hadn't thought about that. I just thought"

Daniel: "If you're not American, it feels a bit iffy. No, and if you listen to ATP, like it's very obvious that he named it this because he saw a YouTube video or something,"

Dave: "of the bird itself rather than the plane. Yep. Mm-hmm. It's a whole thing, yeah. For anybody wondering what on earth a Blackbird jet looks like, if you ever saw the X-Men"

Daniel: "or he's just really into the tech of the aircraft, which I can understand, but still, it's military."

Dave: "cartoon in the 90s, or the films, that's what they fly in, actually. Because it goes. Yeah, yeah, yeah. So to me, it'll always look like the X-Men's jet, and I'll have fond memories"

Daniel: "Oh, I didn't know that. I didn't ever see that."

Dave: "of that. But I digress. That's fine. That's fine. So I was knee-deep in verbally describing"

Daniel: "I did kind of drag you off your trajectory."

Dave: "where the app's going next. So yeah, so the release process was fun. A few things I'm"

Daniel: "You wanted to talk about the release process, I think."

Dave: "trying to do. When I release a new app, I'm trying to have a system for setting up a website. So I'm using a... Somebody made a Jekyll-based template for an app website, and it's brilliant. You have a YAML file, you configure the app ID, the ID number of the app from the App Store, and then put in the text you want to show for a few sections and a few other bits and bobs, and it creates this site for you. You can generate this static site from it. And it's cool because it directly links the things like the app icon straight from the App Store. So it's kind of a living site in that sense. If I change the icon or anything in the App Store, the site changes too. And it handles everything like that. Little thing"

Daniel: "Hashtag drinking sounds."

Dave: "of if you've got the app installed, then you can see an open app overlay at the top on Safari on iOS, or you can see an installer from there. It does everything really quickly. You know, I had a site up and running from it within like 10 minutes because I used it"

Daniel: "Drinking sounds that only the transcript will hear. I know, I know. Yeah, that's cool. Which color scheme did you go with? I didn't even see the site, actually."

Dave: "before for my previous app."

Daniel: "Let me have a look."

Dave: "It's got a green sort of color scheme in the same. Yeah, that's right. Yeah. So it's all about this sort of green thing, this tree thing. And yeah, so that sped that up. And I'm not registering domains for new apps anymore. That seems like a path towards just having lots of domain names that get cluttered up. So everything is just hung off of lightbeamapps.com"

Daniel: "Of course, and the icon is green as well, right?"

Dave: "slash whatever the app name is. So that was up and running real quick. So that's cool. And then the other thing that was new for me is I've made screenshots for years using UI tests and just automating the production of them and then having a script based on"

Daniel: "Yeah, that makes sense. I'll put a link in the show notes. It's on my old blog that no one really reads."

Dave: "a blog post you wrote actually, Daniel. I have a script that goes through. Yeah, yeah. It is. It is. And I come back to it often for this sort of stuff. But I have a script that goes through and runs all the UI tests through that mechanism to go and generate all the screenshots for all the different sizes. And it's good, because you can feed the different types of device through and get all the sizes of screenshots that the app store asks for. But those screenshots can just be a bit dull. They're literally just views inside the app. And I wanted framed screenshots showing the device inside of a real device frame and with headers over the top with text and a gradient of background and blah, blah, blah. I wanted to make it nice and sort of slick. So this time around, what I did is I started a Figma board. I went through a process of figuring things out. So I hadn't actually realized and I should have realized this is something I really should have known. But I hadn't realized that the big deal with screenshots is not that they are literally showing the right sort of thing for that device as such and matching the device that you're looking on. But it's that they match the resolution that the device wants"

Daniel: "Right, right."

Dave: "to display. So doing literally all of these screenshots at literally the right device for the size that's being asked for, that's fine. That works. But then when it came down to showing a device in a frame, I didn't need to do that. So I picked one device and I picked the iPhone 14 Pro to display for everybody. And I essentially had this process where I made an asset and Figma from it. I then made an asset of the phone for each one of the screenshots I wanted. I pulled the screenshot from the iPhone 14 Pro into that. And then I was able to make different slices. Slices is a Figma term, but different areas of content that are the screenshots. And I did one size, put my text in, put my backgrounds in, put my assets in to make it all nice. And then I just duplicated and resized for all the other different resolutions that Apple asked for and made sure everything was censored. So what it meant was that if I changed those screenshots inside of the app frame, then I just need to change the asset and everything else changes and I can just re-export them and I don't have a lot of editing to do. So the end result of that is my screenshots for this app are probably the best out of any app I've put into the app store myself. And I'm really happy with them actually. I need to go back and do the same for GoVJ and I"

Daniel: "Right."

Dave: "guess eventually for my video to audio converter. So my other apps. But yeah, it was an interesting process and I'm a little bit more proficient with Figma now because of it. And that's definitely going to be what I do in the future. So yeah, that was a lot of learning just to sort of make sure that I was working with Figma in the right way. And screenshots are good. App is good. The app website was good. I actually got through app store review for the first"

Daniel: "Fantastic. Ooh, very nice. Very nice. Congrats. I'm going to look at the screenshots. And when I open the App Store, the first thing it does is like, hey, do you want to get three months Apple Fitness for free? And I'm like, wait, wait, wait. I have, what is the thing where you pay Apple and you get everything?"

Dave: "time without any hitches for a brand new app, like first try. Yeah. Yeah. Why are you appetising"

Daniel: "I have that, Apple One maybe, I don't know. And I'm like, OK, why? So now I click no, thank you. And then it says, hey, you have a new Apple Watch. Do you want to have three months of fitness with that thing? And I'm like, no, I want to use the App Store. Come on. But so Topiary is actually, if you search for Topiary, is actually the second result."

Dave: "to me? Yeah, yeah, they're getting a bit bit trigger happy with the ads now. Okay, cool,"

Daniel: "So that's really good. And I love this screenshot."

Dave: "cool. Everything's probably something about trimming trees, you know? Oh, fair enough."

Daniel: "No, the other two are like one of those idle games, basically. Yeah, I love the screenshots. They're very pretty and they're very helpful."

Dave: "Yeah. Thank you. I mean, you know, the screenshots, they do what they need to do, but they're"

Daniel: "I like them."

Dave: "just a little bit nicer than what I was doing before. So yeah, I'm very pleased with that. Well, that was something else that I had fun with, with the app. They don't exist. They're completely fictitious, the accounts that show. And I actually, instead of stealing other people's profile pictures or anything like that, I used thispersondoesnotexist.com and"

Daniel: "Did you select specific accounts to show or are they just accounts that you follow?"

Dave: "Dali, I think, or one of the other ones. I might have been mid-journey. I can't remember"

Daniel: "Ah, very nice. Fantastic. Yeah, that's really cool. Oh, and of course, it says usage data is not linked to you because I assume that you're using Telemetry Deck if you're using any, because it says usage data."

Dave: "now, but I generated fake people faces. Exactly. Yes, exactly. So yeah. So I've gone straight"

Daniel: "So that's why I'm like, okay, you're probably actually collecting usage data."

Dave: "in using Telemetry Deck from day one. To be honest with you, I've not sort of splunked the data yet and had much of a look around. I've been gathering some obvious bits, some typical bits that I like to gather, like first launches and things like that. So that gives me a heads up before the app store updates in terms of people installing the app. But I haven't gotten into any sort of nitty gritty of funnels or anything like that with it. Like I've set them up, but, you know, it's early days and that picture of sort of volume, if you like, isn't really there yet for me. Like plenty of people have downloaded it and"

Daniel: "Okay."

Dave: "bought it. But like I say, it's there, it's doing its thing and this is cool. But I know that I'll be using it in the future for decision making. And that actually probably will link up with something that I think we can talk about, about Telemetry Deck."

Daniel: "Fantastic. Telemetry Deck. There's still exactly one podcast that we're sponsoring, which is this obscure Formula One podcast. And I assume no one who listens to this podcast knows who we are. But I just love how one of the hosts, he has like a really nice voice and he always reads it like telemetrydeck.com and I'm like, oh, worth it. I have a question about the Fediverse, which we might, I'm just going to drag up from the show notes a little bit. What the heck is a calc key? Because like half of my feed has been talking about calc key."

Dave: "Go for it. Oh, good question. Mm hmm. Yeah. I'm going to go for it. OK, so the very quick version is that CalcKey is a different type of Fediverse server. So"

Daniel: "And yeah, I was like, yeah, I could probably find out, but it's more fun to ask you."

Dave: "they're all like Mastodon is a server type, right? And there's other ones, there's ones called Pleroma and Akkoma, and they have various different open source lineages that I'm not going to go into on the show, different forkings and. Yeah, yeah, yeah. And sometimes a bit"

Daniel: "It's worse than the Linuxes."

Dave: "of associated drama with each split as well. So not my circus. A little bit, a little bit. But CalcKey is a new one. It's been around for a while, but it's new in a sense that people are really sort of starting to get into it and discover it. And it differs from"

Daniel: "We've definitely transplanted the Twitter drama over to Mastodon or the Fediverse."

Dave: "Mastodon for me in a sense of its API that you talk to to work with it as a client app is different to Mastodon's. So what that means is, is if you have a CalcKey account, you can't currently connect with TootSDK. That will change because I'm using TootSDK, within the SDK. What we're going to do is we're going to extend out the subsystem underneath to support CalcKey. And then when that happens, everybody who updates to TootSDK will then"

Daniel: "Fantastic, fantastic. So why is it called Calckey? Is it about calculation and or keys?"

Dave: "be able to support it. Yeah. Honestly, I have no idea. The one that preceded it was called MissKey. Then they had a split and this one's called Calckey. Yeah. Yeah. One of the reasons"

Daniel: "Oh, they had a split, of course."

Dave: "I think it's gotten so popular is because it is. It's quite fun. It's got a lot of whimsy in its web UI and you can do interesting things like I can make a post and have text in the"

Daniel: "That is actually very helpful for a lot of applications."

Dave: "middle that spins if I want to or other. Yeah. This is it, right? A lot of people viewing"

Daniel: "I hope that all the third party clients will support that."

Dave: "these posts through Ivory or any other client app from their Mastodon side, they never see"

Daniel: "By the way, congrats to IceCube's app for winning the Swift Community Award."

Dave: "this stuff, but it's just posts. But if you're. Oh, that's cool. I didn't see that. But yeah,"

Daniel: "Yeah, that was awesome. Like I was watching it pretty closely because Telemetry Deck was in another category also nominated. We didn't win against RevenueCat, which is fine because RevenueCat, they're like quality people and they're really cool. Next year though, next year though. And I think both best open source app maybe was IceCube's app, which is also a Mastodon client that's also pretty nice."

Dave: "they don't. Mm hmm. Yeah. That that's right. Yes. Oh, that's cool. That's really cool."

Daniel: "Yeah, I'll add the list of the winners to the show notes I think."

Dave: "Yeah. Yeah, for sure. They're there. No, it's a good app and it's definitely been been useful to have their open source. And it's been useful to me for them that what they've published because actually, when I was building things up with two test EK, one of the things that we hit was displaying emojis and in posts, it's actually a bit more involved than just going, hey, I want to display this text with some emojis in the middle of it. And ice cubes is using a library called emoji text for that. And so that helped me out being able to look at that code and go and go and find that. So that's that's actually been useful all"

Daniel: "Very nice. Very nice."

Dave: "the way through into type. You're actually using that that other library for emojis. They just need to use this library. I say that they don't because they're using UI kit,"

Daniel: "Speaking of, when is Ivory going to support the stupid custom emoji? I think it's all handcrafted UI kit in Objective-C even. But I feel with them because like TapBots is just one developer and one designer. And it's very similar to our situation."

Dave: "I believe, rather than Swift UI, but. It's a very artisanal app. Yeah. Yeah."

Daniel: "And oof, I tell you like somehow I added various ways that you can basically open GitHub issues or GitHub discussions from the UI because I want the feedback."

Dave: "Yes. Yes."

Daniel: "Like the feedback right now is arriving via email, via Slack and via 14 different channels. And I want people to just feedback and bug report via GitHub. But it also invites a lot of more feedback, which is awesome. But I'm so far behind. It's horrible. I get such a... So if you wrote to me, I will read it. I might even answer, but it will take some time."

Dave: "Yes. Oh. Yeah, go for it."

Daniel: "But I can tell you what I've been up to with Telemetry Deck over the last few weeks. So I finished the whole... The final saga is more or less finished for now. There's a few improvements, of course, coming up. So basically, it works. And so the next big thing that I really want to work on is A-B testing. And that's for two reasons because I'm actually going to present a talk about A-B testing and the theoretical backgrounds and how you should do it."

Dave: "Yeah. Yeah."

Daniel: "A platform agnostic, basically, at AppDevCon in May. And I'm also... So many people have just asked me to add A-B testing."


Daniel: "It's just a really important feature. And so I wanted to really read into how to do that properly. Funnels are conceptually pretty easy because you see exactly... From each step to the next, you see, like, hey, this step has, I don't know, 17,000 users and this next step has 15,000 users. So I see like, oh, yeah, number A is larger than number B. But with funnels, what you really want to do is you want to have a decision, like which thing is better and how do you specify better."


Daniel: "You specify it by basically giving you a confidence value. And how do you define that confidence value? And I kind of fell into a rabbit hole because I'm about three..."


Daniel: "I have like three lectures from Stanford's data science course now. MIT has various data science lectures that are really good and well presented as well. I think I linked the Stanford one in the show notes. But so it turns out that... That was one of the examples of the Stanford lecture and it really stuck with me. It's like when you do... Imagine 100 years ago, people would, for example, try to find out like which type of fertilizer, that's the words I was looking for, which type of fertilizer is better for a field. So they would have like two parcels or maybe multiple lots and use the different fertilizers. And then at the end of the summer, they could look like which of these lots would yield more grain or fruit or whatever they would plant there."


Daniel: "And that completely works. And it gives you like, you can calculate a p-value, which is a probability value, which is basically assume that both fertilizers are exactly equal. And then the deviation from that is your p-value basically. And you want... No, the one minus... The deviation from that is one minus p. Anyway, so that's easy because... Or that's actually well done because before you start your experiment, you define how long it's going to run and at which point you will look at the data. And that gives you a pretty solid end result. It will tell you like,"

Dave: "Okay."

Daniel: "fertilizer A is maybe 45% better than fertilizer B with this amount of confidence, let's say 95%. So that's really cool. But nowadays with real-time data, what you usually have is people who set up an A-B test, however you want to do that technically. And then basically each day they will look at the data and they will go, hey, this is... Okay, so how's my p-value today? How's my confidence today? Yeah, okay, I'm going to look at it tomorrow again."


Daniel: "Like I'm going to run it another day. They basically want to finish the test as soon as possible. So they're just waiting until the confidence"


Daniel: "reaches a certain amount and then immediately they stop the experiment. Right. The thing is that because you didn't specify the amount of time the experiment will run in advance, this confidence value will more or less fluctuate wildly while you are like for every day. And it is a mathematically very likely probability, about 60%, I think, that even for an A-A test that will compare exactly the same thing to itself, it will reach an over 95% probability that one thing is clearly better than the other if you look at it every day over the course of the thing. The more times you look at your test, the more chances you get for a false positive. And you really don't want that."


Daniel: "So there are incredibly complicated ways of dealing with that. And I've read into those and they are hard. Well, they're hard to understand. I think they're not that hard to implement. But still, I think this is so interesting. I never knew that there was this whole rabbit hole down there. And I felt like I was pretty well versed in statistics already. But no, there's just universes out there that I don't even know about. So yeah, I think what I'm going to do regarding implementation is at first just implement it the naive way."

Dave: "That's a massive rabbit hole."

Daniel: "And maybe I'm going to try to force people to commit to a date at which they will actually evaluate the data. Maybe we'll just have a little note that says,"


Daniel: "hey, this confidence value, don't rely on it yet. Because then I can basically just add the nicer implementation of the really good confidence value at that later. I think in the end, the UI for it will be very simple. Because it will just say, hey, A is better than B or B is better than A with, I don't know, x% confidence. Or it will say, I don't have enough data. You should wait a little while."

Dave: "Yeah. Yeah, exactly."

Daniel: "And I think that's what needs to be happening. If I want to be clever about it, I can also chart the changing confidence over time to show that it's actually varying a lot and to maybe encourage people to wait until it settles down into a specific part of the chart and not swing so wildly."


Daniel: "I don't know. But yeah, the idea is to have a lot of complexity hidden, especially since with the first implementation, I'm going to ask people to deal themselves with putting people into cohorts. So basically deciding whether a user is in group A or in group B and then just telling me. And also just what are the features that are being switched? I also don't know."


Daniel: "Those are maybe features in the future on the roadmap, like feature flags or cohort management. But that's a completely other topic. And I want to do just the algorithms first. And I think that's already helpful and useful, even if you just have to roll a random dice or something to decide which group your user falls into. Oh, yeah, especially since it turns out that just with all the feedback, it's really good. It's really nice to iterate on stuff."

Dave: "if you want to just pay once, it's a bit more money, but you can do that. And I've, you know, iterated that just by tweaking it manually, but I've had nothing other than looking at revenue"

Daniel: "Right."

Dave: "and going, am I happy with this now? To sort of indicate how much better, like one changes over the other. And that would be interesting in the cohort definition, because actually it'd be after a time-based definition rather than picking A or B at the time, right? It's a, I want to compare the last three months against what I've done since the change, and then have some confidence as to whether this change is better than what was there before. And that's how I've been doing it so far with AB testing, as you're just describing it. I know what I could do now is I could have, if I wanted to, I could have different subscriptions and in-app purchases that people could buy at different prices. And I could literally pick one or the other to put in front of people and"

Daniel: "Yeah, exactly. So what you described before with the time-based A-B testing, that's kind of how a lot of people do it. They change stuff, and they just look how their charts develop."

Dave: "AB test it, and build up my cohort in real time in that way. Yep. Yep."

Daniel: "And in theory, you could even put that into an A-B test, where you just say, hey, cohort A is just everyone who's using version 1.0, and cohort B is just everyone who's using 1.1. And it's not like comparing day by day."

Dave: "Yep. Mm-hmm."

Daniel: "So it could totally just compare the two values and give you an answer. But of course, with these time-based things, it's always a little bit harder, because you have so many variables that you can't control that are kind"

Dave: "Yes. Yes."

Daniel: "of good. If you run it side by side, you can kind of just ignore a lot of them, because they're going to be washed out, because they're going to be random for both sides, right? Other than that, I think experimenting with pricing is always a little bit tricky, I think,"

Dave: "Mm-hmm. Mm-hmm."

Daniel: "if you are a small app like Topiary is, or GoVJ is, or even like Telemetry Deck still is. Because I'm always afraid. I might be wrong, but I'm always afraid that someone will get annoyed, because it feels like a little bit of nickel and diming people. We did actually update our pricing. We haven't even talked about this. We slightly raised our prices, and just no one complained. It was amazing."

Dave: "Yeah. Yes. Which, yeah, I mean, that's what you want. That's a case of like, well,"

Daniel: "So we now start at 19 euros for almost twice the thickness, though. So yeah, it's a slight price increase. I want to say like 10-ish percent or so. But you get way more stops in between,"

Dave: "I'm clearly providing value even at this higher price, and that's cool. Yep. Mm-hmm. That's cool."

Daniel: "and it is nice. Anyway, yeah. So what I'm always afraid of is if I just actually AB test prices, then I'm kind of like, oh, that people will start, especially with an iOS app, they will start to post the screenshots and everything, because they want to discuss the different payment pages and stuff like that. And if they have different prices, people start talking like, yeah, is he taking us for a ride? So that would be a tiny bit maybe uncomfortable."

Dave: "Mm-hmm. Yep. Yes. Yes. So, I probably wouldn't, this is the thing, right? I would sooner do this as a time-based test rather than a literal AB test because of that, right? Because if, to my mind, I only want one set of prices live at any time, and the only variance I want to give is if I do the thing where you can, I believe you can alter the level the price is at for different"

Daniel: "Oh, yeah. Also a good idea."

Dave: "countries. And I might just differentiate in that way, you know? For example, anybody wanting to"

Daniel: "Oh, yeah, also a good idea."

Dave: "buy my app in Ukraine right now, I would probably want to make that cheaper, right? They're going through a lot, you know? So there's things like that. But other than that, like, I don't want it to be a case of like, oh, if I load it today, it's this. If I load it, you know, within 10 seconds"

Daniel: "What you could do, of course, is just AB test the whole presentation, or whether you start, whether you show the yearly prices first or the monthly prices first or whatever. And yeah, that's completely possible with the whole"

Dave: "of that, it becomes something else or two people side by side. Yes. Yeah. And, you know, AB tests different types of screen as well because I've done that. I've had a couple of different goes of"

Daniel: "framework and everything."

Dave: "this paywall. I don't want to get too hung up on this particular aspect of it. I think everybody gets what I'm on about here, but that's sort of my immediate go-to use is like, okay, have I got my pricing right? And I think that what you said before about locking in on an app version and just using that to differentiate, I think that would be enough for me right now is that I go,"

Daniel: "Yeah, that totally makes sense."

Dave: "okay, I'm going to ship version X and then when version Y ships, I'm going to put the price up or do whatever the change is and then compare the data from those two. That would probably be enough for me to be able to make an informed decision. Certainly after enough time has passed, you know? But the reason I bring it up though is because whenever I've made these changes, my gut feel has always been it needs at least sort of two to four weeks before I really know whether it's good or bad. Because it's a small app and because that noise of some days I get nothing, some days I get several people buy it all at once, you know? It's never steady. So what you're describing about needing the right amount of time to be confident, that intuitively makes sense to me because I've"

Daniel: "Right. That's good that it makes intuitive sense, because while I explained it to you, I realized that, oh, I cannot explain it in depth yet."

Dave: "seen that with adjusting my pricing before. Yeah."

Daniel: "I'm going to get a, I cannot explain it in depth yet. So that means I need to dive deeper into this, because I want to be able to really explain it to people in a way that is very understandable. And especially, like, how do I get proper confidence values? I have in my mind various charts with converging values. Like, if you imagine, like, the chart for 1 over x, that kind of converges towards 0 but never reaches it. And that's kind of my boundary for the confidence that I need. And if I ever go over this, then it means, oh, yeah, this is probably, there's actually probably a difference between the two values. But I'm kind of stumbling over the details right now. So I need to do more research. I need to be more clear in my head so that I can really explain this."

Dave: "Excellent. So that's, I mean, to be honest, Daniel, that is also part of the value that you're providing through all of this, right? Is that you can go down the rabbit hole, figure all of this stuff out, get it, and then distill it down to something that I can use without having to know"

Daniel: "It's going to be a YouTube video and everything. Right. The thing is, like, so many people, I have the whole telemetry query language TQL, right, or TCL."

Dave: "any of that sort of depth, other than just go thumb up, thumb down, you know?"

Daniel: "People are starting to call it TCL, which I love, even though there's also already TCL, which is pronounced TCL, but I digress. Anyway, so it was always kind of intended as, like, an escape hatch for the very few cases where it's, like, where you, like, need something that the UI doesn't provide. But people are just jumping in there. And then it's too complicated for them because they haven't listened to Stanford lectures on data science for the last month or two. And then they ask me questions. And I'm like, oh, this is a really good question. And I don't really have a good answer right now. And I would have to research it for an hour. And they're like, oh, I can't do that. So I'm thinking about it. I know we want to end the episode in a few minutes. But I'm still, so I'm not going to go into that now. But I think at some point in the near future, I need to have a way how people can exchange and discuss telemetry deck query recipes. So either I have to write something myself, or maybe you have a Stack Overflow community or something like that. I'm still trying to find something that's, like, not a lot of work, but it's very helpful in that regard. If you, as a listener, have a suggestion, I'd be very open to hearing it. Oh, please. You're saying discourse, not discord, right? Discourse, also Stack Overflow founder,"

Dave: "Yes. Discourse."

Daniel: "what's his name, coding Jeff something."

Dave: "Yeah. Yeah. And the reason for that for me is that I think they may well be becoming federated at some point. They're looking into that side of stuff, and that becomes quite interesting because that means you can then follow discussions and things potentially through your Mastodon clients and all of that side of stuff. So there's a bit of me looks at discourse and sort of goes, if I wanted something kind of forum shaped right now to sort of trap discussion and provide reference points for people to come back to, I think it could be really quite useful for that."

Daniel: "I'm not sure if I'm 100% convinced yet,"

Dave: "That doesn't say that. Yeah. Yeah. Mm-hmm."

Daniel: "because I'm not sure that I want something forum-like. I'm using this CMS called Sanity. And they have a recipes page, and that's really cool. But that's not very forum-y. It's more like how you have a gallery of GitHub actions, for example. I have to think about it some more, because that would mean coding it by hand, and I'm already coding way too much."

Dave: "You've only got two hands, Daniel."

Daniel: "I only got two hands. Yeah, chat GPT is not super helpful, and yeah."

Dave: "Yeah. Well, us and two hands each need to get on. That sounds really weird now I've said it out loud."

Daniel: "Oh, well."

Dave: "But on that note, I think I'm going to have to start my day and get into everything here. And it's when we record, we're always at opposite ends of the day because of the fact we're opposite ends of the planet. So, yeah, I'm starting my day, you're finishing yours."

Daniel: "I am, yeah. But Dave, it has been amazing, and I'm always having so much fun talking to you. So tell people where people can find you and give you even more stuff to talk about."

Dave: "Cool. So on Mastodon, or the Fediverse, we really need a better name that just incorporates everything. It is the Fediverse, but nobody seems to, people know Mastodon better. But I am at @davidgarywood@social.davidgarywood.com. And then if you want to check out Topiary, easiest way other than searching in the app store and looking for a green icon that's got a pair of scissors at the moment, it will be a tree on the next update. That's nicer. But the easiest way to find it is to go to https://lightbeamapps.com/topiary. And again, as we've been saying through the show, we'll put all these things together in the show notes as links. But Daniel, what about yourself, mate?"

Daniel: "Fantastic. All right, so if you're in a car right now, just toot your horn, that's the easiest way to reach me. If you are in front of a device that's capable of accessing Mastodon,"

Daniel: "you can find me @daniel@social.telemetrydeck.com. You can also go to telemetrydeck.com if you want to hear more about Telemetry Deck. And I think that's all the places. I don't have Blue Sky. I don't have Twitter really anymore. So, yeah. Fantastic. I'm going to say bye."

Dave: "Yeah. Cool. Go for it."