James:              00:07          By Frank, it is almost end of the year. Yes, it is December. And I thought it'd be a good time to look back on the year. Now we're going to definitely look fully back on the year, but before we look fully back on the year, I figure it's been 11 months and I remember that you Frank Krueger about a year ago, but some telemetry in your application. I just want to see how that was going and since you've done that, if you've actually take actionable item, because I think in the years since I've been a mobile developer, I've always put telemetry in my applications, but I never did anything with it. And I know that you love quarries and math and data and if there was somebody that was actually gonna do stuff with the data, it was going to be Mr. Frank Krueger. So I am ready for the Frank Krueger telemetry data insights update of 2019. Hit me.

Frank:              01:02          Woo hoo. Okay, we're going to talk about spreadsheets and columns and statistics and graphs. I'm so excited. Um, yeah. Actually, I pitched this idea, so I am excited because, uh, I was looking back, I think we've talked about a lot of things and I usually say something along the lines of that sounds like a good idea. I wonder what the data would show. I wish I had that data. I should totally add analytics to my apps and it's, uh, eight or nine years to do that. And I finally did it and I finally have data. Uh, you know, data comes in fast and so I could've just looked at it for a day and gave you some feedback. But it's been awhile. It's been nearly a year as he said. So I'm excited for this. I'm excited to tell you the things that I was excited about, but I'm curious what you think too.

James:              01:56          Yeah, I mean I'm, I'm at first kind of want to know what types of telemetry did you put in and like what, before we talk about the outcome, like by putting in this elementary case, people didn't listen to that episode a year and a half ago. What, what were you actually hoping to get out of the data? To be,

Frank:              02:13          you know, yeah. Actually I started till Demetry to catch errors. Um, I was always afraid that the app was crashing. It's just, I think every app developers greatest fear, uh, that like maybe there's a whole user base out there that their app just crashes. You don't know. And so I didn't like not knowing, um, and for some reason I just wasn't using the Apple system or if crashes were reported there, I couldn't tell how frequent they were and they didn't relate back to C sharp. So I started an adventure using, uh, you can use any of the millions of services out there, but I used app center almost specifically because I think Yahoo or someone at Microsoft was like, Hey, Frank tryout app center. And so I'm like, yeah, let's just go for it and hooked hook that puppy up. Okay. So you wanted to get the crash data on it and yeah, app center.

Frank:              03:09          That's what I've been using for awhile for my analytics, crash reporting, things like that. And what did you find like I mean in general, I'm sorry I that that was a bad lead and I started it because of the errors. Okay, got it. Yeah. Sorry. But during the process I started adding in their events. Uh, so these are just things that you can randomly tag. Every event has a name and then you can throw on a, some metadata, just a dictionary, key value store of strings to throw onto the events. And honestly in the beginning I wasn't sure which events would be important and which ones wouldn't be. Uh, but there was one piece of data that I've always kind of wanted and that was which elements in ice circuit are used the most? So I circuits the circuit simulator. It has resistors, capacitors, those kinds of things.

Frank:              04:03          People add them to circuits. Which ones are the most frequent? Like can you believe? I don't know that I can assume things, but I don't know that. Yeah. Cause as an electrical engineer yourself, you're going to have the assumptions that everybody is exactly the same as you obviously. And would use the exact same circuits that you would use when you build it or in your test circuits. Um, and, and that's, I think that's how most people build applications. At least I do. So I'll, I'll say maybe not most people, at least that's how I build applications, is I spend a lot of time on the sections of the applications or the pieces of the application that I think I'm going to use the most. However, I know definitely for sure that that's not always the case. So, um, so that's one thing. Okay, so crashes, you also want to see what circuits people were using.

Frank:              04:53          What else from the tracking? So these are two different services, crash and then now we're in analytics, telemetry coming in. What else? There's some big ones, but the next biggest, and maybe even the biggest one at the top though, it's one of the easier ones to get, but just which iOS version are people running? Because I put a lot of effort into supporting very old iOS versions with the app. And I don't know whether that effort is worth it to be quite blunt. And because I don't know if the effort is worth it, I continue to do it. And um, I called myself an engineer, but that doesn't seem like a very engineering thing to say. Uh, or a scientific, you're supposed to measure things and react to numbers and all that. But instead I react to fear and my fear was that people were using iOS version two. I still assume people are using iOS version too. And I wish I could provide updates for them. Yeah. And then at some point you're, you, you are limited to the fact that

James:              05:58          Apple just won't even let you update things at some point and it makes it nearly impossible to, to get all the old versions of X code or have the correct, you know, CIS systems in place. So all right, so that's good too. That's good data to go. I love that type of telemetry because Apple will give you some, and we talked about this, Apple gave, give you some Google give you some, but correlating that data together is important too. Like once you start to say, okay, well maybe I have one user, but that user like that, that user who has an iPad to, gosh darn it that won't update. Um, that user is using my application nonstop because that would, that would be the, the problem here, which is, Oh, it's not that I have five users. Like if I have five users, I'm going to drop that support cause who cares about those five users?

James:              06:46          What you actually really want to know is I have five super users and how do I make them happy still or actually monetize that in some way? Cause I think that's the important part from that telemetry that you can gain, which is Hey, I have maybe 90% of my users on iOS 13 but they're only actually 10% of my usage. Which would be crazy to think about. And you, if you don't have the data, you literally don't know. I mean you can't make that analysis right. In general, which is kind of something to think about. Um, when you build your application, and I think about putting telemetries in, it's not just about here's a bunch of numbers. It's how do you analyze those numbers with, with metrics. It's something we do like all the time at work. I'm always working, we're customer driven, data-driven and figuring out things, you know, there's telemetry everywhere. So

Frank:              07:40          yeah. Which, um, means it's thinks to be in the minority when you say like, Hey, this feature is really affecting me. And then someone comes back and well, statistically you're in the 1%. So good luck with that. Bat can be the frustrating part when there's numbers to back up your frustration and you wish you could sidestep that. But anyway, let's not talk about that. Yeah. If you don't have the numbers, you have to assume the worst. And that means you have to put, I think, a little more effort in than I need to be. Um, honestly, I was going in this, into this data analysis with a bias. You're not supposed to do that, but my bias was, um, iOS nine is becoming frustrating to support and I would love to make the minimum iOS version, iOS 11, but I was Eleven's only a few years old. Like, can I do that? Is that a valid change to make to the app? It would clear up a lot of things in the app. Uh, and an additional one along these lines of which architecture are people using 64 bit versus the other processors. But James, I couldn't answer that one. Unfortunately a app center does not record processor architecture. I really wanted that one though because I would love to drop 32 bit.

James:              09:00          I guess [inaudible] could you have perhaps gathered data of the device and then created an Excel spreadsheet correlating the device to the architecture? Right. Cause you know the architecture.

Frank:              09:14          Yes. You're absolutely right. Uh, and it would be kind of a long spreadsheet. There are a lot of device bottles out there. It turns out, but uh, you're absolutely right. Yes, I could. Um, boy that would be a nice service. Um, might have to do that. Okay. Yeah. So that was the only piece of data that I wasn't able to get out of this. Uh, but I want to go way back to something you said. Um, you said that a bunch of these services provide you some data. I think he used the word kind of casually, but it came to be an actual problem for me because if you go to the app center UI and sort by operating system list, it shows you the top five, five, five to 10, whatever. I forget the number, uh, of those problem is I want the entire list, but top five is not good enough for me. So app center, it's fantastic, especially the free tiers. They give you tons of great stuff. Um, but if you want to do some real data analysis digging, you have to sign up for something a little more expensive that they offer. And this is called, I forget what it's called, but it's dumping all their data over to Azure and then using Azure analytics to dig through the data. Let's call it as your analytics dumping service.

James:              10:37          That's not what they call it. It's called application insights. That's what it's called. It's called, we're dumping all the logs while there was, so there's two, two options here. You can export your data to blob storage or put it into application insights and application insights is that Azure thingy that you talked about and it allows you to query over all of your data in really, really, um, I mean it's like SQL query language. Like you could just use this query. It's a Cory like querying your data, right? Yeah.

Frank:              11:12          It's like a customized version of ms access bill only for Azure. It's kind of hilarious how accessy it is because once all the data's in there, you can write queries not using sequel. James, why don't people just give me sequel? They invented their own query language, so yay. Fortunately, um, the text editor for the query language is pretty good and gives you like IntelliSense and all that. But if you don't know the query language, it's still gonna take you awhile to write your first few queries. And a funny little thing that I ran into was there are, so UIs in Azure and Azure insights and application insights, whatever it's called, that give you query languages. I never knew which one to use. There were just, what's our creating a rapport? Was I doing this? Was I doing a dashboard widget thing or um, eventually I found out the best thing to do is just create dashboard query widgets. They seem to be the best thing, uh, and just create a bunch of dashboards with all the data that you want. And that was a very productive way. I started creating reports, but they're very heavy weight. You have to hike, add them to a subscription and they take up resource groups and I have no idea what any of that means. But if you just create a dashboard, you can start throwing uh, uh, data tables from queries and graphs from queries. I got graphs too.

James:              12:44          And everybody loves graphs. That's right. You can just see a graph it up. And um, what's, what I love about this as well, because now we're in the ops part of it. Like this is like if you think of dev ops, like you're now in the, the animal, you know,

Frank:              12:57          you're [inaudible]

James:              12:58          the ops part is the part of taking that data that you've been gathering where your users have been giving you and then trying to make educated analysis of it. And some of that is sheer numbers like you're talking about. Some of it is dashboards. I mean that you could create, and I do like that application insights lets you do a bunch of stuff from it. You can create a rest API, you can bring it into visual studio, there's power BI dashboards, there's X sporting from it. Like you get like what's funny is you can like import the data into Apple ache application insights and then export your, your query data out in another fashion. So it's kind of like, you know, funnels of data going all over the place, which is kind of ridiculous. Yeah.

Frank:              13:41          Think about, you know, have you actually done it, cause I couldn't quite figure out how to export the results of my query. So I would love for you to point that out to me sometimes. So I have to learn. I knew how to create reports and I think you could export a rapport, but I don't, I couldn't figure out like a really lightweight to just export quick queries.

James:              13:59          I'll need to look at it. I mean, I will say that there's a lot in application insights because application insights is used by everything. If you have a, if you have a website that has application insights, like basically for free, you just like say, give me application insights and done, and then it starts tracking stuff like crazy. But, um,

Frank:              14:18          you can, uh, you can definitely tell it's origins are in the web because like every data column has requests URL. And so for an app that doesn't show up, it's not there. So no big one. Um, so actually a lot of their, uh, data tables, you don't end up using the one that has all the good data's called custom events. I would've just called it events. But the con custom events and, um, inside customer events is something called like custom properties. Again, I want to just call the properties. Um, and inside custom properties you can find, um, uh, that little, uh, uh, well the name of the event and that meta data, that dictionary of uh, key value pairs. Yeah. And you can query through all of that using like Dotson tax. So their query language is a little cool because he can dig into data structures like that.

James:              15:14          Yeah, I really do like that about it. I mean, you have to want figure it all out, but then once you don't, I'm not a data person to be honest with you. Every time I need something I'm like, Pierce Paris, can you just run this telemetry quirks? I don't know what I'm doing in life. Um, I used to do this, I used to do this for a bunch of the apps that I used to rub because now what I do is, by the way, if we step back, what I do is I just look at the, the pretty charts and graphs that app center gives me. I'm like, yup, seems good to me. Um, you know, and they give you the last 30, 90 days. Like am looking at handsome informs, right? Which is very minimal. But, um, I should look at a real app. Let me look at like what I have here.

James:              15:49          Okay. So if I have like a meetup manager, that's a real app that I have in the store and that puppy has, um, hundreds of, uh, hundreds of users. And yeah, you get to see just a few top devices and a few O S distributions. You see three specifically by the way, for me at least it's based on the distributions and some session duration. And I'm like, God this is good. Like, you know, people are using it cause for me when I put telemetry in, I basically care about um, like how many users are using it and crashes. But why I wanted to talk about this is because you specifically needed more data, which is the heavier part to think about. So, um, did the question I have first before we get into the analysis that you did and everything, did you actually get enough data from this to answer your questions right? Cause you went in with questions, did you have, uh, did you have assumptions? Like did you do a hypothesis, like scientific method on this at all? Like did you have the questions then the, the like assumptions and then like did you validate it? Like did you go through those steps like the end, did you have enough data along the way to make the analysis?

Frank:              17:07          Oh, that's very scientific. Um, yes and no. No in that, um, I, I didn't have a great hypothesis other than what I said before about my constant worry that a bunch of iPad two users are using the app constantly and no one's using it on modern devices. What I was more thinking about, which I decided to do was my decision criteria. So if 10% of people are using iOS nine, would I be okay with dropping them? So I tried to pick a percent number. Humans are terrible at percents. Um, it's, it's not a great practice. Like you never can pick a great, but I decided

James:              17:50          before I even look at the data, this is a number that I'm going to say I'm, I'm definitely careful with like for you James, if I said it was 0.001% of users, would you be okay with dropping them?

Frank:              18:04          Probably. Probably

James:              18:06          no, no, maybe. I mean, yes, yes. I would say 1% yes. Yes. 1% probably. But it depends on your scale, right? Yeah.

Frank:              18:15          Yeah. I asked a friend this question and uh, they said 20%. Wow. If 20% of people are using it, they would still be a comfortable dropping. Yeah. So I, I picked lower numbers than that. I was organizing more. What was your number? Uh, I, I didn't have an exact one, but let's call it like two to 5% or zero to 5% I guess 5% at the top. But I, I would,

James:              18:43          I want to actually dig in even deeper and look at which devices they're using and what their scenario is. That kind of stuff. Because you mentioned correlations earlier. Uh, everything's recorded at a user level. And so every event has a user attached to it. Every user has a region which 50, 50 is accurate or not, who knows, but sometimes has data. Uh, and a few other, uh, what, what's the most important like device model and region. Like I wanted if people are using iOS nine, I wanted to know why were they, was it all because of one certain device once it all in a certain area, that kind of thing. Um, yeah, so I picked a, a lower number with caveats. Nice. Yeah, that's a good idea. I like that. Okay. Before we found out even more, let's thank our sponsor this week. Our good friends over at st fusion.

James:              19:34          Listen, you know, sing fusion, you love sing fusion. I love saying fuse and I you saying fusion and all my applications in fact meet a manager, use this infusion and I'm about to drop a whole bunch of hotness on Hanselman forms, whistling fusion because they have over 145 controls for any application that you're building. Yes. Xamarin applications, but web with asp.net UWP JavaScript, angular blazer, WinForms WPF react view. They got it. All their controls are stunning charts, graphs, but a whole lot more. I love sometimes the little things that they add, like these little effects, that little ripple effects that are inside there or the fact that they had these great little shimmers like things are loading and they go, Ooh, that's the sound. They don't make that sound. But in my mind them that's what it does. Right? But they have all these crazy big entrails to like PDF viewers and image editors and they have an entire file format framework for Excel, PDF, word and PowerPoint to rewrite and do all that stuff. Listen, you got to get saying fusion, go to sync fusion.com/merge conflict to learn all about their amazing controls. They are actually, um, part of visual studio dev essential. So if you're part of that, you get some seeing fusion and they have a community license, there's literally no reason not to be trying sync fusion. Go to sync fusion.com emerge complex

Frank:              20:48          to learn more. And thanks. Thank you Jen for sponsoring this week's pod. Thank think fusion

Frank:              20:57          still love their, uh, I love their calculator thing. I still think about their calculator thing all the time. I tried to look at it and I couldn't figure it out. So you're going to explain it to me. One of these podcasts beforehand. Um, okay, so, so that was your threat data, data, data, data, data, data. I forgot there was one more thing I wanted, um, languages too because I had just added translations to the app and I wanted to, I was just curious. I knew the rough numbers already. That's how I picked which languages to support, but I wanted to know if it was working and all that stuff. Okay. So rough numbers, James, out of 4,000, um, what is this? Yeah, users, I forget the time period, but 4,000 users, most recent, 4,000 users. What percentage would you guess were using iOS nine mm, I was nine.

Frank:              21:51          Yeah. And actually I should bring these numbers up in front of me. If I turn off the top of my head, it's only like four years old, so I'm going to say 10% no, 10% I think as your educational twos, I think maybe, you know, maybe a little bit. So 10% is my assigned answer. 10% 5% 5%. Hmm. That's a lot. [inaudible] I mean, yeah. Yeah, it's a lot. Uh, the funny thing is it's about the same, uh, going up. So iOS nine, 5% iOS ten five percent. Uh, and what you see there is people are definitely stuck on a few devices because it's all the same kind of devices that show up in those lists when you drill down, which is great. That's what the query is for. So you can do these kinds of correlations. Uh, they're nice and low numbers though. Um, but it was on my edge of indecision, unfortunately.

Frank:              23:00          Um, but, um, I want to get more accurate numbers, but the good news is that means that iOS 11 is clearly in the 90% iOS 11 and above. So there is, the vast majority are on iOS 11. So this becomes the tricky part of what do you do with the data. But at least now I have the data. So the old versions definitely people, they are not common. We're talking about 200 people out of the last 4,000, uh, ended up using it. It's still a decent percentage, right? Because if it was 5% of my users, I'd be like five people, right? Cause all my apps like nobody uses. But if your Snapchat right, and 5% of, I dunno,

James:              23:46          100 million people, I was a lot of people. So, and then I guess the question becomes, it's not that you, you don't have to like drop support, right. What you're doing is you're just not giving them updates. And how does that work in the app store? I mean if you were to update to version five and drop supporting quotes for it, they just wouldn't get version five, but what version forest will be available to them.

Frank:              24:13          Yes, absolutely. Uh, so this is something that gives me kind of a clear conscience cause I was always worried about this exact problem. Uh, so when you upload a new version of your app, Apple asks you a somewhat weird question that like, does this contain very important security fixes that should, uh, deprecate get rid of all previous versions. And as long as you always say no to that, that no, this is not a critical update, then they will leave those older versions on for, uh, older devices and older operating systems. Yeah, thank goodness for that.

James:              24:58          However, if you do have a critical fix and patch an update, then I guess they would go away.

Frank:              25:06          Yeah, I, gosh, I, I'm sure there's a very well written document explaining all this. I think I'm oversimplifying it to be honest. So, uh, we'd have to look into details like that. James, I want to update you on some numbers. I just hit refresh. Okay. So in the last 4,000, I still have no idea what time frame this is. Um, in the last 4,000 users, only a hundred had been using iOS nine 90 using iOS 10 11 using 80. So we could say still roughly 200 people are pre iOS. Oh, I'm sorry. I misspoke. 200 people are uh, pre iOS 11. Ah, too many numbers. I said it was going to be a data episode though. Yup. Yup. So 5% of the users are pre

James:              26:03          uh, pre I was 11 and I think also I just went through this on my phone. I have an iPhone six. Do you know what iPhone six does not get?

Frank:              26:14          iOS six does not get iOS 13. I believe that was on the latest round of not getting at the IO. Uh, six S I believe is still updated.

James:              26:25          That is correct. I was a testing a bug in one of my libraries and I said, I need to, I need to put a device on the latest beta. So I said, okay, let me open update this phone. I opened the phone, I hit update, I'm on iOS 12, and I know there's an update button and I click okay. And I'm like, Oh, it's weird. It's only iOS 12. Dot. Whatever. I said, well, maybe it needs to update to the latest iOS 12 before it updates iOS or teen. I don't know what Apple did, you know, iOS or teen's a hot mess. And then I enrolled it in the 13 beta and it's like, no updates available, no updates available. I go up and this phone is done. So this phone, that's the best thing right, is because now I have a phone, a locked in time that will always be able to test iOS 12 because it will never update it ever again. So it's great.

Frank:              27:12          Um, so I that's exactly the phones I keep for all my testing are the ones that get stuck at these versions. And so I have um, I have a test device that's each at its max version and those are my best test devices because there's not too much point in me testing on two modern devices. So I always go back a few generations. They've been invaluable because uh, currently I do support iOS nine obviously. That's why I have users. And so that means I have to keep iOS nine devices around to test on. That's one of the reasons I want to get rid of it. But another one is, I have a 30 minute build time because supporting all those old architectures is very annoying. You have to do four compilations of the same app and it's a very slow build and I would love to bring that build time down and throw away a bunch of if statements throughout all the code. But it's tough man. A five to 10%. Um, do you move on or not?

James:              28:15          No, probably not. Well I mean it's, the question is for you. I think here's the difference is, well now this comes to the other question is are those users happy with the features that you have, right? They've already paid you for the product, so they've already paid you for what they have currently on the device and are they happy with that product, um, in its current form? Or are there bugs that will make them not get it or less likely to upgrade if you do a new version or do something else or introduce an in app purchase. Right. The, the difference really ends up becoming like, are those people going to, you know, recommend that to other people, but are your, are, are there going to be new people? Like if you drop support, are there going to be new people that basically would benefit from the new features that you're going to add to the application?

James:              29:13          Because your application, you're in a very lucky state and you know, is because it's, it's an app that you download once you pay for it once and then you're done. Unlike Snapchat, which is if you stopped supporting that and you change your backend, then now that application can't update anymore. Right? So, um, that would be problematic in general, um, to, to think about, unless you keep an old service around, then you're managing to backend. So that becomes a problem. So I think for you the question becomes, okay, these people are using it, but like is it, is the application healthy on those devices? And you know, are they going to give me one star review because they're not happy and I'm not going to update them anymore? That's kind of the question. Again, ask yourself, I guess.

Frank:              29:57          Yeah, yeah. Well I would never abandon a group an abandoned such a strong word, but abandoned a group without making sure that the app was super solid. The good news is it's a 10 year old app. The core engine is about as solid as any code base is ever going to get given its feature sets. There are architectural bugs you could argue against, like it can't support certain things, but that's more about limitations of the engine and not a bug per se. Uh, but the kind of feedback I get from users now, because it is an in that way, so solid is always just they want more elements, more circuit elements, you know, so people always just want more features and that's, that's common for out the existence of an app. You're just going to be adding that stuff. So when we talked about, it's nice that I can upload a new version saying requires iOS 11 but it's not going to prevent anyone from using the old version.

Frank:              30:56          The problem is what you were saying about new users. Am I decreasing my market? Are there a bunch of iOS 10 users out there because a certain iPad is used and that iPad is used in a school somewhere, then I would feel a lot more achy of how dropping it just so I can feel better about my build times, you know, as long as the build tools support it and it's within reason I think I'm coming down on your side. But gosh, I can't wait to like wait a year and check if I can drop iOS nine yet.

James:              31:35          Yeah, because if so that's the thing is, so if that number had come back at zero, I mean, would you just immediately dropped it and said, bingo, Bango done. The telemetry tells me it's done. Not good to go, or what'd you've been like, you know what, I actually needed to track this over X time to get it back.

Frank:              31:52          Yeah. You know, um, I think the engineer in me says this number, this 5% is low enough that you can move on. Absolutely. Like, um, Microsoft I'm sure has a much higher number for deciding which devices to support. An Apple has a much higher number for which devices to support, but at the same time, it's not that big of a burden to me. It's an annoyance. It frustrates me. But the truth is I've been dealing with it for 10 years. I know how to deal with it. It's fine. Um, gosh, I, I do wish the problem is if a, if I had only ever showed myself percents, I would've been fine. I probably would have dropped it. But instead I'm showing myself numbers and those numbers represent people. So under the iOS nine column, there's a number here, one to one that is 121 people on this planet earth that I'm going to say, sorry, you're never going to get an update ever again. It's not the worst thing, the apps in a good solid place, but, um, it, it, it does change things. Let me think of it. It's people not numbers know.

James:              33:06          All right, so we know the number of users. Now. What else have you gleamed from this data and was anything actionable that, like was he, were you surprised by anything else? Like, as you were sifting through this data that you put into the application or maybe removed from the application?

Frank:              33:21          Well, I think my biggest surprise is if, um, I was looking through, uh, that other number that I wanted, which, um, elements are people actually using. What do they add to the circuit? Yes. That's curious. And part of me was 100% justified. I figured people are using this for very basic circuits and Oh my God, yes, they are resistors, capacitors, diodes, LEDs. Uh, they're, they're definitely up on that list. One thing that I was really proud to see up on that list was the Arduino component because I had put a lot of time into it. And so this list isn't just, um, Oh, I'm curious for the sake of curiosity, I almost see this as marketing and promotion. Like, do people notice that this thing supports Ardwino and it's awesome and are they using it? And if they're not, then maybe I'll do some kind of pop up or something and be like, Hey, go use an Ardwino.

Frank:              34:22          It's amazing. Uh, things like that. So I was happy to see the, what you would guess all the simple components are at the top of the list. But I was happy to also see that, uh, the more advanced ones, I don't have a, you know, I, I ordered by count, but they don't give me like a, a ranking like number one, number two, number three. So I'd have to count, but I'm going to say Ardwino is around, well, we have to talk about this. I haven't the second subject, but are we knows around number 10. So that's pretty awesome.

James:              34:54          Oh, it's pretty good. Yeah. I'm really curious about like, I mean for me, maybe you didn't do it yet, but maybe this will encourage you to do something else. I was like, now that you have this data and there's, there's certain, you know, things that you're noticing maybe aren't used as much or there's a lot of things that are being used, like hat, have you thought about making changes to your application to try to adjust those numbers? Like AB test it? You know what I mean?

Frank:              35:23          Oh, AB testing I'm a little bit afraid of, um, because I prefer to do stable updates over long periods. AB testing means that you have to do quick updates over short periods. And I don't want to play with my users if I'm not willing to make quick adjustments and follow through with them. One thing that I was hoping to capture but I didn't quite capture was flow a flow event flows. Like what, what operations do people normally perform? They do this then that, then this and that and app insights has, well, I'm going to start with, it's an amazing tool that ostensively does this. So for every event it'll show you a graph of all the events that proceeded it and, and, and to see posts seated it and to seated it, whatever came after it. And, um, the problem is it's quite a dense graph with a billion lines because in an app, especially an editor like this, people do a million different operations and they do them in a million different orders. So it's basically every event is followed by every other event. Not too useful. Yeah. Yeah.

James:              36:36          I found, um, even when I do Google analytics and they try to map like where people have gone through a website or something like that or I've done that in my different apps. I was like, yeah, people are just like going in and like, you know what they do after they check in one user, they check in another user. You know what I mean? Like, like after they click that's like turns out and I was like, you know, cause like when I read an email, do you know what I do? I read the next email and then I read the next email and then I probably archive an email and then you know what I do after that I archive the next email. You know what I mean? Like it's like there's only so many things that I could have done. And then is that data even useful?

James:              37:13          Right. Is it useful that I swipe to delete, swipe to delete and then read an email? Right. That to me, what's more important? It's like, if I'm in Gmail, is there a reason that I long pressed, you know, to, to make it selectable is like, do I prefer to long press to do that or swipe away? Right. And that data is interesting in general. Like how often am I swiping away or doing this? And do people know that you can swipe away? So should I make it a little pop up? Like, Hey, did you know that you could do this? And a good task could be like, for you, like how do you make it so your Arduino thing is actually fifth or fourth, like do people know about it and be like, Hey, did you know that you can do our Duino and let you do a little popup in your app?

James:              37:59          And then people like, Oh, I can do this. And then you could, um, you know, you could even have little questions in there like, did you know this was here? Yes or no? And then you could get that data back. And if they didn't like, this would be cool. Right? Like, did you, like, did you know that you can use Arduino? Yes, no. And if they say yes, like you're like, cool, I don't do anything right. But if you say no and then there's a pop up, it's like, let me tell you about this feature, right? And then you know, you could use that telemetry to say, you know what everybody said yes. So I'm going to assume that people found it discoverable. But if everybody, no, that means that that feature was not discoverable at all. I'm like, that's actual real data that you could be like be using. It would be very insightful. I mean, I'm just telling you this, uh, I'm just thinking of cool ideas for your app all of a sudden. And I know that your app is a marketing team.

Frank:              38:51          James, will you come here and do all this? That sounds absolutely wonderful. And it doesn't even sound that obnoxious. I like that. It's just a yes, no question. Have you, uh, have you, do you know about it? No. Well, you're here. Look at it. Blink and led. Look at it. Blink. It's fun.

James:              39:08          And imagine doing this for like more features going forward. So when you roll out the feature, you could maybe preemptively say, Hey, this user updated my app. Like it has the app in there and then after 30 days, you know, app it, you know, I do it or maybe after whatever. Right. And you could, you could go and do that. And um, and the thing is I would even pop it up for everybody cause if they may be accidentally clicked it, they might have known what it was. So it might be very useful to be like, Oh, this person actually, you know, 20% of the people actually went to the Arduino thing, but then they still answer no. Like the correlation there is like they didn't know what that thing was. Right. They, they may have seen it and clicked on it, but they didn't actually know what it was. So you could sort of map those three things together in a very nice little telemetry query. Man, I'm really like, I should become like a data analyst.

Frank:              39:59          Yeah. I mean, what you want to do is capture intent. What were they trying to accomplish? What steps did they do see? Uh, do a group buy on intent and then look at the variety of steps and see if you can simplify it. Want to be great. If we could do a user studies and UI analysis like that, but we can't, it's still the 1970s. I'm happy to know that resistor is still the number oneL a. But something funny came up in my list here and this is where I did incorrect data recording everyone, there are mistakes you can make. Um, I was very proud that I added translations to my app, but I was so eager that I seem to have translated a lot of the event data that I sent up to the analytics. Yeah. And so for ELA, every element I have like Chinese, uh, the Germans aren't top ranked in the list. That might be Japanese. It's hard to tell, but there's pretty much every language on the top of this list. Uh, so everyone out there don't be like, Frank, you're going to record data. Make sure you normalize it to some language. For me, I'll probably pick English just cause I know it. Um, so that was a small mistake, but a fun, uh, outcome of that is that I see the Chinese translation is very popular and it's coming in at the top of the list here.

James:              41:32          That's cool. I, that's actually a really

Frank:              41:34          great, um, I wish that you had the telemetry beforehand to be, to say, I mean, what would be very fascinating, right, is if you had the telemetry before putting in the translations and said, you know what, I'm seeing a lot of people from, um, Japan install the app, but they only use it for X amount of time. And then afterwards, Hey, I put in the translations and now I see it go up or down because then you could say, was that investment necessary? And usually the answer for translations is always yes, but you know, are they correct? Like did it, you know, was, was there other things that you could do in the IDE for those users and making that correlation? Again, you have to, you kind of have to have these [inaudible] we could just sit here all day and come up with different, I could just come up with different ideas for you to research.

Frank:              42:28          Right. But then the question becomes is that actionable? Like what, what are they trying to, what is their intent and then what are the actions that you could actually take over it, right. Because, um, cause you can just come up with thoughts all day basically about telemetry. Yeah. Um, so what you quickly learn is there is definitely an art, maybe even a science to picking which events you actually log. Uh, when I was first starting out, I was just high on the excitement of look at, look at this data flow of events and look at all this data I'm capturing. Uh, but now I see a lot of that was useless. Uh, for example, um, I log every time someone deletes something, I log what the type of a thing is a, I don't care how often people delete things. It's not a part of like a complicated process.

Frank:              43:21          You're editing as surrogate people delete things all the time. I'm just wasting space recording that. Like that. There's no data to be gained from that. No. You know, no actionable items to be gained from that. Um, boy, what else? Every time they resize an element, I log that. That was stupid. So stupid. I gotta take that log line out. Uh, there's another one that's a little interesting though. Um, and this, this is one that I think is actionable, but I haven't actually dug in and done the analysis. Uh, every element has properties and you can go change properties. I'd be curious to see which properties people change the most. Not because, not because I'm going to like take some away or remove them. They're lightweight, easy to support. It doesn't really matter to me. I want to know if my defaults are good. Like is everyone changing a default value away? Because my default value is terrible. So I think that could be a nice actionable item to get out of this. Yeah. Yeah.

James:              44:22          That's good. It's cause you're like, you're, you're actually learning from your telemetry, right? You're like this telemetry. No. Good. However, this other could be really good. And then you sort of have to adjust off of that and like, you know, you then you make a new, uh, uh, you know, hypothesis off of it. Like, which is that, I don't think that my, and suddenly there they give me the opposite. They don't have to be always positive. You can say, Oh, I don't think that people are not, I don't think that my defaults are correct. Right. Or I think that people are changing my defaults like you're saying. And then, um, yeah, go from there. That'd be actually very insightful. Yeah.

Frank:              44:55          Mmm Hmm. Yep. Yep. So it sounds like we both, well, we're nerds. We like data. It's fun. Especially once you learn the query language, it becomes a lot more fun when you don't know the query language. It's quite annoying. But um, yeah, I can't wait to dig in more. And now that I've done one kind of deep analysis, I'm going to go through, check every one of my events, think through if there's data here that I'm actually interested in because also like why collect data if it's not useful? That's just a waste. It's, yeah. I don't, sorry, I just wanted to throw in one nice thing is none of this is personally identifiable information. I can't get any user information. Everyone's a good, but they are good words and so you get to know a few of the GUI [inaudible]. Um, but like the less data the better. I just want, I want to collect data, make a decision and work from there.

James:              45:50          Yes, it's very true. Obviously eight four DB three D one D hyphen four Oh three five hyphen four Oh nine nine IFE and eight five three five hyphen D four ACA, five six five six E three nine really likes default transistors.

Frank:              46:06          There is, there is a [inaudible] that comes up at the top of almost every query that I do my ear a champ. You're just covering all the bases.

James:              46:15          The power user power user. I like that and I will say that. Yeah, you, you, I think your core right there, right there in that analysis of you should feel very comfortable removing telemetry if it's not important. I've done that in a lot of my applications. I got to the point in, um, and one of my apps or I just, I removed everything besides the information about like the devices said I don't need to track anything. Nothing that I put in and tracked was useful at all. Right. And if I need to go back and maybe do some of this AB testing for features, yes. But what I realized is that this app, this application that I have does not PR, it's not providing any,

Frank:              46:54          you know, yeah. It's real useful data. Now that could mean that maybe I was just tracking the wrong data, but you know, that can be for sure. For sure. There's a whole science here. I, I here. Um, but yeah, I honestly, I think I'm going to reduce this down to two events. Uh, they're the ones that I get the most data from. Everything else is just as you would expect, kind of just more noise than signal, I guess, as they say.

James:              47:19          Yeah. Because putting in, putting in telemetry for the sake of putting in the telemetry, putting in telemetry for the sake of putting in telemetry usually does not good results. Putting in very specific rational reasons of telemetry is what is important, right? Or reporting telemetry when stuff goes wrong to help you diagnose why it went wrong, right? So if you track an exception, you can actually pass additional data, like basically events to it. So what you could do, by the way, is you could have a bunch of like information and state, right? Because now you're just like sending a bunch of events. But what if you had, you know, your own mini in-app tracking? Like if you're doing it to log, but it's just in memory, right? And then some crash happens and then when that crash happens, like here's the list of events that led up to that crash, right? But you only need it for that crash because I don't need it for every single person. So there's a lot of different ways of using those telemetry. Like, you know, telemetry that we said to diagnose stuff is helpful to, to glean information. But really you can use telemetry when things go wrong in your application to make it better. Any that's really valuable data.

Frank:              48:33          Yeah. Uh, we went this whole episode without talking about errors. Maybe we'll talk about errors another time, but that what you just described changed how I handle errors and all my apps. So I'm a, I'm a classic try catchall log error person. I don't like my crashing apps. I'd rather just try to move on with life, you know, but log it, log it and upload it to a server so I can find out that it's happening. But, um, it turns out like a stack trace isn't always enough, especially in the async world. Stacks can easily get lost if you're, I'm bouncing around between managed and native stacks can get lost very easily, especially if you're doing async managed and native. And so every time I call log dot log, this error, I pass in a context string or data, I have both overloads but usually just a string. And I just in plain English say, this is what I was trying to do at this point. Just a nice simple sentence. You know, I don't overdo it, but something that I can easily, um, search for in the code. And that kind of explains everything. Right. And app center itself right in your face. It's not hidden in a stack. I don't have to go dig through something. It says people were trying to add an Arduino elements, something crashed.

James:              50:05          Yeah. Yeah. That's good. I like that. That's a, that is, that's when data becomes like useful. Right? And all these different sort of ways of using it. So yeah,

Frank:              50:17          fun times. Well thanks for letting me talk about, uh, my nerdy times of learning a new query language and building lots of graphs and dashboards. I could say a lot about

James:              50:28          web UI wise, but eventually you can learn this thing and it becomes powerful in the beginning. It's very frustrating though, you know, I'm glad that, I'm glad that I'm glad that it was a relatively positive outcome. Right. Cause I know like really early on talking with you and probably even on the podcast, if we go back 170 episodes, you were very anti four years ago, Frank, you were very like anti anything. You will not even, like you said, track crashes or do anything in your application. So it's um, is peculiar, not peculiar, but it's, it's, it's good to sort of see that you give it a try and, and, and you're like, if it works, awesome, if it doesn't work then I can just remove it. I think that's always a nice thing of it too is, is that, but I also believe it's a really nice time because all these backend systems have to be super GDR.

James:              51:18          P GDRP GDPR compliant GDRP like Google remote protocols. I'm GDPR compliant, which means that kind of when you use a lot of these systems, just like you're not doing any of that personal and identifiable information unless you are reporting it yourself and don't do that. Don't report user data ever. And that's silly. So anyways, well Frank, I'm glad that you did it and I'm happy for you and I, um, I hope that you maybe listen to some of these crazy experiments that I would like for you to try and now I think that would be really fun to talk about. One, how you implement it and then if you actually get data and actionable things from it. I'm just saying, I know it's the holidays are coming up, but you know, I just, I hate pitting the elements against each other because the way I see it, each one of those little icons and I circle is like a mini app because each one is kind of a little mini app from like pitting my children against each other.

James:              52:13          I'm like, who's ranked higher in the list? Ah, so sad. Yeah, that is, ah, ah. All right, well before we get outta here, Frank, we had to have an indie classified read, um, from Timo Partel that is um, um, that is name Timo portal and he has an application called working hours. This is for iOS, Android and windows of course built with C-sharp and Xamarin. Uh, Timo is a developer, solo developer from Germany and he made his application for freelancers and also for employees who work from home so they can track his or her hours. It's just a few clicks to get starting. Um, and honestly what happens is you just start tracking your hours. There's powerful features like tags, filters you can explore to Excel cause who doesn't love expels Excel spreadsheets. Um, you can do a whole bunch of other things like, um, again the notifications, widgets, live tiles, NFC tags, geo-fencing and do a whole bunch of really deep integration into it, which is really, really nice.

James:              53:12          They have is cloud sync built right into it and the app is free to use. There is a pro version, it's a onetime purchase, no subscriptions. And you can even try that pro version for seven days. Go down into the show notes and you'll see working hours right there. Or you can go to Timo Partel, T I M O P a R T l.com/working hours. Check out that application and all of his other applications. So thanks Timo for being an awesome solo dev and for that classified ad. That's awesome. I love indie ads. Thank you. All right Frank, I got to go. Um, and people don't know, but this episode is coming out on, uh, I'll be back in the country. Well, I mean I'll be out there also. It was supposed to be a quick one at the beginning. James said, w w let's try to make this a quick one. I'm like, yeah, yeah, totally. Totally. And we're staring at 54 minutes. Great. Yeah, so now I've got to go edit this. I won't have a great night, Frank, and a great weekend, and, uh, and I'll talk to you when I'm back in the country.

Speaker 3:          54:14          How's that sound? Safe travels. Bye, buddy.