mergeconflict266
James: [00:00:00] is back on the path. Welcome 
Frank: [00:00:16] back. Did you miss me? Did you miss me, James? Thank you. 
James: [00:00:18] Every second, every second. Every hour, every single day. I'm just like, where's Frank. How's he doing? He won't respond to me. What's going on? Where's Frank. Oh my gosh. I miss him so much that, I mean, that's my life. I didn't take days off of work because I was so worried about you and just, I was, I was just, you know, um, just like, you know, lonely.
I was just like, where is he? And then he's not in my life. And then, 
Frank: [00:00:39] then you're back. Wow. I wish I had a way to respond to that. Um, don't do that, James. 
James: [00:00:46] Wait, uh, you don't feel that exact same way when I'm apart from you. 
Frank: [00:00:52] We talk once a week. That is more than I communicate with pretty much any other human on the planet.
So congratulations, James. That's how I feel about you. 
James: [00:01:03] You communicated with me for a lot of hours, uh, on the podcast alone. What we've done. Uh, what, what are the 260 years? So let's do some math calc, can you open a Calca for me? Uh, a cow, 
Frank: [00:01:19] a cow cache. Yeah. Sure. What, what, what are we calculating? Uh, so we have, 
James: [00:01:23] um, 265 podcasts.
average length. 38 minutes. Yeah. Okay. How many days is that? 
Frank: [00:01:35] How many days is that? Yeah. Okay. Uh, days. So I'm doing 265 podcast, times length and days. Yeah. Is, uh, well, Hmm, interesting. Hmm. Seven days of continuous discourse. 
James: [00:01:54] And I loved every moment of it. 
Frank: [00:01:58] That was a lot of clickety clack, any typing there.
That was 
James: [00:02:00] good. You did it quick. I'm a  with you. I'm appreciative. Oh, funny here. You know, fireside. Let me just tell you a really funny thing right now. So let me tell you really quick average duration, actually 44 minutes. How about it? Ooh. 
Frank: [00:02:14] Okay. Updating the numbers into the sheet. We go that is now 8.1 days.
Hope continuous discussion discord. 
James: [00:02:26] And that's good. And you, we not only use one of your apps Calco but we also use the word continuous, which is another one of your applications. Um, yeah, no, I, I. Uh, it was fun. I enjoyed, uh, doing a life in time of Maddie because Maddie's one of my favorite people that I work with.
And I didn't really know a lot of her background, like I said, in the podcast. And, uh, it was, it was really cool too. And I don't, we don't interview people. Right. We, we barely, I mean, we have an upcoming interview, so I lie, but like we bet 265 podcast. Fred we've interviewed like 
Frank: [00:03:00] eight people. Yeah, we, we did, um, a whole burst when we were at build one year.
That was fun, uh, because that was actually in person too. So not only did we interview for the first time, but we did it in person for the first time. That was a real fun trip, but we really haven't done it since then. I'm going to blame myself here because I am really bad at calendars. My hate organizing things.
It's a small miracle. We get together to chit chat every week. So I think, um, Not been good at scheduling, but it is fun to have people on from time to time. Fresh perspectives always look good. 
James: [00:03:37] I do, I do believe from time to time because you know, that's on our podcast. There podcast many podcasts out there, even the ones that I listened to there, there's very minimal podcasts that are the same hosts every single week.
And we strive when we created this podcast to do this. And I don't know listeners you're out there. There's, there's at least one of you, uh, right into the, to let us know. Do you want more interview stuff? Did you enjoy that? You know, because you know, in five years, Frank has never missed a podcast and I basically.
I, I didn't even give him a chance to record the pod. Cause I said, sorry, I already recorded the podcast. Have fun in New York. Bye. 
Frank: [00:04:12] I got bumped. I got bumped. Well, it's not true. I didn't show up for the recording. And then I got bumped. So it's flacid Hollywood. You know, the director was being very impatient, I guess your producer in this case, uh, impatient producers, 
James: [00:04:26] editor or producer.
Co-host. 
Frank: [00:04:30] Now it worked out really well for me because I do vacation very well. I check out completely. I'm not checking on the internet. I'm not doing anything. So that worked out just fine. 
James: [00:04:43] Yeah. It's uh, it was good. And that's why I thought about it too. I was like, you know, obviously there's a reason Frank didn't show up.
He wants to completely disconnect. I'm going to give him the complete disconnect he, he needs in this life for this one vacation and never again, Frank. 
Frank: [00:04:58] No, I'm sorry. I'll try really hard. I do kind of love our remote recordings. It's always fun to be like, Hey, it's 3:00 AM over here and we're coming live from random place on earth, but we haven't been able to do that for the last few years.
I look forward to doing it in the future though. 
James: [00:05:15] One day Frank, one day. Well, this week, Frank, I want to talk about something very special because I recently had a feature request coming to map. Do you have ever feature requests, Frank that come into the application? 
Frank: [00:05:26] Yes, I do. James. I have a, I organize them with user voice and emails, and then I create issues.
If enough people mention it, that's kind of my system. How do you 
James: [00:05:38] feel about feature. 
Frank: [00:05:41] I mean it's hard. Um, okay. How do I feel about them? I wish I could do every feature requests. That's actually how I feel. I wish I could do that, but I know I can't, um, I make lists of features and I'm like, I'll knock out four of these in a day and it ends up taking two days for each feature.
So I'm bad at estimating and it feels like. Baker and older, a piece of software gets even the small features become bigger. You really learn that when you're developing an app from scratch, like you can add a million features as you're developing the app, but once it's a little bit established and someone asks for something a little bit random features can get harder and prioritize, even though I want them all, uh, uh, lots, uh, you just can't get to them.
I, I. Honestly, 
James: [00:06:32] I wish that I could do every feature where there's, there's a, there's a balance, right? There's not only the technical debt that you've accumulated that, Hey, this random feature requests that came in from somewhere, someone that doesn't know anything about the code base now, why should they write, does it make any sense?
They're like, oh, this would be cool. How it could actually impact your code. Then the other flip side of it. Is that, um, it's only time invested, but it takes away from other feature requests that you may want to have in your application. And like is your code, you architected in a way for your use case and you're like, you made it how you wanted the app to be.
And there were of course iterations where you're adding features. So some of the feature requests, because you didn't architect the app that way it's sort of as if the feature requests. Is that much harder to implement into your application and the other flip side that there's the other one that I actually wanted to get to is is that a feature request may not even vibe with your vision for the application?
Well, that's a great idea, but like I'm an, I'm a never do that because like, that's not what I want in my app ever type of thing. Do you have those kinds of on. 
Frank: [00:07:44] Sure. Um, I, I try not to ever push them that far out of the box. So I say, I'll never do it. I tend to think of it as like, oh, maybe I circuit version 10.
Maybe I could do it there or something like that. Um, it's hard because. So you, you get, I don't know, I don't want to, I'm having a hard time categorizing the kind of feature requests you get, but there are the easy ones that fit into your architecture and those are wonderful. Unfortunately, you can get a million of them.
So in the case of like my apps, people want, um, a million functions for Coca. They want a million elements for ice circuit, but each one of those, you know, can take a day or something. But it's totally within the architecture. It's just work effort, putting it in, testing it, all that kind of stuff. Then there's the terrible ones.
Um, the architecture ones aren't even so bad now, purpose of the app ones, you know, I always get these little like web services and things like people are like, why don't you talk to this service and do that kind of thing. That's where I think I I've felt the most pushback where I'm like, oh, I don't really want the app to work that way.
Uh, I have one where I want everything to be done locally. I never require a network for anything. And that's honestly a really restraining condition I put on things and people will put in a request. Why doesn't it do this? I'm like that because I can't do that locally. That's why, and I feel bad for saying.
James: [00:09:20] There's a few that I get. So specifically my cadence, which is my bike, cycling Bluetooth little device reading thing. Right. I get a lot of feature requests, um, that. Are for other sensors, so, oh, can, can you also do heart rate and can you also do this sensor? And it's like, not fam, like that's just like out of the 
Frank: [00:09:42] scope.
That's tough because you are the single use app or single purpose apps, not single use my apologies. Um, that's so much harder. Right. Yeah. That's, that's keeping a real focus, but I can easily see that, like, if they really like your app and they don't like the apps for showing heart rate, it would be totally cool if your app showed heart rate.
So I see where people are coming from and that's where I say, I never say no. It's just more like, uh, I don't know if that's a whole separate app or should I make that a separate app? You know, that kind of stuff. Yeah. And what I've learned over time is I don't like having a million it's for sale. I'd rather have a few big ones for sale.
So that's why they tend to become the kitchen sink of features because I'm like, oh, I'd rather put it into this app. And rather than start a whole nother app for that fee. 
James: [00:10:35] Yeah, I had a few good ones recently. I'm implementing them right now. I call it features twenty, twenty one because I haven't updated the app really, since this came out and I've fixed a bunch of bugs and things like that, and it's been solid for months and I go, okay, like, you know, I was looking at the reviews and these, these are reviews as feedback is another way that I take them in a lot of times, people email me cause I have an email button in the app, but I am getting, you know, ideas.
Um, from, from reviews, like someone said, oh, it'd be great if I could resume a resume, a ride, because let's say you get a phone call and then it pauses it or whatnot. I'm like, that's good. My app is not architected that way at all. So as you all my mathematics and I have to, okay. Cause everything's date time-based and calculating numbers.
And it's like, okay, well, like I know that that happened with my stream timer too. Like, oh man, it'd be great if I could. Oh man. Like, I never thought about pausing. I want to pause, you know? Uh, and, and it's funny, all of my apps, everybody always wants to pause, like, that's the number one feature. It was like, pause this.
I'm like, like, ah, dang it. Like, okay, fine. So I did do that, but the bigger one than I got in recently and this, so here's when a feature requests like touches your heart a little bit. Okay. And you almost feel bad if you don't do the feature requests. Right in and they just had surgery. Um, I don't know what kind of surgery it was specifically, but they were doing.
Um, an exercise routine had involved, uh, cycling on their indoor cycle and what they needed to do because they needed to monitor, um, like their, their heart, their VO VO to like the oxygen levels. And they also needed to monitor, um, sort of the resistance and the cadence. They needed all of this. And, uh, they need to keep a journal of it so they can report back to their doctors.
And like, you know, what I really need is all my other apps that I use for heart rate and VOC. They, they give me a scale. He's like, I need to cross compare my  to the cadence and the timelines. Uh, so I can, you know, get healthier and you're like, oh, ah, man, I can now do this. Right. Um, and, uh, and this was cool because.
Oh man. Like all my data is just meant to be wiped away. And I was like, once you're sort of saving data, you're just like, oh God, I gotta make a database. I gotta do something. And, uh, so I said, I said, what is your, what is your like, minimal. Viable. Like, what's the least, what's the least thing that I 
Frank: [00:13:03] leave to work.
I can do. 
James: [00:13:05] I, I, that was one, that's a little of my emails that what is the least work that I can do for you? And what a, what a sweetheart. He said, he said, you know, it would be, it would be fine. Um, and what I really need is just a historical data of, of my average is, and my maximum and how Y how long the ride.
Okay. But you know, the other applications that I'm tracking that are really nice and add a lot of value to get me back onto the rotor recovery is, is, you know, a graph over time and, and, uh, XYZ rallies. That's what I'm like. Ah, it's like a new grass now. Like, come on. I got it. I mean, sat out to our spot, not sponsor this week infusion, but it's like, I've done graphs and charts when I was like, all right, we just got from like, entering like one.
You know, data point into, I got a store, maybe a thousand or 3000, you know, data points for every ride. Plotting that out. I was like, ah, do I really have to like, oh, okay, fine. I'll do it. And then I'll go. I waited two weeks. I mean, because I was bad, right. I to go home and I had to do a bunch of stuff. I had to go do some traveling.
And, um, I was like, all right, I'm gonna, I'm going to do that. So, like I started by not doing the graphs and charts in the historical data. I'm like, I want to do the zoom feature first. Like that sounds so I did that. And then there's a few other quick wins. I was like, okay, I gotta do this. I gotta do this thing.
So I texted you Frank, and you, you should remember this. And I said, Hey, I need to store like a few thousand data points. And they're just an integer. Do I do, I need to create like a thousand rows of data into my SQL database. And, um, and I believe that, that you said, no, I believe you said, I think you should, but not necessarily.
Frank: [00:14:52] Okay. So in this case, my answer is no, no, uh, this is. Data sure. And databases store data. Sure. But don't take that analogy too far. Uh, that timeline of data points, you can think of that as just one datum. It is, although it is a collection of data and that fits into a column very nicely as a little Jason array or something.
That if you ever need to do processing on it, you can extract it and decode it and then process it that way. I don't see any reason to overload the database by having a row for every data point. You're just, databases are good for relational data and queries. If you're not doing relations and you're not querying and, and beyond, you know, select star, that's not a query.
Um, If you're not doing those things, then just Jason and coated and then throw it wherever it needs to go. Or, you know, do the right column type. That kind of stuff that I don't know if that that's a philosophy or something someone might argue, well, you might want to add a comment to every data point or something.
No, you won't. You won't. And if you do, there's other ways to encode that comment in a relationship so that you don't have to worry about it. So everyone, if you're storing historical data, You know, serialize it away and throw it in a column. 
James: [00:16:17] This was very controversial by the way I live coded this. Yeah.
Awesome. I love it. Let me tell you that pretty much everybody I've talked to it told me that I'm doing it. Pretty much. 
Frank: [00:16:31] It's hilarious. Um, but I see people spend so much time creating very complicated database schemas, and then you end up with complicated queries and then you have complicated insertions.
Don't create busy work if you're not doing relations and you're not queering things out of it, just encode it and move on with your life 
James: [00:16:54] coding going on with it. So I did, I did a few trials, so I talked to the stream was. Uh, fascinating. In fact, one of the person in my stream definitely related to me, they have an application in which they record, like all the historical data points of stocks.
And they're like, we have millions upon millions of rows of like relational data, this and that and dates. And this, I was like, oh my goodness. You know, they're probably querying for specific dates and times of specific data and they're getting chunks back. They're not yet. Like here's, you know, 20,000 data points all at once and you need to load in and create 20,000 of them every single time.
Maybe they are. Um, I decided silly cause I have one here's the train of thought I was talking to my wife because she's a database expert. Like that's her, that's her main thing is databases and web APIs. And she goes. Uh, so you could do that. That's what we're on a big hike. Of course. I'm talking about database.
Yeah. You could do, you could do that. I mean, you could, you could do that. And she was like, but you know, she's like, well, your, this is your app, single, single developer, your feature, you know, no one else is writing to the database. Right. She's like, because what if someone puts a string? Like what does someone inject something in the database?
So like, that's not going to happen to your app. So, um, should you do that? But, but you're, you're going in understanding that this is not a scalable solution. I go, oh, I'm going in explicitly understanding that this is not a scalable solution. Like as far as, I mean, as far as if I want to add more data, besides here's a list of integers, like let's say that every data point had a date timestamp, and then it also had the VO data.
It has this other stuff, right. It's like has a bunch of data on it. That Jason blob would get very big. 
Frank: [00:18:44] Sure. But so would a bunch of columns and a table sizes size, you can't avoid the size data's data, no matter, I mean, all these encodings are roughly equivalent to each other. We don't have any magical compression and coatings.
So it's just, where are you putting the data? All these arguments are valid. My whole thing is even if you have all that metadata for every single point. Are you wearing it and chances are, you're not what you're acquiring is a time chunk of it. And you're spreading it out. Another way to think of this is just store everything as CSV files.
Yeah. Right. Um, and that would be the kind of classical way to do it without a database. Every little one of these tracks is just a CSV file. Multiple columns, uh, time progress. 
James: [00:19:34] Someone in my chat, I kid you not, uh, maybe it was Laughlin. He said that he had a different app where like the database was more temporary, so it didn't need to persist even after someone uninstalled or whatever.
And what he said is he said he literally stored the, this is like, you know, Jason blobs basically, or CSV blob or the data file out to disk. And then what he did is he just wrote the location of the file into the data. 
Frank: [00:19:59] Yes, I love it. Except I would just put the CSV file into the database. The entire text SQL Lite is really good.
Yeah. Or I'm sorry, I'm assuming you're using SQL Lite, but every database is really good at storing huge amounts of text. It's they're great at it. So just throw that CSV right into a column. That's my argument. There is no reason to pollute your database with a billion rows, because what will happen is so that CSV has five columns in it, right?
So the natural. Put that into a database is to create a table with five columns, and then you're going to have a row for every row that you would have in that thing. Makes sense. Right? Life is dandy. What happens now, when you want that second one? Oh, now you have to put a session ID inside of every row.
And that session ID has to relate to a user ID and all this kind of stuff. So those rows are going to inflate anyways. So might as well just encode it and move. 
James: [00:20:56] Well, the cool part too is. You know, by the time Heather and I were talking about this for like 30 or 40 minutes of our hike, and we were talking about this.
Cause, you know, I, I understand the, the, the bigger case and this and that. And by the end of it, I wonder over on this, on this, she said, and this is what she said. She said like, you can always change it later. And then she said, and you wouldn't even have to migrate it because what you could do. Is, you could first see if the data exists and then just show the old style chart.
And then if it has, if it doesn't have the data, then query the new stuff. So you wouldn't even have to migrate the data. Right? Because there would be nothing to migrate. She goes, I think that this is definitely the way to go. She's like, I think that this sets at the end of the day, the way to go. And then I said, well, you know, I have to make a decision.
And Frank, isn't a here because he's off for holiday off in New York. And I said, well, I want to. I want to serialize this data to be small. And you just said it by the way, which is there is no magical. This is what I wanted. Frank. I wanted to have a magical data shrinker like shrimp on my data. I wanted, I wanted like an array of like a thousand lists to be like, yeah, Two characters, like how come that does scanning?
Like just, it's just shrinking down. I want it to be so small Frank and I was like, I just want my data to shrink the magic, that data doesn't shrink. I dunno if you know that really? 
Frank: [00:22:21] No, no. There's this whole field of this thing called computer science. They'd started this thing called compression. You can actually do a pretty cheap compression in.net.
So if your Jason blob is huge, absolutely run it through a zip stream. It's built right into a system dot I O dot compression, something like that. Uh, there are Jesus' streams and deflate streams that you can use. So you can totally do that. But, uh, if you're just throwing it into a database, let the database deal with all that kind of stuff.
There's no reason to, there are a great serialization formats out there. I want to give a shout out to protocol buffers. So if you want to take the time to model your data, it's a declarative language. They'll output a serializer for you with a new. So it's really easy to use all this stuff, and it has very, very efficient and codings for all the data.
So if you actually care about space and size and all that stuff, but I would argue for an unless you're dealing with, I don't know, Hundreds of megabytes tend to make a bytes. Then I would start caring. But up until that point, I wouldn't care. 
James: [00:23:38] Um, I'm doing okay, here's this? And, and that'd be another California, but you don't, you don't have to do it.
It's it's very simple. It wouldn't be a good Calca is I did a trial where I said right now the sensors log about every second. So let's say you have, there you go. Yeah. They log about every second. So make sure that the, the, the, the, the, the one in this case is, is a constant constantly you can change.
Right? So normally they, they, they log every second and they will. You know, uh, four, you usually ride for 30 minutes. Right. So that would be a very simple calculation of one time, 60 times 30. Right. Right. Um, and how many data points is 
that? 
Frank: [00:24:19] Alrighty. I mean, uh, sorry, how long were they running for? 30 minutes.
James: [00:24:22] For 30 minutes. Yeah. Okay. 30 times 60 1800 data points, right? No. 
Frank: [00:24:32] Yeah. Did you yeah. You said per, per second, one per second. 
James: [00:24:37] Yeah. So, so then I said, um, and I charted this, if you actually. Like 1800 data points on a graph. It doesn't actually look very good. I don't know if you know that Frank, just fun fact for you.
Frank: [00:24:52] I'm having to do this a lot because a lot of my apps are time-variant like I do simulations. So I circuit I'm constantly looking at a lot of data points. People will run ice circuit at megahertz where it's trying to collect a million data points every second. It can't usually keep up with that, but it's trying to, and so it's just spilling out huge amounts of data.
So you're up to 1800 data points, let's say on average, it takes about four bytes to store one of those. So you're still only looking at about seven kilobytes of data. It's really not much. 
James: [00:25:28] Yeah. And, um, so I'm, I figured like, well, what's the longest someone would do maybe two hours. So, you know, a hundred, 120 minutes.
Uh, I can't imagine people going longer, but maybe if you're, if you're training for a long time, but most people it's like a 45 minutes, but let's say it's two hours. Right. Um, you know, you're looking at a 7,200 data points, right. And, and that's really not too long, but then I was like, man, 7,200 data points are not gonna look great on a graph.
So I decided to change it. And I said, I'm going to take a sample every five seconds. 
Frank: [00:25:59] Yes. Okay. The problem with doing that is you're going to create something called aliasing. What you're getting into is called you're down, sampling the data. So you're going to skip every five samples and then take a sample, skip every five samples, take a sample, a better way to do that is to average five samples.
Then average five samples than average five samples that's called windowing. And that will give you a much smoother graph. What you're doing is fine, actually, statistically, it's fine. It's just not going to make the graph any prettier. If your goal is to make the graph prettier, I would recommend, um, uh, downsampling with a window.
That's 
James: [00:26:40] going to be, that was my next average moving average. And I figured that'd be even better because now. Instead of it being 7,200 data points is like 1400. If it was like two hours. I mean, it's a lot less data. It's a little bit more calculation up front, but I do think that that's going to be my final strategy, which is averaging over five seconds.
And, and then I feel like that data is very small energy, so blob, and by the way, I tried two things. Frank, I tried both Jason and B Baeza. 
Frank: [00:27:11] Ooh, be on, you know, I'm using the sawn for ice circuit 3d. I started 3d saves its object graph as adjacent graph. Uh, but then, because that can be huge. It zips it on all up.
So that's a fun, um, not zips it, it does it pit, sorry, but then it be sons' it all up. 
James: [00:27:35] Uh, BS. I do want to describe BS on too 
Frank: [00:27:40] attempt to sure. I I'll probably get a few things wrong here, but everyone, I think probably knows what Jason is. So it's, we have our ASCII curly brace and then a quote and then a key and then a quote, and then a colon, those, those curly braces and quotes, they can get a little repetitious.
So there is a binary format called , which represents the. Um, the Jason graph and a binary form. So not readable by ASCII, but obviously very readable by computers, computers, love binary data. And so the neat thing is all your, uh, All your floating point, numbers will get encoded as eight bytes, 64 bit guaranteed, no matter how long the number is, that's how big it is.
That's actually also a downside, uh, James, because for example, the number of. Only takes one bite to encode an ASCII, but requires eight bytes to encode in binary. So it's not necessarily a smaller format. It really is data dependent, but in general, it's faster and all that stuff. That's why I use it too, just because it does make, um, if you were trying to serialize like a biter Ray, it can do that much more efficiently than if you tried to sterilize a byte array inside Jason.
Where it has to encode it as a bunch of ASCII text that makes it has a lot of advantages, uh, but basically binary versus ASCII. That's the difference 
James: [00:29:13] binary versus asking. And I did a little trial and I went on a little five minute spin. I just went out there, did a little five minute spin, which generated.
How many data points in five minutes? 
Frank: [00:29:23] Oh, I'm not even on the thing anymore. 
James: [00:29:24] 300, five times 60. 
Frank: [00:29:27] He's not even paying attention. 
James: [00:29:30] So I did 300 data points before I did the sampling, uh, and the, the, uh, Jason equivalent, because you know, these numbers are either one, two or three digits, most likely two digits and, and a semi con and a, and a, and a comma.
Right. So. They're in there. So if you have 80, 80 comma something, uh, except for the last one, obviously, and you have brackets on the front and the bottom. So it's in there and if it's a hundred, it's going to be four because it's going to be a 100 and then four. And I did this and the Jason. 
Frank: [00:30:02] I'm sorry, I just wanted to insert one little tidbit here.
If you plan on doing this at home, as it sounds like you did James, if you're not doing integers and you are doing floating point numbers, please make sure to encode it into an invariant culture. Yes, because I've seen a lot of people make a mistake here where, um, uh, a French user versus an English. Yeah.
Well, get a different encoding of the data. So just be careful when doing text, um, don't do it yourself, use a Jason library or something, but whatever, keep going, James. Sorry. Had to answer it that, 
James: [00:30:35] by the way, when I worked at Canon, I totally wrote code that totally broke a bunch of users data because they were using it in France and there was commas and not periods and this, and then I learned about in varying cultures and how, how important those are, but rest assured Frank.
Into Ray. That's all it is. It is an answer, right. Because I keep it simple and that's it. Um, that's all you need to know because that's all you see on the screen. You don't see like, oh, I'm going 85.5 cadence. That doesn't make any sense. So we always round, uh, in general. So the Jason was 1000 characters long in the B's on was 3000 characters long.
So I decided to go with Jason all up in this format. 
Frank: [00:31:17] I am so excited. A that I won the argument and B that you got to do serialization of numeric data. Cause I feel like that's all I do in my career. I was doing, um, I needed yet another format for Numerica data and I was trying to, uh, do the numbers.
Format. And that was a rabbit hole. They actually mix ASCII data and binary data. So that's a whole thing. So I am happy that you at least conducted an experiment. That's wonderful. Um, but remind me one more time now, what did you decide? Which one do you 
James: [00:31:53] want? Jason? Uh, we're doing a string, a list of integers that are in Jason form.
Frank: [00:32:02] Perfect. Yes. And it's going to work out great. It's going to work out great. Even if you wanted to query it, you could totally still query it. You know? Did they ever hit 100 RPM? Is that what the unit is? Yes. Uh, so you could, uh, query the database for comma 100 comma and find out if they ever hit one. Huh?
The text 
James: [00:32:24] is searchable. Text is searchable. It's very true. Yeah. And yeah, you could do that say, oh, how many rides were XYZ? Right. And you can do that about, yeah. The here's the, the other fun part too is, so I had to make this chart and graph and, but I also need a list of history, you know what I mean? Yeah.
And, um, And then I thought about it on my, on the ride. And I was like, oh, wait, over time. I've written this bike this year, you know, you know, a hundred and some odd times. And I go, well, if I just say, you know, database dot table rock. Get all, you know, two lists async that is going to get all of the rides and this mega string for a hundred rides.
That seems like not a good idea. Correct. 
Frank: [00:33:17] Correct now, uh, this is where you could have put it into another table and used some relation. Um, but it's fine. Yeah. You, you, you throw it into the table now you're using SQL Lite hyphen net. Is that right, James? . 
James: [00:33:34] I like to call it dash live, live your life library, creator of the library 
Frank: [00:33:41] member.
What is that? I can't remember what I call it. That's the problem. Uh, yeah. So that has one feature that you're referring to, uh, which is usually very convenient, but can really bite you. And that is whenever you, um, ask for all the rows of a table or, you know yeah. Have a query. That's going to return a lot of data.
It's going to give you every column of that table also. So yeah, in this case, I, you know, honestly, 10. K what's 10 K between friends. I think it would still be fine. Honestly. I don't even think it would show up in tests, but you're being a good programmer. And you said why waste the CPU's DMA resources, even though it's a whole separate piece of hardware and it probably isn't even bothering anyone.
You said, I am only going to query a few columns. 
James: [00:34:30] That's very true. I'm going to just do a few columns and actually I'm probably going to get your help on this podcast because right now I have a, uh, have a, uh, it's it's close. I'm going to post it into our Zencaster. You can see it and you can immediately see what I'm doing wrong.
Frank: [00:34:47] Um, yep. Nope. Uh, so you are doing a query ISA async. You are trying to get a bunch of things from a table called ride. That's all fine. What you're running into here is the classic problem with SQL. That is you have to escape identifiers that are also SQL terms. So things like max average, those have to be quoted.
Oh, brilliant. Yeah. Yeah. The list of things that you have to quote and sequel is actually a bit long, and that's why you should use an ORM that does those things for you. And normally SQL Lite would do that for you. But when you're hand writing a query, you have to do it yourself. Now, something that's also awkward.
Do you know what the quoting character is in SQL? 
James: [00:35:40] No, I don't talk 
Frank: [00:35:41] to at all. It's double quotes, which is really annoying to put into C-sharp because then you have to do backslash double quote, backslash, double quote around your identifiers. 
James: [00:35:51] So th this does seem to work for me. Do you think it's because the it's capital, average and capital man.
Or 
Frank: [00:35:57] no, Nope. SQL is doesn't care about that stuff. So if that works for you, you're good. I was anticipating that it wasn't working for you because of average and max. So if it's working great, um, the, the trick with SQL also is there are contextual keywords and non contextual keywords is the non contextual ones you have to watch out for.
Yeah. 
James: [00:36:16] Select from those are keywords. Those are the things that I should change for, for best practice. I should change average and math. 
Frank: [00:36:25] Yeah, I would say like, um, average RPM, max RPM, something, you know, SQL would never have just for convenience. 
James: [00:36:32] UTC. Okay. 
Frank: [00:36:34] Yeah, that's fine. I think there's, there's one that always gets people and I think it's useful.
I think like users not allowed that there's a few common column names people want to use, but they conflict with SQL. And so, 
James: [00:36:47] so here's the here's, here's one of the cool things about what like, oh, data does. So, oh, data. I use this very heavily in my first job, uh, at oh, was saying your job is. He OTA to quarries and Odie to is what they'll let you do is they let you write the LINQ query and then do a select.
And then you could say, you know, a link where, and then select, and then you create your new object base out of the information, uh, from that ride. So for example, you could say, Hey, database query, oh, you know, are, are two lanes. Are you going to say, table.select are, you know, Lambda, new ride, and then you can just fill in four things and then the O data query, would it be smart enough to say, Hey, what this user is actually saying in the old data query is, Hey, only request these four fields, but from my understanding, there's not that in SQL light dash net at least.
Okay. 
Frank: [00:37:42] There is not, but you're embarrassing me now. Um, can I give an excuse for that? 
James: [00:37:49] It seems hard. That's why I seemed like you need a source generator for. 
Frank: [00:37:54] Well, I, I am doing all that stuff. I am doing the hard problem already. Um, it is, it is harder though. It's because I wrote my link support in one week and haven't really touched it ever since.
Uh, there are two, uh, two or three big glaring missing things joins are missing from my link support where you can do, um, yeah, the syntax has joined when you're in link. Um, but if you're doing the functional interface, it's select. It's the command. Yeah. So there's that one. And then select is actually what it's called in link, where you pick out exactly which columns you want.
Obviously the case where you're just pulling out columns, it's really easy that promise select allows you to pull out arbitrary expressions. And that it's that little part that I've gotten myself hung up on. But now that you've mentioned this, um, because you've sent me this, I absolutely do need to get it implemented.
And in there. 
James: [00:38:50] Well, I'm seeing that it doesn't exist. I did make the manual query, which, um, I sent over and it's very good, but see what I do on right after that is I query the database for all of the rides. And, and so I get all I get basically, instead of doing select star, which, you know, here's the fun part.
Yeah. About SQL Lite and databases. And mobile is like, you can, I could have wrote this app and it wouldn't have been just fine to just, you hate table. Give me all the things. Um, but I decided that I'll write the SQL query, even though I'm not a SQL expert. And I Googled the heck out of how to do select certain columns, which it wasn't document.
They've gone, like anywhere, not yours, but you know, I did, I did look at your tests. I was like, I bet Frank has a test for this. And then somewhere, I think you did a in there cause you do a lot of Cory async, which is good. Cause it's like the Ross, the, the we're all SQL that's happening. Um, how does that do this, this, the seagull court.
And it's really not that hard, like that SQL query it's, it's pretty straightforward. You know, it's not, it's not that complicated and I feel okay about it. 
Frank: [00:39:57] I I, yeah, I think it's a language designed for the purpose of storing and retrieving data. It's great. Um, the only problem is it just, I didn't like any of the libraries of how it talks to software, you know, apps.
And so that's always been the disconnect. That's why we call it the, um, Uh, the impedance mismatch between the object world and the relational world, but the language itself sequel, I like it just fine. I have no problem with it. I, I feel a little bit bad because what you're doing is you are acquiring the table and you're getting all the rides out of the table and just the columns that you want.
And then after that, you are doing an order by, with link. Do you see 
James: [00:40:44] that that's a problem that there's my, and there in lies my issue, which is I got real lazy, real quick. 
Frank: [00:40:49] Yeah, it's funny too, because you wrote the C-sharp ride stock order by our.date UTC and the SQL would have actually been shorter than that.
You can just say order by date UTC, it would have been actually less code if you just put it right into your query. So the very end of your query, 
James: [00:41:07] that would have been more efficient too. 
Frank: [00:41:10] Yeah. Um, yeah, sure. You know, from a theoretical computer science standpoint, not really because chances are, you don't have an index on that.
And so the SQL engine would have had to do the soar anyway. So it's really just a question of, could the SQL engine do the sort faster than C-sharp. Link and chances are yes, because it's optimized for that kind of junk, whereas link link is optimized for sure. But it's handling a much bigger general problem.
That's not even true. I'm talking in circles and the SQL probably would have been more efficient though. Yes. 
James: [00:41:47] So I would do select columns. You put all the columns in that you want from ride order box. 
Frank: [00:41:55] Yeah. With a space 
James: [00:41:57] and then date UTC. And I think I actually want it to be descending, right. Because I want to be the newest one first, 
Frank: [00:42:03] then put the word descending.
James: [00:42:06] Is it ordered by DEC? 
Frank: [00:42:09] Oh gosh, you're going to make me look that up. I thought it was the whole word, but now you're making me doubt myself. You can Google that. It's either the whole thing or DSC. Okay. I'll return. I'll I'll try and ORM, you know, so I don't have to remember these things. 
James: [00:42:24] Exactly. Now I will say this below, um, uh, a fun one that I ended up doing is I was having this, all this other issues in my app.
And I did something really, really silly is I had in my, in my, in my app, there's all this, there's all this magical code behind this, like doing the serialization destabilization. I had all these in all areas that I was fixing. And I thought there was a problem with my SQL query because what I wanted to do was say, Hey table, give me this item ID, like whatever the ID is, given this one thing.
And I couldn't get it working for some reason it worked fine, but because there was another exception happening in the code behind it, with the right object, it was having an issue. So I wrote this line, which is select star from ride where ID equals ID. 
Frank: [00:43:10] No, you're not allowed to do this. I do. This is called SQL injection for those who are wondering what James just said over the last few seconds, he just wrote the classic SQL injection.
In his case, it's probably a little bit safer than that because he is injecting an integer into a string, but in general, never, ever, ever, never, ever, ever, ever. Ever ever, ever, ever put data into your query string. So, uh, what you would do in this case is, uh, queries can be parameterized and the parameters take the form of a question mark.
So wherever you need to put data inside of the query, you put a question mark and said instead, and then. Uh, at the end of the query, you do a comma and you pass in your data just like you would for string formatting or a console write line. But instead of a curly zero curly, you put a question, mark, and that's how you 
James: [00:44:10] parameterize.
Now what I should do though, is I should do this, right? I should say database table first or default. Async where r.id equals equals. 
Frank: [00:44:23] Yeah. So there was using proper link using that 
James: [00:44:27] syntax. I should do that. I should do this. This is what I should do. 
Frank: [00:44:31] Yes. Um, you could also do it as aware, you know, there there's a few ways to write this kind of thing, but you know, I, I, I like to throw in CQL myself, you know, sometimes I feel like link sometimes I feel like sequels.
So it's whatever sparks joy, James. 
James: [00:44:48] Yeah. Because if he did where you would do well, R equals ID and then first or 
Frank: [00:44:54] exactly. Yeah. 
James: [00:44:56] Is it a, is there more proficient, which one's more proficient? The where, because the idea is in next year, 
Frank: [00:45:02] they should actually compile down to roughly the same thing. So you, you would need a micro benchmarker to find the exact difference, but computationally they're identical.
Let me 
James: [00:45:11] ask you a question. Should I put an index on my date, UTC, since I'm going to order by descending. 
Frank: [00:45:18] No, it's not worth it because, um, okay, well these rides don't happen. Sorry. Sorry. Let me start over. Um, If you, if you're just doing a query where it's just happening once in a while, uh, you don't really need the index.
If this is a page that's showing up in your app, 8,000 times, and someone accumulates 8,000 rides and you never do, um, paging or anything like that. Then maybe an index. Yeah. Maybe an index would be useful, but it is that kind of extreme case me, I would never display all 8,000 rows to the user because no, one's going to scroll 8,000 times.
So you show them the most recent 200 rides or something like that. And in that case, it really won't matter. 
James: [00:46:10] Yeah, that's what I'm going to do at some point. But right now I'm just going to show them all the rides. It's all 
Frank: [00:46:16] in that case, you might just, okay. It doesn't hurt the throne index on. So don't, you know, I feel bad for databases.
I know how much work they're doing. I just feel bad when people put on way too many unnecessary indexes indices. Originally 
James: [00:46:32] here's a quick one to be, it would be select star from whatever order space by space, column space, Des. 
Frank: [00:46:40] Oh, it is short. It's funny because sometimes SQL is long. Sometimes it's short.
Yeah. So it's DEC and probably ASC then for us sending yeah. 
James: [00:46:49] How come Frank CQI dash net? My select is lowercase and it doesn't need to be uppercase. 
Frank: [00:46:57] Well, SQL does not care about casing. Right. It does for data, obviously your column, I'm sorry, case sensitive, but the syntax SQL your table names, your keywords.
This, this is a language design in the fifties. It was meant for corporate users typing in live queries to databases all day long. And they can't be bothered with uppercase or lowercase. Who's got time for that. So, yeah, it doesn't care. 
James: [00:47:26] Th this is just good knowledge. I didn't know. But 
Frank: [00:47:29] yeah, the th th the traditional way was all upper case for the key words, and then normal kind of camel casing for your tables, but modern SQL is everyone does lowercase for the keywords.
Awesome. 
James: [00:47:43] And this episode of merge conflict, James Frank teaches James cul. Um, uh, I know this has been fun. I really wanted to get out of today was a, uh, Kind of talking about feature requests and how they can dramatically really alter what your application. Okay. What I want to talk about today, I'm going to re-emphasize I'm, I'm going to cut it because we starting 
Frank: [00:48:10] over, 
James: [00:48:12] but I really wanted to talk about today, which I think I got the point across and tell me if I hadn't is like, there are feature requests or features that you personally want to.
But then there are these additional questions and things that take into consideration that often can take a lot more time. Like putting in this feature was not trivial. It was substantial, which is why, by the way, it's going to be a pro feature. But, um, you know, it was fun to do because I actually, haven't got to write Ross.
EquiLend a long time. So. 
Frank: [00:48:43] That that's great because I almost forgot to bring that up. And I wanted to bring that up earlier. This is a perfect case for our pro feature, and I'm trying to convince my own self of that. So watch out everyone I'm coming out with pro versions of my apps, uh, because some of these features are huge compared to the app.
I would say this feature is probably just as big as your app was prior without all the Bluetooth code. I know that Bluetooth code is ridiculous, but, you know, take away the Bluetooth code. It's probably roughly the same. Uh, the other thing I'm thinking is I was just making, uh, a little one wheel at myself, not so different from your RPM counter.
It's just telling me the speed of the one wheel. And now I'm thinking, oh gosh, do I need graphs in it now? So now I'm feature creeping my own app, but at least this came from a customer. Who gave you an a very explicit use case. So, you know, at least there's one person in the world that's going to appreciate this feature.
I, as much as I would love to do features that I just, myself personally want, I give a much higher priority to features that, uh, people ask for. I mean, how can you not, I'm just trying to help out the customers. 
James: [00:49:52] That's all you can do, Frank. That's all you can do. Uh aren't we did it 15 minutes. SQL Lite. 
Frank: [00:50:00] I'm glad we came back around to that.
Cause I wasn't sure where we're going with SQL. I mean, we, we can get on to joins. I mean, there's a lot to discuss about joins, but what we'll save that for another episode, I 
James: [00:50:10] guess we will save that for another episode. I mean, I I'll tell you this much. I've definitely done the first approach by the way, the first approach that I did was.
Am I am I a meetup manager that used to be for, for meetups, who was the log people in every RSVP, whenever you would tap on someone was a data entry. So it was like in the night I had, you know, the primary key foreign keys and all this stuff. And I did all that, that stuff, that joining, if you will, um, manually w which is not that hard either, but.
I just for this one, I was like, man, I just don't. I just, I just felt wrong. Storing thousands of rows of data. I know that it's not because like, these databases are awesome, but I was just like, In my soul. I just don't want to, I don't want to, I don't know. I don't know. 
Frank: [00:50:57] No, I, yeah, it's we won't reiterate, but sometimes a numerical matrix is just a numerical matrix.
It doesn't need to be encoded into rows and columns in a relational database database. Sometimes it's just a bunch of 
James: [00:51:12] numbers. It's okay to not be okay. Um, all right, Frank. Well, thank you so much for coming back and being here with all of us this week. Thanks to all of our patron subscribers that got a cool bonus episode of this week at a patrion.com/merge conflict FM.
It was going to merge conflict out of him for all of the fun stuff, and we'll be back. Next week with even more goodies, Frank, I cannot wait. And then we have, we have like an upcoming interview upcoming, then, uh, I'm going to go on holiday. Like we won't skip a beat. It'll just magically happen. I'm super excited for it.
Um, but it's gonna do it for this. Week's a merge conflicts. So until next time, um, Jason wants bag now and I'm 
Frank: [00:51:50] Frank Krueger. Thanks for this nice.
Hm.