mergeconflict235-1
James: [00:00:00] Happy 2021, Frank, we did it. We made it, we did it. It happened 
Frank: [00:00:14] living in the future. It's amazing. I don't know if I was going to make it this far. Just kidding. Congratulations to everyone. Welcome to 2021. It's a whole new. It's not even a new decade, is it? It's a new something. 
James: [00:00:28] It's a whole new something. 
Frank: [00:00:31] I guess thing.
James: [00:00:32] What's every 10 episodes. It's a whole new lightning talk episode. That's not this one. Actually. Wait a second. Now, what is 
Frank: [00:00:38] tricking me? 
James: [00:00:39] No. What is, what would, what would be a, it's not a century, cause that would be 2000. 100, 
Frank: [00:00:47] you know, it's like programmers off by one errors. Like is the new decade? Does the decade start in 2021 or did it start in 2012?
I think it started in 2020. It's counted zero. So it's the first year. If we're counting by zero new decade. Okay. Let's stop. This is a terrible topic. 
James: [00:01:07] Well, you know, the best thing out there possibly is that, uh, we now can sign our checks with. Just 21 at the end. We're not to put 20, 20 anymore. We can just put 21, 
Frank: [00:01:19] but they might confuse it with 1921.
Isn't that 
James: [00:01:23] true? That's every year. 
Frank: [00:01:27] Uh, yeah, I guess I'm still doing the long hand. When was the last time you wrote a check? 
James: [00:01:32] Oh, I write checks all the time. I'm always writing you a check all the time. Frank. 
Frank: [00:01:36] I love those checks. Thank you. Thank you, Patrion. Thank you 
James: [00:01:40] listeners. That Heather was asking me, she was like, how does Frank survive?
And I was like, I just keep writing him checks. Nah, I'm just kidding. I was like, you know, 
Frank: [00:01:49] whatever scraps 
James: [00:01:51] are leftover, I go, no, Frank is a highly successful. Independent app developer. In fact, I saw on his live stream, a new app icon or emerge out of nowhere. And I was like, Ooh, that looks fancy. I hope that Frank has released his app cause it's 2021, but I don't actually know if it's happened 
Frank: [00:02:10] yet.
Oh, Oh, that little arrow to my chest. Thank you. No, no it hasn't. But here's the good news. I think I made, um, on this show, I believe I made a commitment, not a new year's resolution because we don't do those. I made a commitment that I would release an app that year. And then a pandemic happened and I got lazy and I didn't finish the app.
So, you know, that's what I have you for James, you release all the apps so I can sit back and just live vicariously through you. I don't need to release apps, but truly I do make my living off of selling apps. So I do need to release this app, but I haven't. Sad trombone 
James: [00:02:54] support the brain. Well, you know, that actually gets me to the topic that I want to talk today is I'm calling this episode, holiday hacks completely, 
Frank: [00:03:03] uh, success, success, 
James: [00:03:07] successful holiday hacks, maybe that's what successful holiday hacks.
Frank. We did it. We didn't even announce it exactly what we were hacking on. We kind of did, but you know, then holidays emerge and then sometimes your holiday hacks change. Frank. Sometimes they just change. 
Frank: [00:03:22] Uh, they do. Uh, but we've been covering, I think your holiday hack pretty carefully through the last few episodes, it's been quite a documentary.
We've been building up of the app and release of a single number on a screen with a star and the upper left that, that quite dramatic. Would you consider that your holiday hack. 
James: [00:03:45] Well, that's the thing is what started it as I'm going to do this over the holidays, and then randomly had a few hours at night after Heather went to bed, then I really started working on it before the holidays.
It almost doesn't count because we did an entire episode mid holiday. And that means we recorded a few days earlier. So I don't know if it necessarily counts, but the fun part of the holidays is with shipping delays, all the goodies claim in. During the holiday and buy goodies. I mean, my SB 32 wifi BLE 0.96 inch one 28 by 64 O led lipo, a wifi kit.
I'm so excited about my 
Frank: [00:04:28] phone. Is this a new Amazon purchase? I can tell from the title BLE, so great. We're doing another Bluetooth episode. I'm so excited. I'm just kidding though. I'm sure we will talk Bluetooth. Uh, I totally lost track of the rest of that, but all I can assume is that if you spin this thing around in circles, It'll tell you how many circles per minute you're making 
James: [00:04:52] no.
Frank is that right? The reverse is that if you have a thingy, that if you have a thing you already, that will spit out some Bluetooth junk, this will put up, this will put a number on the tiny little led screen. That's on this little SB 32. 
Frank: [00:05:07] Oh, fantastic. So you've you built your you've turned your app into an IOT thing, or is that what you've 
James: [00:05:15] done talking about this lack last podcast.
And it happened, Frank I'd live streamed it. I built the entire app in two hours. 
Frank: [00:05:25] Live that I CA you built the whole thing. I, sorry, I'm just a little awestruck because I don't quite believe you. Um, so define what you mean by you built the whole thing and to find the thing, because I feel like I was totally off track here for a bit.
So last I remember is you were going into my favorite device, the SB 32. Okay. And you had bought one. I'm not sure if this was on the show or not, but you had bought one within, uh, Oh, led screen, a fancy old led screen. And your goal was to take your RPM measurement app and turn that into a little cutesy IOT device.
James: [00:06:07] That is correct. Yeah. This little puppy is very impressive for $18. Frank. I am. I know that we've talked about IOT devices before we've done entire podcasts, but I am impressed at what $18 gets you. And what it gets you is this tiny board that has a bunch of circuits on it. Right. And like you said, I don't need any soldering, which is good because I'm just using the core of what's on there, but it has a tiny little led screen.
One 28 by 64, which is, I mean, who needs more than that? Frank? Let's be honest. 
Frank: [00:06:45] Yeah. One 28 by 64. Fine. Can you fit asteroids on there? I don't know. I, I, I want one 28 by one 28, but fine. One 28, five 64, please continue. 
James: [00:06:58] 28 by 64. It also has a wifi and it has Bluetooth built into it. And you just plug it into your computer and you can code on it.
So my goal of this was to take my mobile app and turn it into a IOT device that would. Um, display your cadence. So my application, my cadence, because that's the best name that I can come up with it, which we did an entire podcast on. I'm not going to go into it. All it does. Is it. Connects to Bluetooth sensors that are on stationary bikes, like cycling, indoor bicycles.
And it tells you your RPM, your rotations, per minutes of the crank arm, we call it the cadence of how fast you're moving. And that's important because when you're exercising, they call it the cadence. Well, it would be really nice, right. As instead of having to bus out your cell phone and plug it up. What if you had a little tiny display that was just always on always powered.
And I was really lucky. Because one, these devices are legitimately $18 and it gives you everything you need. But I was enticed and intrigued by a, um, a hacker news commenter who was kind enough to dump the source code that they had written, um, that did something very similar, but very specific to a, um, Bluetooth enabled bike.
Right. So, so same, same concept, except for all different. Um, logic and Gwydion and things like that. Right. So, cause it has its own bike. It's like a fitness bike and it, and you have to like write data and read data, but I was like, Hey, the guts are here. Right. It's like good bones. Like when you find a house like that house is kind of like not what I'm looking for, but it's got good bones and that's what this source code had.
And that was my starting point. 
Frank: [00:08:50] Yeah, I, I think I keep repeating myself here. When I say I'm really impressed with the Arduino community. They've been writing really good high quality C plus plus code lately. And. Not to be too silly, but, um, C plus plus has taken a weird spin. There's like modern C plus plus, and it's really weird to read and it's kind of hard to read.
And I'm just saying that because it's not the C plus plus I grew up on, you know, But the Arduino community has been producing these open source libraries, predominantly open source where people are writing really high quality object oriented C plus plus code. That's very easy to read, very easy to consume, very easy to translate into different languages and all that kind of stuff.
And so, um, Kind of, I'm not shocked at all that that code existed because that's the whole maker, IOT community totally happening right there. And it's been fun. I dip my toes in it from time to time. I generally just write silly things that turn lights on and off, but at the same time, I'm kind of inspired by that community and yeah.
Everything they're producing. 
James: [00:10:06] Um, I'm actually impressed with a lot of these little tiny microcontroller communities, including, uh, the raspberry PI community, the adder fruit community. I don't know if those are the same thing. Um, but what I've noticed is I've seen a lot of tutorials online and it's great because it's like, here's how you do the one thing, because these boards are allowed to do one thing.
You know what I mean? Like that's the cool part about IOT is like, they can ideally do more than one thing. But they oftentimes do one thing. Does that make sense? 
Frank: [00:10:38] Sure. But it's kind of the software dream where you're given a bunch of hardware that can do a billion things. Like there's so much those sports can do.
And even that microprocessor like that board isn't even doing that microprocessor justice there's things that processor can do that, that board's not exposing. Um, but, um, the magic is in the software. So it's like, you're turning this general purpose piece of hardware into a specific device. And that's kind of the joke.
I, I, I S I call them single-purpose devices because they're kind of wonderful. You're just writing this one app. It's not an operating system. You're not logging into it. It's the app that this device is going to run for until the end of eternity, as long as you keep that battery charged and that.
Permanence is kind of what I'm in love with that. You can take your software, take this general purpose piece of hardware and tune it to your application. Sorry, I'm getting kind of like silly philosophical here, but this is really why I got into software is like this kind of stuff. 
James: [00:11:46] Yeah. It's super duper neat.
And I think what's cool about it is that there seems to be a lot of people that are. Like, it's almost like how I develop a Habs. And that's what I feel like is really cool about this community because how I develop apps is, Hey, I have a problem and I'm going to solve it for myself. Like, that's why I built my cadence is because Apple fitness plus was coming out and I go, Oh, I just want to, you know, I want a little cadence thing for myself, so I'm going to build an app.
Cause that's the thing I know how to do. And what I've seen since looking at this project. And forking it and making it my own is that there are tons of people that are doing the exact same thing to like building these little tiny IOT devices, like for their set up. They're like, Oh, I have this, this trainer, I have this sensor, I have this bike.
And I am now going to make this little thing. I got someone literally sent me. Um, it was like Swift code that runs on a, maybe it was an adder for, I dunno, some IOT device. I was like, I don't even know. Swift could run. On a, on a tiny device. I don't even know how's that even possible. You know what I mean, really like blew my mind.
And I was like, that's a really cool, and that's what I really started enjoying about this process of building this application. And, and, and that was how I was able to do it. And like, maybe it's two and a half, three hours Frank, but I mean, I was able to do it in like three hours is because this community had for all intents and purposes built a thing.
And when Frank. You have a gooey user interface service surface. That is one 28 by 64. Like, guess what? The user interface is really easy. 
Frank: [00:13:29] Two labels, two icons is what I remember most from your screenshot, something like that. And I kind of want your font to be bigger. I don't even feel like you're using the false one 28 by six 40.
I think you're kind of squeezed up into a corner there a little bit, James. Um, I ha I hope you somehow find a way to post that screenshot to this podcast. Sorry from everyone. But yeah. It's, uh, it's a mixture of things. So you have this processor, you have an L L O led screen, and you have your Bluetooth code running.
The only thing you're kind of missing is like, Button to be a proper user interface you need ger on click handler and that kind of stuff. Uh, and I think that that's just kind of the next logical step. The fun thing is you can write companion apps to this because it's talking Bluetooth. So you could have my cadence talk to my IOT cadence very easily and do user interfaces that way too.
Lots of possibilities. Isn't software. Wonderful. 
James: [00:14:40] It is. Well, this thing is actually neat because there are two buttons. There's the traditional reset button, but there's also a program button, which this person actually took into consideration in their code. I, I got lucky this, this code scan for wifi, or was a scan for Bluetooth devices.
It had the UI to like display them, refresh it, and also pick one or just connect to one automatically. Again, I had to, I had to write all the new logic, which was nice because. Actually porting code from C sharp to C plus plus when you're just literally creating integers and doubles. Is pretty easy to be honest with you.
Like it's not that hard. There's like if statements like they're the same. 
Frank: [00:15:22] Yeah. It's called the C family of languages for a reason. It's not called D sharper, G sharp. It's called C sharp. We're giving you the C programming language with a few restrictions and giving you a garbage collector, have fun. Um, yeah.
So, uh, we've talked in the past about, um, Necked wino and wilderness slabs, a meadow. That's where they're trying to get C-sharp code down onto these same kinds of devices, but the most basic way, the simplest way, just because we've supported since the 1970s is to use a C plus plus. And it's funny that like, it didn't have to be C plus plus it could have been objective C you mentioned you got Swift running because any compiler out there nowadays can output.
Native code something that can run on a chip. So you'll see, even out there, there are versions of Python called micro Python, where they've gotten the Python runtime to compile down onto these chips. So I think we've also seen a little revolution where these programming languages were, which were designed for desktop operating systems.
We're like, well, we're just a little bit of. Hard work and some forethought and maybe removing a feature here or there, we can squeeze them down onto those devices too. So I'm really curious with all the props to the Ardwino community and their wonderful use of C plus, plus I am curious in 20 years or 40 years, what the dominant languages will be, but the good news is, um, A Thursday's wonderful open source libraries out there be.
You can probably figure out a way to get your favorite language on there somehow some way too. 
James: [00:17:09] Yeah. I would say that the largest hurdle to overcome for these is, is getting into development. Right? This is a Arduino board. I used Arduino studio, the biggest hurdle. Is the IDE the ecosystem, the documentation, this thing is all over the place.
If, if anyone joined my live stream, that I had a bunch of people that was super helpful, like Samir helped out and a bunch of other people that were like did IOT stuff. Because as a developer, like Arduino studio is like so complicated to use, even though it's like so simplistic because I don't know anything.
Right. I didn't understand that. I needed to go add a special, you know, RSS feed that would add a library specifically for this board or that I needed to download. Like the board, like device to be able to deploy it, to like know, to compile the code a certain way for this board. So I needed to go to this random website of the manufacturer, um, to like browse their PDF documentation, like healthtech.com.
Like I went to this Chinese website that was like, here's how you do it. And. They're good. They're like, just go to this good hub page. And it's like half an English haven't ended, which is great. I really, I, I, I figured it out, but I was like, wow, this is like not intuitive at all. Where like I had the meadow and the mat.
I was like, cool, install this extension and plug it in. And you're like, cool. So I 
Frank: [00:18:43] mean 
James: [00:18:43] that, I mean, that's, cause what's what I'm used. Sorry. I installed Xamarin and I'm like, like plug in a phone and you're like, cool. And then like, like I think that I just have taken so much for granted over the years that when it comes to this type of development, I love the simplicity of.
The types of applications, not that they're simple, but the types of applications that I can build in the world and ecosystem them living in this tiny little board, but getting into it, it was so frustrating. Like I got to go over here and do this thing. Like nothing really made sense, like go to this preference screen and then add this thing.
And then. Okay. I'm like now, which one do I select? And like, that's not documented. And I was like, I guess I'll try the first one. Like, okay. 
Frank: [00:19:24] That, that screen is hilarious. The Arduino screen. So once you figure it out, um, yeah. How to install the device or the board, I think they call it boards and the IDE boards.
Once you get that installed, like you said, you can go select the device and it brings up this list of like 50 different devices. Mine's not on there like that this random company, I bought this from that one's not listed on there. A rule of thumb. One of those, those is called like a dev kit. That's probably the one that you want, or one of them's called, um, um, micro node or something like that.
Node MCU, something like that. There, there was kind of an established standard that everyone started adopting. The difference is the other ones have some specialized features that they'd light up for you. Chances are, you're not using those features. So it kind of doesn't matter. As long as you pick kind of the generic board, it's funny that you said this is an Arduino board because it's not actually how that board started.
It started out as. Uh, Lua board, you know, Lua, 
James: [00:20:28] I know Lou popular scripting language for games. Oh, that's 
Frank: [00:20:32] right. I forgot. It was a gaming thing. Yeah. In fact it was, um, a neuro network thing for a while. 
James: [00:20:39] Really? 
Frank: [00:20:40] Yeah. Uh, torch is a pretty famous neural network library out there. The version everyone's probably familiar with is called PI torch, which is released by Facebook, but long before PI torch was.
Torch, which was a Lua wrapper over the torch library. So Luma's, it's funny what pops up and it's when they were first designing the sport, they wanted to make the development experience easy. And there was totally the Arduino community, but they kind of had their own boards and everything, and they thought that they could do a little bit better than the Arduino communities.
They released this as a loo aboard and you're supposed to program it in Lua. Turns out the Arduino community is quite big and powerful and everyone just wanted to program it as an Arduino. So it's just, sorry. I just had to bring up that funny little history that for a short time there, these were actually mostly Louis things.
But, uh, you're right. This is kind of like it's starting back out and computer science, like we reset computer science with a new board comes a new operating system, comes to new programming, languages comes, new ideas, all that stuff. You have to reinvent the wheel. Every time you start on a new board, it's kind of gross.
And that's why we like cross-platform languages because we don't want to have to reinvent the wheel. Every time this happens. But, um, it's interesting because you can pick your level on the stack. This board, the SP 32, that's the microprocessor, uh, that is from a company called SPF and they've been kind enough to release all their libraries as open source.
So you can start a C program using just the SPF libraries themselves. Now. They're kind of low level by kind of, I mean, very, very, super low level and take a lot of reading and a lot of work to understand. And that's why most people go one level up. And that's the Arduino level of all you're working at, where you have all the basic art, you know, IDE you have the Arduino Aries that run on it.
It's just moving up on the abstraction of all. And then once you get tired of that, you can actually move up higher. And the abstraction level and go to something like C sharp in Python or Lua and in this case, 
James: [00:22:59] yeah. It's um, it, it was an adventure. I would say. I definitely completed it. I tested it. It totally worked.
And, uh, then I went a little bit further and I did fill that blank space with a little power meter. So you could determine like how, like how far you're going, because I did use the maximum font height, surprisingly that I could figure out I had, maybe I could hard-code something, I don't know, but I. I thought about that too.
Um, but. Yeah, I just 
Frank: [00:23:25] interject there. It's really funny how they handle fonts or at least I'm, I'm assuming you're using this one library. You there, they're all bitmap fonts and they're actually included as C files that are just giant or res of the bitmap kind of, you know, so it's kind of funny how that works out.
And so if you want. Bigger fonts. What people do is they release these C files and these header files that just have larger and larger and larger characters with different stylings and things like that. But fundamentally, it's just an image. It turned into a CRA and released on GitHub. 
James: [00:24:03] Yeah, that makes sense.
Because the images that are on it are an image that are converted into the array and then they're then displayed, which is, which is funny. Um, but yeah, this, this health tick is the name of the company and they have a nice library for displaying that even a progress bar it's like built in, which I think is really cool.
So there are some basic building blocks that they have and yeah, it's done. It's out there. It's on the get hub. I wrote a blog post about it. So there's that. Um, and now my cadence is on iOS, Android, and our, do we know? So that was my holiday hack holiday hack success. Frank, I'm very pleased with myself on this.
And now it's, I'm all about supporting and you printed me a, um, a little, uh, thing, image the thing from a thing 
Frank: [00:24:48] averse. Yes, I am sorry to announce live on the show that I have yet to mail it though. 
James: [00:24:55] Wow. Well, I just bought batteries. So you got two weeks to, 
Frank: [00:24:59] Oh, is that right? Oh, shipping. It's going slow, huh?
Yeah, it's, it's kind of fun. Um, along long with all the electronics, the whole 3d printing thing is happening, right. So you sent me a design. That's an okayed. It's fine. It's all right. It's all right. It's it's it's not what I would've done. In fact, I gave a presentation. You know, it's hard to say because time doesn't exist anymore.
But between 10 years ago, and yesterday I gave a presentation online about how to design enclosures. We call them for these kinds of circuits. So if you have a board or a screen or something, you wanted to design a 3d object around it, um, Nice nice ways where you can like form it to snap on to objects.
Like maybe you would want it to snap onto your handlebars or something like that. 
James: [00:25:49] Yeah, that'd be cool. 
Frank: [00:25:50] But like for my one week when I did a little spin amateur for my one, we all had it snap onto like the rail and it was super cool. Hard to get on. Sometimes I don't get off, but it was super satisfying when it would make that click and everything.
But, uh, yeah, you sent me a case. I printed out very quickly and it's just been sitting here. Um, it's been the holidays and all that kind of stuff, but it'll, it'll get sent to you soon. Good. 
James: [00:26:14] Well, what's your holiday. Heck Frank, because you have been busy coding. I am pretty sure. 
Frank: [00:26:20] Oh sure. Um, I've if we're just talking coding, I always have a million ones, but I think the holiday hack that actually deserves attention has been what I've been Twitch streaming about, which has been, I've been having fun with the LIDAR on the iPhone, and I've been trying to build kind of a scanning app and I've been doing it live on Twitch and it's been kind of hilarious because it's all just a bunch of math.
And a bunch of just like, it's so hard to debug because it's just all math and everything, but it's fun because there's the chat room and we're all hanging out. Yeah. And all the bugs look ridiculous because it's just my face on the screen getting distorted and really embarrassing ways. But I've been having fun building the scanner app and it's getting so close, so close to done, but James you've released an app and an Arduino project.
And all I have is some code that maybe works, uh, maybe two more Twitch streams and it will definitely be working. 
James: [00:27:26] One thing that's very fascinating is that when I tuned into your stream this last weekend is that you are writing some. C plus plus code, right? Or objective secret where you are. You're in some C plus plus objective C or C plus plus.
Frank: [00:27:40] All of the above objective C plus plus even though that's a language. Yeah. Isn't it weird? It's funny because it's actually going back to my roots. When I first started out in iOS programming, I started out in objective C plus plus, and that's because. I'm a C plus plus programmer. And when it came time to learn iOS, you know, objective C is weird.
It was really hard to learn in the beginning and I ease myself into it by writing objectives, C plus plus. And so what happened was there was just this Apple sample out there and. It's so much easier to just hack on someone's sample sometimes then to translate it, like all the code I totally could have just converted over to C sharp or F sharp.
And at this point really wish I had, but, um, for better, for worse, I just kept hacking away at this, um, objective C plus plus, but I also took it as an opportunity to learn how to do some Swift programming. Which has been kind of hilarious because I've produced some Swift code. I have written libraries for professional organizations and Swift I've written two different compilers for Swift.
I, I would say on my resume, I can write Swift code, but it turns out I'm not very good at it, James. And so it's just been a real fun. Um, just, you know, stretching your wings. See if I can figure the language out, I've made a lot of conclusions. It's an excellent language with a terrible debugger I hundred billion percent missed the.net debugger that can just inspect any object completely.
Yeah. Oh my God. You missed that feature so much. I forget in native code, how like dumb the debuggers are. Okay. Okay. Stamp on me. I mean, 
James: [00:29:38] I was, I was coding C plus plus in the Arduino and you know how I debugged it serial dot print line. 
Frank: [00:29:45] It's all you got. Cause those, the fuckers are terrible. And I say that because, um, I was complementing LLDB sorry.
I dunno. Which just going to rag on debuggers all of a sudden. But when I was working on the catalyst stuff for Mac and iOS, I was just living in the debugger because it was all native crash, native crash, native crash. I'm just like these debuggers are so bad, James. I, so miss the dotnet debugger it's I am never going to complain about an IDE bug ever, ever again.
I am just thankful that we have our beautiful managed runtime. 
James: [00:30:21] Okay. And that's true. It's true. It's true. It's true. It's true. I agree. 
Frank: [00:30:27] I didn't mean to make that a commercial for a managed memory frameworks in particular.net, but Oh, it makes you appreciate it. Uh, on the good side, um, you have Python. If you want to try that or Lua Louis carpets collected also, I really don't know why.
Arduino is stuck in C plus, plus I hope someday the Arduino community finds a different programming language 
James: [00:30:55] one day. Well, so what is this app? So we get, I want to step back and talk about your holiday hack a little bit, because I saw I came into it. I'm like, I don't know what you're doing. It's just like a bunch of gibberish and your phone's on there.
Okay. So let's talk about what was your app idea, right. And we can go from there. 
Frank: [00:31:14] Okay. Um, fundamentally I'm interested in just one simple problem, given one 3d scan of an environment, and then a second one, figure out how much the phone moved. So it's more of a tracking problem. Fundamentally. It's like, um, imagine like your eyes are open and you see the room, shut your eyes, rotate 45 degrees.
Look at the room again. And then, you know, stitch those together in your head. And we've had image stitching software kind of for the last 10 years. It's been really good. We can stitch two 2d images together. But we now have 3d sensors on phones, the LIDAR sensors. And what I discovered is that any phone with face ID has this built-in LIDAR sensor, and you can do 3d scans of the room.
It's like the connect sensor on your phone. And what I wanted to do was not 2d image stitching, but 3d image stitching. So figure out how much the phone. Moved translated and how much it rotated between two different frames. That was 
James: [00:32:29] it. Interesting. Is there a use case for this 
Frank: [00:32:32] thing? Yeah. Um, a million, because it's kind of a fundamental problem to multiple use cases.
So the first use case, the kind of obvious simple one is stitching together multiple scans of a room. So let's say I want to scan my entire apartment. What I need to do is, you know, the phone only has so much of a field of view. It can't see the entire apartment all at once. So. I have to take a picture of the wall, you know, whatever you want to think of it and video pictures, same difference.
Uh, and then I have to walk around the apartment and then some magical software has to take all those frames and merge them together to create this total 3d scan of the room. Yeah. And yeah, so, right. So the process is between each two frames, you figure out how much the phone has moved. And, you know, within reason within errors and all that kind of stuff, you can keep moving around the room and it'll just stitch it kind of altogether.
And what was kind of neat. I always knew that that was the process. And there are multiple algorithms out there to do that. And I had just from my college experience, some rough ideas of how you do it, but there are all kind of complicated algorithms. I never thought I'd ever be able to implement them, but it was just some, one fun day.
I don't even know how it happened. I just ran into this research paper. That said very clearly, like in a four page paper, which is not at all normal for a research paper, they're not usually very readable. Um, but I could actually understand this one and it's like, Hey, if you have this problem of you have a 3d scan here and a 3d scan there, do these five steps, these beautiful five step program.
And we will give you the optimal solution to that problem. And I'm like, Well, gosh, darn it. I like optimal and it just seems so intriguing that it was this beautiful solution that I just had to implement it. I didn't know it was going to take me so long to implement though. 
James: [00:34:45] Now when it comes to LIDAR. So describe LIDAR for me because, you know, I think a lot of people have heard of it because it's in self-driving cars and we know that Apple put a LIDAR sensor into things, but what exactly does that give us as developers?
And like, what is the fundamental technology? I think it's like, is it, is it sonar? Is that what it is? No, it's not sonar. Is it sonar? 
Frank: [00:35:13] No, it's not it's light. It's not audio. Sonar is audio. So this is light. This is light. Um, the exact precise physics of it. I cannot describe, unfortunately I have a friend that can, but I cannot, um, something about laser beam, diffraction, putting a grid out into the world.
Um, the, the mental model that you want to use though, is that there is a grid. Of 640 by 400 laser beams shooting out of your phone. And every single one of those little laser beams can report how far away an object is that it hit. So what you get as a programmer is a six 40 by four 80 image, but instead of color, you get depth.
How far that laser beam had to travel before it hits something. Hmm. 
James: [00:36:08] And then I can give you a 3d representation because if you were to put that on the Z access, that gives you that it's Oh, you know, Oh, you know what it is. You remember those toys that were, um, little, like things of metal, like you'd put your face into it or your hand into it.
And the metal little sticks. That's LIDAR. 
Frank: [00:36:31] I think I'm there, but I'm thinking more like Krypton technology. Did you see the new Superman movies when they're on Krypton and like the little needles come out of the thing to make and user interfaces? No, no Bueller? No. Okay. 
James: [00:36:47] Do you know what the thing I'm talking about?
It's like in 
Frank: [00:36:49] silence. 
James: [00:36:50] Oh my gosh. It's like hand. 
Frank: [00:36:53] In hand. Okay. Is it at the store? Yeah. Okay. I think I know what you're talking about. 
James: [00:37:00] Oh, pin art. Is that what it is? Yeah. Come on. Sure. 
Frank: [00:37:04] Good for pin art. 
James: [00:37:05] There you go. Look at this Amazon link. Boom. I'm putting in the show notes, everybody. This is what I'm talking about.
Penn art. 
Frank: [00:37:12] Uh, yes. Okay. Yes. I'm here with you. I remember this was a popular Christmas gift at some point. Uh, this is exactly it. So the neat thing is on the iPhone. It can tell you what you need to know is these lasers are being shot out, but you need to know exactly what angle. They're being shut out at and that's called the lens intrinsics so the iPhone can report to you exactly how its lens works also.
So if you couple how the lens works with how far that laser beam had to go before it hits something, then that will tell you it's 3d coordinate. Now the problem is it's a 3d coordinate in the camera frame of reference and it's space. So, if you want to stitch together a world, you have to come not to the cameras, uh, frame it's frame of reference it's perspective, but you have to establish more of a global perspective, a global reference frame, and that's the little trick that I'm performing.
So the idea is take one picture. Move the camera. Take another picture. It figures out that transformation by taking that depth data, turning it into 3d points, doing that for the second frame, and then figuring out how to turn that one set of 3d points into the other. And that gives you the transformation.
How did that explanation go? 
James: [00:38:40] That's good. That makes sense. I'm counting of envisioning it. And what I've been doing here is like, Opening looking at my monitor, closing my eyes, rotating 45 degrees and opening my eyes and like trying to figure out like that process to get from X to Zed. 
Frank: [00:38:55] Yeah. So the cool thing is you can use that.
So getting back to your original question five minutes ago, um, you can use that for the problem. I just described of creating like a 3d scan of a room, but you can also use that for navigation. All right. So here's another fun experiment I did with it. Um, I treated the central laser beam as a sonar beam, so I turned LIDAR into sonar.
James, you should be happy. And what I did was, um, The farther and object was a way the lower frequency. It was the closer it was to you, the higher frequency it was. And then I would blindfold myself and walk around my apartment and try not to walk into walls. And it was really fun because it's the same kind of, it's actually a very simple case of this.
I don't even have to do the 3d projection because I just know how far objects are away from the phone, even without being able to see them. 
James: [00:39:57] Very cool. Very cool. And then did you feel, as I know you said you're not done yet, you feel like you've successfully at least sort of gotten movement on this thing.
It sounds like you have 
Frank: [00:40:10] Oh yeah, absolutely. Um, it's it was fun because. At first, it was like so much data. I was so worried about it. Like, so I was like really worried about performance and all that kind of stuff. But then I forgot the iPhone is infinitely fast these days. 
James: [00:40:27] Very, very true. Very true. 
Frank: [00:40:29] Oh my God.
So like it's just chewing through this stuff. You know, the biggest problem I've had with it is the stupid LIDAR can overheat. Ooh, because it's just shooting all this energy out into the world and, you know, nothing comes for free. So it's some things are just absorbing that energy, unfortunately. And so I'll run into unfortunate situations where it'll just overheat.
And so when you use these API APIs, you actually have to track that kind of stuff. But in general, as long as you're not like running the screen running under her own network, downloading a video from. Twitch, you know, doing all these things simultaneously, it'll be just fine. But it's funny that you have to watch out for those funny kind of things.
Now I will say I've gotten the algorithm that I mentioned it before that beautiful, gorgeous, elegant algorithm that I can't say the name of. And I can't remember it because it's like this German name,  optimization, something like that. Uh, it's a beautiful algorithm and it's giving some results, but I haven't been able to verify them because it's just one of those problems where you're trying to relay ridiculous amounts of math, to the physical really real world.
Yeah. And that's always a bit of a problem you're running into this. Like you're, you're taking this random data out of the ether and trying to turn them into an actual number that represents the physical really real world. And it's, it's tricky to get that right. 
James: [00:42:05] It's tricky to get it right, because.
There's a lot of variables involved. Like in my case, there is so many different types of sensors. I can only buy so many Frank I'm going to have, I'm going to do a review of all thousand sensors that I have sitting around ways. And, um, and they're all D like there is a specification, but in my instance, I'm like, Oh, I forgot about this use case, or this year I've only optimized for one use case.
So I think in the, I imagine the LIDAR case, there's a lot. And a little bit more complexity than what I'm running into, maybe. 
Frank: [00:42:40] Yeah. But it's the kind of complexity I kind of love. Um, It's a big data kind of complexity. Uh, it's a signal noise, kind of complexity. It's funny because not all those laser beams have good Vives.
A lot of them just fail to do their job. Um, if they hit a window who knows what's going to happen because a window can either reflect or transmit. And depending on the angle, depending on how dirty it is, all that kind of stuff, mirrors, mirrors just create like an alternate reality that is in computable for these things.
And it's just, it's how they handle those. Um, but then I have like fun objectives too, because again, with the same problem of I scan the world and then I look at how the world changed in the next frame. I want to do other things like robot navigation with them. So detect the floor, the tact, where in the room I want to go and then do obstacle avoidance.
So it can navigate hallways. It can navigate Lego bricks that someone left on the floor probably named Frank and you know, other things like that. So I'm kind of interested also in it from the robotics algorithm. And that's why I'm having so much fun with the holiday hack, because although I'm working on this one problem, I know I can, uh, do so many other things with it.
James: [00:44:00] Very very cool. I'm excited to see where you go with it and what the final output becomes, you know, in the next few weeks that you're working on the application and I'm just happy that we could take some time. And, uh, we, we both went out today. I did a little hiking and do a little one wheeling. Hope you also got some other downtime.
I know we're still stuck in doors. Um, I know it's, you know, I had some good zoom calls with my family over the holiday. Um, lots of phone calls, lots of, lots of indoor activity. Um, but I'm, I'm glad that we're still able to hack a little bit and be motivated to do it. I think to be honest, I felt very motivated this holiday.
I don't know about you, but for me, it was really nice to have some time off of work. And I told Heather, I was like, I did a lot of outdoor stuff. Um, you know, hiking and skiing. And then, uh, I was able to work on the application. We were able to put up a glow and a little Christmas tree. We had our Christmas, we had our zoom calls.
We had, uh, our FaceTime calls and then I had the indoor cycling thing. And that was really rewarding. I wrote tons of blog posts. I'd just been pumping out blog posts. Um, I've had a lot of people reach out like that, you know? But I put up the one about how I built my bike, um, put it together, like for under $400 and it's, it's nice to see other people joining in and, and kind of like, you know, motivating them to get into the world of, of cycling indoor, cycling at least.
And some other people, even from our last podcast, I was on the Twitch stream. Like they were encouraged. I don't know if it was from that episode, at least from one of our episodes talking about the SB 32. To do an SB 32 development. So it's kind of cool that you get to hear this, and I'd love to hear from our listeners, if you had a cool holiday hack, you know, send it to us, go to merge conflict that FM hit the contact button, send us an email.
We'd love to shout out, to see what our listeners are, are working on and what they're hacking on. I've had a lot of people reach out. That's like, I read your blog posts. And I did this really cool thing. I was like, Oh my goodness. Like, this is amazing. Like, you know, sharing this stuff to the world and letting more people find it, I think is, is really cool.
Like had I never wrote that blog article. And had a knockout randomly picked up by hacker news. I would have never randomly had that person post a comment about their project in which I turned into my holiday hack. Like that's freaking crazy. They're like, that's so cool. I dunno. It's bananas. 
Frank: [00:46:17] That's the world.
That's the wonder of the internet. 
James: [00:46:21] All connected, even though we're all stuck inside. 
Frank: [00:46:24] Oh, you just had to go. It sounds like you're doing a lot better at life than me right now, James. Good for you. Um, I did get to go one way away now, today, the rain finally led up. Um, I'm I'm glad that you got to go skiing.
That sounds. Oh boy. Uh, yeah, definitely. I'm I'm so interested in people's holiday hacks because those are some of my favorite, uh, Twitch streaming programming episodes, where I'm constantly shocking myself because I'm just so careless. I have like an audience, so I'm just like touching random wires together.
And I usually. Make something Exploder or another. So you're kind of inspiring me to get back to a few, uh, shocking experiences on my, 
James: [00:47:10] well, definitely everyone feel free to write in and just go to merge conflict that I found, like I said, I'm and let us know what you think of our holiday hacks, but also we're, like I said, super interested in your holiday hacks, and don't forget that soon into the future, we will be doing lightning topics.
So make sure you submit those as well, by going onto our discord or going on to our Twitter account or onto just write us an email, whatever you want. I think that's going to do it for this week's emerge conflict kicking off 2021. So until next time, I'm James Monson Magnum 
Frank: [00:47:44] I'm Frank Rivers. Thanks for listening.