00:00.38 James Welcome back everyone to Merge Conflict, your weekly developer podcast about all things in the world of software development and new LLMs or not LLMs, who knows, new models, new things. I'm one of your hosts, live from Vienna, Austria, James Montemagno, and with me, as always, the man on an island, Frank Kruger. How's it going, buddy? 00:15.100 Frank Oh, it's going well, but where in the world is James Montemagno this time? Because this is not the normal background. You are showing me something different. I like it though. It almost looks like, are you in a cubicle? 00:27.38 Frank Are you in a cube farm? 00:28.71 James it It really does kind of look like I'm in a cubicle. so 00:31.60 Frank but 00:32.22 James ah Yes, I was at NDC Azo. I recorded the last one from. Oslo if I'm pretty sure I did at least. Right? 00:38.49 Frank Yeah, sure. 00:39.15 James Or London? not I don't know. Wherever I was. 00:40.27 Frank We'll go with that. 00:41.53 James I was in London and then I was in Oslo and then I was in Germany. Now I'm in Austria. We went to Budapest today and then we'll be back to... London and then I'll be home. in oregon by sunday night at midnight and then fresh 00:52.86 Frank Amazing. 00:55.52 James to work at 7 a .m Very exciting. 00:57.20 Frank Well, I'm glad you get to be our foreign correspondent yet again. 00:57.50 James So. uh 01:01.92 Frank I can't wait to hear about what's happening in Vienna. i'm I'm sure this is just a travel podcast now as far as I know. 01:09.39 James you know uh vienna is very beautiful we're just talking about very very beautiful budapest also very beautiful we were in innsbruck doing in leo gang we're in salzburg all very very pretty vienna just we were walking around even though we'd already been here once i'm like man so It's just beautiful. 01:24.52 James just Everything is so regal and big. 01:24.81 Frank Yeah. 01:27.04 James and just beautiful and just like 01:28.95 Frank Seed of an empire. Yeah. 01:30.90 James ah It's... ah It's crazy. If you've never been to Vienna, I definitely recommend it. It's very, very lovely. And yeah, so excited. 01:41.12 James uh to be here on the road we took this this week off between ndc oslo and heather as an event here in vienna so that's why we're here now but we went did a bunch of sauna stuff in between and we took 01:48.94 Frank Oh, okay. Fine. 01:51.50 James In five days, five aerial tramways lifts up to mountains. 01:54.88 Frank Wow. 01:55.75 James Five of them. 01:56.16 Frank Phenoculars 01:57.49 James funiculars 01:57.46 Frank cool that's your favorite topic i can't wait to see what ai is going to generate about these ones this time 01:58.52 James Yeah. 02:04.14 James It was good. I'll tell you this. Okay. So before we get into some new models, so there's. A few things that... 02:08.04 Frank Wait. Okay. 02:10.51 James I think that... I wanted to talk about because I just we had a little bit of fun. On this trip we took... um We drove a bunch. We took some trains. We took so a bunch of buses. 02:21.42 Frank Mm hmm. 02:21.68 James and one thing that we forgot to bring was a keller had a as a is ebook you know uh I have my Switch 2. 02:30.94 Frank Good. 02:30.67 James But we sort of forgot to bring something that we could do together. and we saw some people playing cards and stuff 02:33.30 Frank Right. Right? 02:36.09 James uh and one thing heather and i both read like we're like family feud but i was like wheel of fortune heather's really good at it So I built. uh a two two or three player wheel of fortune game it's called wheel of wisdom because i don't want to get sued 02:44.16 Frank Oh my God. 02:49.40 Frank Okay. 02:50.75 James It's at wheelofwisdom .app. 02:51.93 Frank All hallucinated trivia. Sorry. 02:53.84 James ah Yeah. 02:54.10 Frank Oh, say the URL again, please. 02:55.96 James wheelofwisdom .app 02:58.98 Frank Wow. 02:59.46 James um it's a full progressive web app you can download it offline works there's 600 puzzles stores your history, stores your games. It's got a bunch of different categories. 03:11.28 James ah It's got trips and mysteries and all sorts of stuff. and There's going to be a... i'm I'm just getting ready right now to start a ah full fresh build. uh for an ios application too it's going to be a dotnet maui hybrid app so i'm going to reuse this 03:26.56 Frank Oh yeah, cool. 03:27.37 James eight HTML and JavaScript, it turned out really good. I built this but baby with baby Luna ah that we're going to talk a little bit about this week. 03:32.23 Frank so okay 03:35.50 James And a bunch of other models too. I used a bunch of models on it based on the task at hand. But it's really straightforward. 03:39.47 Frank Okay. 03:40.05 James It's just HTML and JavaScript, nothing fancy. But it's just like, I built it from my phone, Frank. Like this was the thing I wanted to talk about. 03:45.87 Frank Uh-huh. 03:46.58 James I didn't even, I have my computer with me. I'm here at the beautiful Moxie Hotel. um If you've never seen it on Moxie, it's like very... funky and you know exactly what you're getting, which is a tiny room for not a big price. 03:57.20 Frank the 03:57.37 James um And they're everywhere. And it worked out for us for our location we needed to be. And but. as we were going via bus via whatever it in my laptop and i coded this 100 from the github mobile app Uh... 04:11.88 Frank Pretty cool. 04:12.16 James i even bought the domain name set up the dns everything 100 from my phone ah using just the cloud. you know, agents basically, which is it's absolutely. 04:24.77 James ridiculous. And we've been playing it on every single thing that we did. So I just want to kind of throw that out there. 04:30.21 Frank Fantastic. i Honestly, I think you're just feeding into your domain registration addiction. I think you're just looking for excuses to buy new domains. But um this is pretty cool. I haven't actually made that leap of doing it all from a phone yet. 04:43.49 Frank um So once again, you're ahead of me. I'll give it a shot. I'll write some trivial game as a web app on the phone. And and then this is ah there's a fun, trivial game because already there's dollar signs at the top. So I can imagine you two trading money already. I hope there's gambling rules and drinking rules with this game. 05:00.90 James Oh yeah. 05:01.16 Frank um But. it's pretty fun i'm excited to hear what your baby luna experiences are because um I don't know if you've been getting the news over there in the Europe. 05:12.58 Frank But um over here in the Utsa. ah We're not making models anymore. We're slowing down. We're we're slowing progress down. no No more model releases. And by no more model releases, we mean... 05:24.18 Frank both major model providers are going to release a whole new suite of models in the last week I don't know if you've gotten that news, but we're we're both slowing down and speeding up over here. How is Uruk doing? 05:35.63 James ah er yeah The interesting part about Europe... is There's actually more AI kind of everywhere than... 05:43.06 Frank Oh. 05:42.98 James you would you would think so. Oh, definitely. like There's a few small towns we go on the Oregon coast where we like are on the Facebook. It's the only time we go to Facebook because usually each community has a Facebook feed. 05:52.76 Frank Yeah. 05:54.19 James and almost all the graphics for like the the restaurants are all ai generated for 100 percent 05:58.92 Frank Hmm. 05:59.83 James Here, what's really fascinating is they're really into AI video. like stuff. 06:04.41 Frank Scary? Okay. 06:05.18 James ah a advertisements, and they all say AI in the bottom right corner, but there's there's a lot of them. 06:07.33 Frank Right. 06:10.49 James Uh, quite a bit in train stations and things like that which i haven't really seen in the states maybe in san francisco or something 06:17.23 Frank Mm hmm. 06:17.87 James uh more i haven't really seen it in seattle but there seems to be more of that 06:19.81 Frank Wow. 06:23.11 James around and definitely more seeing But I'm not seeing Like big ads, obviously, even here in Vienna for like Claude and, you know. 06:32.09 Frank Hmph. Yeah. 06:32.71 James open ai and chat gpt and all this stuff so i don't really know how that's going necessarily But I will say. Even though I'm here in Europe. My Twitter feed is still filled of a whole bunch of stuff. And funnily enough, Frank. 06:44.32 James um Both. Burke and I have been tweeting a bunch about Luna. And I tweeted... 06:49.11 Frank Yeah. It's caught on dude. 06:50.20 James A day before. A day before GPT-6, Luna came out, and I said, Luna Max. 06:55.82 Frank Yeah. 06:56.08 James And i sweared I swear to God, I had no idea that this model was coming out. at all. But GBT 6. 07:05.75 Frank Yeah. 07:05.50 James Luna and GPT-6 Sol. No more Terra. 07:08.55 Frank So. 07:08.68 James Terra's gone, apparently. I guess. 07:10.81 Frank Who needs Earth? Earth overrated. 07:13.08 James Rep. 07:14.07 Frank That's what I hear. 07:15.14 James Those. 07:15.33 Frank planets. Blah. 07:16.37 James Both landed at like half the price or whatever. 07:20.12 Frank Yeah, it's really weird because both companies are playing the game of like our token counts are changing dramatically and we're modifying our prices. So it's just it's I take it as the money people just messing around so that people can get roughly the same amount of work done for roughly the same price. Technically, they lower their prices, but. 07:39.83 Frank If you look at how token hungry, especially on Macs, some of these models can be, it's made up for with those new trainings. It's a little bit all over the place. Honestly, it's so hard to keep up with this stuff now. I'm surprised we're even trying to keep up. Thankfully, I have you. You you keep me in the flow. 07:57.04 Frank um But also. I noticed with um GPT, like... They flat out say, like, come middle of October, we're throwing out the old models. So, like... 08:07.76 Frank update or gtfo or you know you got to keep up with us and so i used to make an argument like oh we we should keep with the old models for a long time you get to know them um you get to know their Um... pros and their cons. 08:23.32 Frank But no, that's not the world we live in. We're moving in a fast-paced world where these models change every month or two. um Yeah, and you... you I think it's our podcast, but Luna's really caught on. 08:35.97 Frank ah For good reason, it's cheap. especially luna max is performing just as well as like 08:39.36 James Super cheap. 08:43.82 Frank I forget the exact numbers, but GPT-5, 6, GPT-6, Sol at like lower medium thinking level. Luna Max is crushing it. So you're such a trendsetter, James. like luna Luna's out there. i I haven't been using it too much. I'm an idiot. I've been using Terra. 09:03.01 Frank What a dummy over here. but um it seems like the rest of the world's caught on to your little game cheap luna for everyone 09:09.44 James Jeep Luna and like, you know, you have to. there's trade-offs right when you put on a an extra higher or max reasoning and While it is crazy fast, it is going to be... 09:21.41 James chit-chatty a lot more it's going to be burning more tokens That being said, I did a demo at NDC London where I built an interactive i've done this a bunch of times i did this at um Co-pilot day, but I built I built and deployed In real time, a full... 09:38.59 James interactive. Oslo. day tour which brought in a bunch of coffee shops, sites and saunas. 09:48.52 Frank Hmph. 09:48.37 James And then you could plan your day or do a whole map. And it was like, it was an absolutely, it used Impeccable. used It used this beautiful website. I set it to Luna. if This was 5 .6 Luna at the time. Max. 09:59.62 James And I said, then you know publish the GitHub repo. set up to get a pages verify everything like click around do the thing It did all of it. It cooked the whole thing. 10:10.16 James for about six minutes. you That's including the CI, CD pipeline, six or six or seven minutes. 10:14.72 Frank Okay. 10:16.18 James 18 cents. 10:17.60 Frank Yeah. 10:19.02 James just And it was it was. 10:19.04 Frank That's pretty cool. 10:20.50 James Gorgeous, in fact, let me say I can find it for you because I think you know if you go to my github and i i'll post the the link here for you but if you go then just type in uh pi oslo i bet that'll pop up i don't know what it was it called let me see what i call it here it's called Uh... 10:38.47 Frank We can all see what baby Luna is outputting for CSS. I have CSS opinions because I have been fighting um some web development with it. And it loves to bounce between coordinate systems. 10:50.29 Frank Like sometimes it uses M's, sometimes relative M's, sometimes PX. 10:54.54 James Oh yeah. 10:55.78 Frank It seems like these models just... um They love CSS. They love generating CSS and they love being inconsistent with CSS. So I'm excited to see a little baby Luna's CSS. 11:05.74 James Yeah. And this one, honestly, is just HTML, JavaScript, CSS. It's just like, here it is. 11:11.48 Frank Yeah. 11:12.45 James And I've run this demo a bunch of times. And sometimes it does pick a frame or something. And this was on autopilot. It asked no questions, did no things. 11:19.72 Frank Hmm. 11:19.72 James And obviously I had very little input, but the whole goal is like. here it is you know it's not perfect there's ah a little bit of overlapping text here and there but it has a grid as a coordinate system it has like you know all the different places you scroll down you have like different you know different beautiful pictures you know as it's you know it's it's pretty good you can copy to a link you can like filter i mean for a one shot no big deal like it's a pretty 11:43.69 Frank Yeah. 11:44.44 James Nice looking app. Uh, 11:46.34 Frank Well. you know I know it's like it even did mobile pretty well. um It has, like you said, a little defects around mobile. but What I've really learned, and i've I've had to change my whole attitude this last week, I was getting frustrated with models creating bad UI and bad UX, as I always am. That's my perennial topic on this show. 12:08.46 Frank But I had a recent epiphany. I'm like, these models are never going to create the perfect UI for you. because I have opinions and all that stuff. So I just need to accept that it's going to do a rough draft first and then we'll refine that draft and then we'll refine and we'll refine and we'll refine. 12:25.22 Frank So I used to get upset and I'm like, you big dummy. Why are you making that text bold? That text is obviously. secondary it doesn't matter hide that away make this other thing bold you totally forgot to do this thing i told you i used to get frustrated and then i had a moment i'm like what's happening here frank you're you're getting frustrated at a bunch of a bunch of matrix multiplies. Like it is not helping the situation by you getting emotionally involved with this model producing bad code. 12:52.85 Frank So I had to do a forceful mental shift this week of. It's going to do a crappy job the first time. And I'm just going to joyfully point out all its mistakes, ideally with dictation, so I don't have to type. I'm curious how you were um interfacing with your phone if you were doing dictation or not. um And then just, you know. 13:15.44 Frank If it takes 10 iterations, it takes 10 iterations. That's just what it's going to be. um 13:19.39 James yeah 13:21.02 Frank So one-shotting is cool. It definitely makes for good tweets, one-shotting and all that stuff. But I've come around to... Using a slightly dumber model is okay because we're not doing rocket science here. 13:32.67 Frank A lot of time, it's just human stuff that I want to change about these. And that's why I've been interested in the smaller models like the Lunas and the Terras, even though Terra is not cool anymore. But I totally get where you're coming from of using these smaller models, because if I'm going to be iterating, I might as well get. quick iterations and not be spending a fortune on it. 13:53.01 James yeah it's hard like uh opening i was doing so many promotions with soul that like soul was at some point cheaper than terra which is kind of crazy and i am a big terra fan it was like one of my is basically my default model for a long time until i sort of really started experimenting with luna extra high and and max Um, But I agree with you on the UI bit. The thing that I've been doing the most. 14:14.02 James is like taking especially in my mobile apps taking so screenshots and then i mark it up And I save this, this, and this, and this, and you know, this here. 14:18.92 Frank Mm hmm. 14:21.49 James And that's really helpful. And then the one thing that I've been really interested in, it's like I use Impeccable on the web. It's kind of built into the Copilot app, but you can also just install it. It's just an NPM package, Impeccable. 14:33.60 James And it has a bunch of design guidelines and things, but it also helps you put design. best practices into sort of and dot impeccable folder. And then. 14:44.63 James with the impeccable skills it's like a whole bunch of skills that are like i'm doing design let me check and let me generate it's kind of keeping itself up to date kind of like your thing that you were talking about i think you had the styles at md or design 14:53.47 Frank Yeah. UI .md. Well... 14:55.95 James yeah ui .md yeah similar 14:57.08 Frank It's a difference, and I want to be clear. Like, I don't like... style and design and user experience being mixed together. 15:04.73 James That's a good point. 15:05.24 Frank You know, data flow, information flow, what inputs go where, all that, that's user experience. 15:05.56 James That's a good point. 15:10.94 Frank That's UX. That is, how does the app feel? 15:12.32 James Yep. 15:13.82 Frank Apple used to call this human interface guidelines. Then it became UI, but now we don't say UI anymore. We say UX. to separate it from, I don't care what color it is. I really don't. 15:24.23 Frank um you know Maybe when I'm bored, I'll work on the design and style. like That's why I don't care too much about impeccable, because to me, CSS styling is completely secondary. 15:34.45 Frank I would. If I could convince these models, that they all refuse to, I'd say, don't generate any CSS. You're not allowed to do any CSS, no styling. Let's work purely on information flow and all that stuff. 15:45.44 Frank first and then we'll get around to styling I just want to point that out because you're you're mixing the two and it just it's a little Peeve of mine. These things are terrible at UX. 15:54.89 James Yep. You are correct, Frank Krueger. Correct. you know we talked about many times on this podcast thank you for calling me out and correcting me Now let's get this merge conflict back into the correct Commit state here is that... 16:09.03 James I was just having this conversation with Heather about AI and design. 16:13.60 James And then I said, you know the thing that I talk about with Frank all the time is like the actual user experience or the taste. like There is a design taste, but there's also a flow taste as well in the user experience of how people are touching, interactive. Does it fly out? Does it do this? Does it come up from the bottom? What's the swipe interaction with? What's the button press? Where's the placement? where is that like whats how How are we moving around the application? um that's like really important even more important to be honest with you and and an AI can see a thing and it can also obviously like. 16:43.59 James Press buttons. Bye. it's not going to have that sense of is that a satisfying movement Touching the mic. 16:52.68 Frank Yeah. Yeah. um Yeah, and and and I find that that's it's the information flow and such that they tend to get wrong. All these models are honestly good enough at the design. they They love their purple to blue gradients, as I like to say. They're they're really good at generating all those gradients and all that. 17:10.91 Frank And it's fine. I let them play on that all day. What I find is they're just bad at. information flow and prioritizing items and all that stuff. But like I said, um i've I've learned to not even be frustrated with them anymore. I just sit back and regard that as Yeah, iterative. we'll We'll work on the information flow. one bit at a time. 17:30.51 James Yeah. ah Well, these new models are out. There's obviously the the Luna, the 5 Luna, and also the... Five Seoul. They are half the price of the previous. 17:39.56 Frank Yep. 17:41.10 James And also I will point out. that uh The default before for Luna was 200. Okay. on the context window now it's 272 it's gone up there so it's it's matching the other ones 17:51.86 Frank okay 17:55.49 Frank Yeah. 17:56.01 James But we also, like you said, got another we have more bottles, but we did get another model from our friends over at Anthropic. Uh, ah Claude. 5 .5 I know where that. 18:05.82 Frank Yeah. 18:06.53 James ...from Nes... 18:06.70 Frank What? 18:07.23 James Reclaude Opus 5 .5, is it? 18:08.38 Frank Opus. Opus. Yeah, it's almost a happy birthday. 18:09.44 James Opus? 18:11.04 Frank like can Do you remember Opus 4-5? I think that's when we all lost our jobs because that was the first model that actually did a good enough job where that was the big transformation last year, I think, from... 18:23.51 Frank ah definitely for me it was like Oh, here's here's some code I have in front of me. I don't feel like typing. Please modify the code to do this. to here's an app i want just write the stupid app for me and that was the big epiphany with opus four five so i think the five five was them doing a little bit of marketing and pushing and getting out um 18:34.50 James Mm hmm. 18:44.03 Frank this ah 5 .5 anniversary models, how I'm thinking of it. like We had to show that we've actually improved over. opus four or five because as we all know opus five is garbage put that put that thing down delete it we don't need that thing 19:00.11 Frank um So hopefully 5 .5 is their swan song. Oh, that's the wrong analogy. um Their apology letter to all of us for a bad opus. Because we've all been rocking Fable. Fable's been great, but... Opus 5 has been trash. um So. 19:15.52 James Interesting. 19:15.40 Frank i'm I'm excited to try maybe op oh Opus 5-5 now again. 19:20.37 James I definitely haven't had that experience with Opus 5. I feel like when I'm doing UI work, it does still do a... a pretty really good job of of different things. That being said... 19:28.24 Frank Okay. 19:29.50 James It's still really expensive, right? It's definitely the most i mean this is the cheapest opus yet and obviously it's you know 19:31.34 Frank Yeah. 19:35.92 James 25 % cheaper than the other one, which is 50 % of Fable, but still really expensive. 19:41.61 Frank Yeah. 19:41.64 James model at the end of the day, especially when you're thinking of, for example, you know luna or soul which is going to come in soul is going to come in uh let me see let me look at five soul here ah Yeah, it'd be cheaper, right, than than it. 19:57.55 James Or I guess maybe it's the same price. it's very so No, it's cheaper. 20:00.01 Frank it's so hard to judge because it really depends on how many tokens they use and how 20:04.32 James It is. 20:04.57 Frank Our pricing models are absolutely ridiculous right now. 20:05.91 James Okay. if If you were to use the default contacts on six sole, it's half the price of five. 20:14.65 Frank Yeah. 20:14.87 James um And that's not including like. you know the cash other stuff and stuff like that but yeah so there's a bunch of things in there And I bet that being sad. Um... Like... 20:25.95 James You got to remember also Fable is still a model that doesn't have that. I mean, I think they're now doing it with 5 .1. It doesn't have that zero data retention policy, which is also of interest. 20:33.77 Frank Right, yeah. 20:34.91 James you're kind of doing I think it now does in 5 .1. but i do think you're right like i think when they do the the dot five they're trying to make a statement right that okay we had five four five dot one or whatever that this is like the thing right that we're doing which is important 20:46.75 Frank Mm hmm. 20:49.85 James But the question is like, yeah, how often do you need these? like crazy super heavy models, right? That's the real question at the end of the day. 20:54.27 Frank Yeah. 20:55.21 James Especially if you're going to go on a higher reasoning. 20:55.36 Frank And I do like. 20:56.70 James I mean, those tokens are going to be crazy. 20:59.13 Frank Yeah, and I like that we've We've gotten to a place where the benchmarks and the performance graphs that people are publishing, the x-axis has become dollars now, which I really appreciate because it used to be other arbitrary things like tokens or time and all that. And in the end, it kind of doesn't matter. What we care about is dollar bills. um How much does this task cost? And I like that the benchmarks are actually judging themselves like. 21:25.15 Frank What was our success rate and how much did it cost? What was our success rate and how much did it cost? And I think it's taken us a year, but we finally settled on that's the graph everyone's interested in. Because when you make that graph, that's when you see your beloved baby Luna. 21:40.62 Frank far on the left side, away from everyone else being super cheap, but still going about halfway up that graph. vertically. And if you want to get the other vertical distance, then you got to move to the far right of the graph where everything becomes very expensive. So there's actually a weird kind of empty zone a little bit in the middle territory. And also, everyone, please pay attention to whether that x-axis is logarithmic or linear. 22:05.79 Frank A lot of people cheat and you can get really confused with how much something is costing because a lot of times they show a logarithmic one. And that makes the very expensive models look like they're only a little bit more expensive than Luna, when in fact they're 10 times, 20 times more expensive and all that. 22:20.08 James yeah 22:22.30 Frank But I like that that's where we've gotten. 22:22.49 James Yeah. 22:23.62 Frank as an industry. like here here's Here's how much it's going to cost for you to get 50 % on this benchmark. 22:30.41 James Yeah, that makes sense. I still think, you know. Pick the right model for the job. You know, I still use a lot of my code one dot one flash at the highest for bug fixes specifically. 22:36.89 Frank It's hard. 22:39.55 James I still use a lot of Luna. I still... do some deep dives into to soul and into to opus on occasion back and forth Um, 22:48.86 Frank And. 22:49.35 James But it's hard, man. 22:49.63 Frank Maybe I'm a weird programmer, but I still like to chat with these things. So I still judge them a lot on their ability to explain and talk. So their conversational ability. And so when I was making fun of Opus, a lot of it was because of that. It was just a really bad writer. 23:05.24 Frank It would just it would introduce words you don't know. It would throw away concepts that just. bad english basically it was just bad at being a chatbot which is kind of what the basic skills should be so i've i've learned if i want to actually have a good conversation i need to be talking to Soul? 23:22.02 Frank roughly something like that um some of these smarter models they might win on the benchmarks 23:24.05 James Yeah. 23:28.50 Frank But when you're trying to have it explained to you how a... bilateral filter works it's better to sometimes to have the slightly more medium models that are just better at explaining things and speaking english basically 23:41.96 James And I also want to point out. 23:42.40 Frank What's up? 23:43.45 James don't sleep on these model routing um things that we talked about recently like hydrofusion i just I think it's out in VS Code Insiders now. You can use it there. I'm not sure if it's in the app yet. 23:55.59 James But I'll point out that I took this wheel of wisdom. That's probably the most complicated thing in the world. But I wrote this big prompt. I talked to the the CLI. I turned on Hydrofusion. I said, hey. And I pointed at the .NET MAUI. 24:08.35 James hybrid web view. docs and i said hey listen 24:10.63 Frank Yeah. 24:11.61 James here's what i want you to do i want you to do blah blah blah blah blah this whole thing with the shop blah blah blah it went and it cooked dude it cooked 24:18.48 Frank Hmm. 24:18.79 James Or like... 20 minutes. 25 minutes. Came back. 24:22.56 Frank Yeah. 24:23.32 James one shot of the whole thing, like tested it, did it. It's like, I ran it. I verified it. I put it in the mobile camera, did all this stuff, blah, blah, blah. $4. the whole thing 24:31.91 Frank Awesome. yeah 24:32.75 James like it was 24:33.04 Frank Yeah. 24:34.32 James It was... It was amazing. And it was rubber ducking and back and forth. And like, that's the thing where I'm like, I'm doing a big. 24:40.85 Frank Mm hmm. 24:40.67 James a bigger thing, right? Let me do something that's going to really Hmm. duke itself out between these different models and and i thought it was so cool but frank there's a new model that isn't even a new a new model i don't have access to this model but i know that the i think that the internet may have gone i think the internet may have broke 24:53.38 Frank I do. 24:57.69 Frank Yeah. 24:57.61 James because while baby luna seems to be cheap and fast apparently there's the The cheapest... And fastest thing. in the entire world. 25:06.62 Frank oh 25:07.54 James I believe it's... 25:07.89 Frank Hmm. 25:08.23 James it's Bev? Is it Bev? 25:11.37 Frank Bav? Jab? 25:12.47 James Yeah. Stav? 25:15.33 Frank No. 25:14.93 James Jeb? 25:15.49 Frank Now it's Jev, which is... a terrible name, but whatever. We're all getting used to terrible names. um yeah it's it's you know you know the fastest way to do something is to not do it. Then it's done instantly. 25:32.76 Frank So it is a large language model. 25:32.30 James Checks out. 25:35.82 Frank But um unlike all the ones we've just been talking about and we use every day and have proven themselves useful to us, we're not using them because of marketing. We're using them because they're actually useful and we all like them. 25:47.49 Frank Jev doesn't output. text. It doesn't output code. It doesn't. Chat? So it's kind of a funny, it's still a large language model because you still input to it with unstructured natural language. 26:02.75 Frank But it's output. is not unstructured natural language. Its output is... A category, basically. It's hot or not for prompts. That's what I call it. um It's a new model out there and it took the the internet by storm. My God, there was... It was the only thing you could read about there for a couple days. 26:22.29 Frank and it is fascinating because it's a new modality for these large language models. There's a lot of things we can do with these models, other than turn them into chatbots. We've found a lot of success with chatbots and then agents. But someone thought, hey, what if I train an LLM to do something else? What if I train it to be? 26:39.70 Frank Oh, really good. classifier? What about a really good general purpose classifier? And what if we make it kind of fast and train it really well to do that? And that broke the internet, this classifier. 26:52.72 James Yeah, I watched the CJ over at the Syntax FM. ah youtube and podcast break it down and the thing that he kept saying over and over again was classifier classifier classifier classifier classifier he's like you know what's good at classifying he had some really good um demos you know ah you know he he was like hey imagine you're in a word processor and you're trying to analyze if this is 27:03.66 Frank Yeah. 27:14.84 James you know a certain tone a certain if it's ai generator or is this and you can like write these rule structures He also used it in their podcast metadata generator, which. everyone's got one apparently and and go currently syntax .fm does too but they were saying like they take 27:24.89 Frank and sometimes 27:28.11 James their recording, it goes through a a transcription model, then it goes into LLM to write all this stuff. and he said the biggest thing is like what you can do with jeff is basically write rules like it can answer questions like you give it 27:41.58 James the the data and you give it questions to answer and it can basically do a yes or no or like a categorization type of thing which allows it to check check itself right so Given, hey, the LLM output this, it can then classify. yes was it accurate was it not accurate xyz and and and then and then it it it goes down and... 28:01.68 James is classifies correctness or needs human review then they they route it in a certain way which seems interesting uh or they were using it i saw a bunch of demos for like self driverless cars and like these decision trees and all this other stuff i don't know 28:13.41 Frank Oh God. 28:15.72 James Um, someone did like an air airplane tracking price thing or something like that. I don't know, but I couldn't turn off the internet and I still didn't. get it, I guess, ah in general. 28:24.59 Frank Yeah. 28:25.34 James but Because I didn't try it. I didn't have my computer. I didn't even know. I don't even know. How do you use it? how do What do I got to do? Do I got to go buy something? 28:31.54 Frank So no, it's it's actually pretty easy to use. um Like I said, its input is still natural language. So you can give it all the information in the world. So you can you know tell it a story or ask it a question or something like that. 28:44.09 Frank But what you need to do is specify its output schema, kind of. It's not even a schema, but basically given a question or something, some information I want, here are the potential answers for that question. 28:49.28 James And... 28:56.62 Frank So you can't have it like calculate a number. But if there's 20 possibilities and you wanted to pick from one of those 20 possibilities, you just enumerate those 20 possibilities. 29:06.89 James Yeah. 29:07.07 Frank And then it will pick one of those possibilities and give you a confidence score. But that's a trivial thing for a tablet because all these networks are probabilistic. They all have probability scores. They just don't show us the probability score. 29:18.77 James Yeah. 29:20.27 Frank But they all have it. That's how that's how they work. Um. And it's it's clever because let's say I give it a whole bunch of text and I have 20 questions. 29:30.50 Frank Well. We all know there there's basically two parts to using these LLMs. First, we do what's... annoyingly called the pre-fill, where it's taking all the context and feeding all that information into the LLM. um And then the next half in a normal one is the decoding stage where it starts generating more text and all that stuff. 29:52.02 Frank Well, it turns out when you're doing that input thing, it's already doing predictions. And so it's already predicting the next token. That's how all these things work. And so Jeff is just kind of clever in that it's always. 30:04.12 Frank Kind of predicting just the next token is going to be one of those. answers. It's a discrete set. it's one out of two answers that's your true false kind of stuff it's one out of 20 if you say here's 20 potential answers to this question pick which one of these is best um And it can be a rating even. So like, you know, um it's just not just hot or not, it's how hot. So is it cold, medium, hot, you know, so it could be three levels like that. 30:32.88 Frank And because all these models are just always outputting the next token, this one is just always outputting the next best. guess. And if you have 20 questions for it... 30:44.06 Frank You don't have to do that pre-fill step. 20 times. It can just kind of. fork the model at that last little token and then um very cheaply answer the 20 questions. 30:56.72 Frank vaguely in parallel. um so that's why everyone likes it because of that inexpensive nature of it all you're paying for is those input tokens which as we all know are kind of the cheaper ones prayer. 31:09.50 Frank reasons. it's doesn't actually make that much sense but that's just how the world has evolved the markets have evolved But the nice thing is, if I have 20 questions for one set of inputs... 31:20.43 Frank it's pretty dang cheap because it's only really generating 20ish tokens then as output tokens and it can do all those in parallel so it's also fast So that's like at a low level how it's working. 31:32.66 Frank Bye. Pretty cool. I mean, it's just a clever way to use these models that we've had for a long time. 31:40.68 James So I wouldn't, for example... Like... Let's say I wrote a blog post. 31:46.86 Frank Mm hmm. 31:46.89 James I wouldn't take this blog post and then I wouldn't say, hey, go give me five spicy titles. 31:53.00 Frank No. 31:53.24 James that you would come up with them. But... If I said, hey, my blog has... 50 different categories. given this text. Go. figure out. 32:04.40 James what categories it falls into, it could do that. 32:07.70 Frank Yes, absolutely. Or you can... Go for the hybrid approach. Have one network, Luna, generate 20 different titles for the blog posts and then have this pick between them. 32:18.11 Frank Because maybe it's a bigger, smarter model. Maybe it has better judgment. Maybe it has better taste because it's been refined to have better judgment, better taste. but it's not generative in the normal sense. 32:27.95 James Oh. 32:28.38 Frank So you could do a hybrid trick then. 32:28.50 James And. 32:30.16 Frank and 32:30.19 James And then you could write specific questions about how you want your titles. to be? Is it click baity is it blah blah blah is it whatever and is it can answer those yes or no and i could go into this algorithm to get there now that's kind of cool and i think that's what cj maybe was saying where he's like okay this is where the lm is generating their podcast metadata and then we use this to like 32:36.48 Frank mm-hmm 32:38.88 Frank Yeah. 32:50.75 James And then they come up with the ideas. 32:51.05 Frank Shoots 32:52.28 James That's really neat, actually. 32:54.85 Frank Yeah, because it's not spending all of its time generating text. 32:54.53 James Um, 32:58.79 Frank they can make the model. Really? big and powerful and smart. And. tuned to this capability of choosing between multiple options really well. Whereas the current models are tuned with with the Right. 33:13.69 Frank reinforcement learning, the human feedback reinforcement learning that we all use. They're tuned to be chatbots and agents and all that. 33:21.15 Frank They're not tuned to make decisions necessarily. they're They're obviously making plenty of decisions, but they're not tuned to that. This one can be tuned to that exact problem. And so it's just smarter. it's It's just a more efficient use of these models. And you're guaranteed the correct schema output because it's not like we're parsing JSON in the end, even though it's all presented as JSON or not. It's really just generating. one option out of. x number of options its efficiency. 33:46.70 James That's cool. 33:47.99 Frank Yeah, it's... You know, when when Transformers were first invented with the paper, um attention is all you need. the LLMs had. were two big models. There was the encoder and the decoder. 34:01.57 Frank And then a couple years later we decided, you know, we can simplify this whole and caboodle. to just the decoder. So now that's all we use today. That's happened with GPT. So we took the transformer model. We threw out a good half of it because we said, you know what? That part's actually just extra complication. We don't actually need that. And look at all this amazing stuff we can do with just the decoder side. I like Jev because it's saying, 34:26.18 Frank Yeah, that's cool too, but you know, we don't actually need to be generative all day long. We can chop off um the the recurrent part of this whole thing and just output something. an answer. 34:37.94 Frank And so we're just we're refining. It's still the same thing. It's still a transformer. It's still an LLM. um It's using all the same architectural principles. People have made junky versions of this already. 34:48.67 Frank It's the easy enough. 34:49.01 James this 34:49.82 Frank Any idiot can train a network to do what this is doing. Hard to make a good network, obviously. Okay. um But I like it because it's just... Another. version of the same thing. Like this architecture, this transformer architecture that was invented. has way more potential than just what we're doing with it today. 35:07.37 Frank Chatbots became a thing. First predictive text became a thing. That was a little boring. Then we got chatbots. Those were great. Then we turned chatbots into agents. Fantastic. And now this is just like, hey, here's another thing we can do with these LLMs. And it's efficient. We're not paying for all these output tokens. 35:23.60 James Yeah, it's smart. uh yeah that makes a lot of sense have you used it for anything yet have you tried it out 35:29.54 Frank I definitely got access to it, thankfully. My little brain wasn't able to come up with any great scenarios for it, so I was asking it the dumbest questions. Of course, I fed it a bunch of stock market data, and I'm like, hey. 35:42.17 Frank Which one of these stocks should I buy? Which one should I sell? Which one should I hold? Do not use it for that. it's a trap um but you can you know because you know buy sell um whatever hold you know those are three options you can ask any network for those um Other people were doing a lot more clever things, but that's always my stress test. I like to just see what it has to say. So I would say like. Um, 36:05.32 Frank uh i would make up update i'm like here is the microsoft stock is like crashing should i buy it and it's like 33 chance you should buy it okay it's crashing but they're like yeah but it's microsoft it can't say that because it's not generative but you know that's what it's thinking it's like it knows what microsoft is so even even though i gave it false data of microsoft crashing it's still like 36:18.35 James Yeah. 36:21.94 James Yeah. 36:25.71 Frank Maybe you should still think about holding it. 36:27.92 James Yeah. 36:28.66 Frank And so it was fun to just like do those kind of little stress tests and everything of it. But I think about other things like um people had it playing Doom, which is. really hard it's a fast model but it's not that fast because it still has to take input tokens so you still have to translate um 36:44.97 Frank images and visual things and other media into text because it's predominantly a text model other than people are working on multimedia versions is where you can give it a picture and have an answer. you know, multiple choice questions about a picture. 36:59.90 Frank um It's... There's a lot. to play with it. And again, it's just it's just another use of a transformer. We've had classifier transformers in the past. 37:10.62 Frank um We've all used them, especially in the early days when we were using much smaller transformers. So this is kind of honestly just revisiting an older idea of these classifiers that we all used to use and just making. 37:24.36 Frank Hey, for the last... three years two three years we've really learned how to scale up these llms and make them huge and intelligent and we've learned this trick um Human feedback reinforcement learning. Let's apply that to... classification also. And I don't want to be reductive by keep calling it a classifier, but that is what it is. Fundamentally, at a machine learning level, it's just a big general purpose classifier. 37:49.13 Frank But the neat thing is, it's a big smart model that's been tuned to be good at classification, and that's why it's useful. 37:55.88 James specialization as I like to call it. 37:57.19 Frank Yeah. 37:58.03 James yeah 37:58.22 Frank Exactly. Yeah. 37:59.39 James Yeah. 37:59.47 Frank So I don't know. 37:59.99 James well Very cool. 38:00.64 Frank Did it deserve to take over the internet by storm? No, that's the bubble. what Generative LLMs have proven themselves. 38:05.72 James Yeah. Yeah. 38:09.53 Frank They're fun to chat with. um They help us in our day to day work. They help you write. weird apps while you're on vacation. they've proven themselves This one is an old idea given a modern twist. 38:25.18 Frank um And it took over the internet because we're in a bubble. That's fine. but it is But that shouldn't detract from it either. It is a good idea. It is a more efficient use of all these things, and it is nice to see a modern version of that. 38:39.08 James and maybe because of the the cost and the very specific use case there are applications today that might be doing some things like think of wordpress right how this can maybe be able to infuse a bunch of little things and give you all these recommendations and all these little classifications for your blogs or for your youtube for example like your videos and all these other things that are doing i think i saw someone 38:51.33 Frank Mm hmm. 38:58.28 James they They wrote a plug-in I think for Twitter that was like a classifier of all the tweets and it would automatically hide the tweets or whatever like in real time that are maybe biased or or spicy or whatever so stuff like that but i could imagine a different use case for even some of the apps I'm talking about, the podcast generator and some of these other things that maybe I'm using an LLM like you're saying, but I'm also using this to classify. 39:03.73 Frank Yeah. Right. 39:08.34 Frank Yeah. 39:19.39 James um in correlations that might be a good trial run to do so very cool i'm make excited to give it a try 39:22.87 Frank Yeah, and you brought up... Yeah, and you brought up probably the most useful use case that is probably going to be used for at least in the immediate future is if you're using something like open router and you have a choice between 12 different models to use to actually answer the question I need a classifier. 39:38.03 Frank Choose a model for me. 39:38.27 James Yep. 39:38.97 Frank you know why Why pay for an expensive model to do that classification? Might as well pay for a cheap Jev. to choose. which model should actually do the dirty work. 39:48.60 James Yeah, I'll be interested if Jeb takes over the router world. So very cool. 39:51.64 Frank Yeah, basically. yeah 39:52.46 James all right well i'm gonna go hang out with my wife thank you so much frank for Give me the lowdown on Jeff. I did see it all over the feed, but I said by the time I get back. there'll be something new but frank's gonna tell me all about it and then i got a bunch of new models to talk about so i was like wow just like crazy never stopped man um 40:05.32 Frank yeah Yeah. No. 40:10.31 James Alright, well go play with all those new toys and set that Luna to max and have just give it a rip it's it's it's a good time it's it's so cheap it's just it's it's awesome 40:16.87 Frank Yeah. 40:19.08 James Um, 40:19.17 Frank Yeah. 40:20.27 James all right that's gonna do for this week's merch conflict so until next week i'm james montemagno 40:24.27 Frank And I'm Frank Krueger. Thanks for watching and listening. 40:27.42 James Peace.