Misko Hevery November `23
===

[00:00:00] Hi, and welcome to PodRocket, a web development podcast brought to you by LogRocket. LogRocket helps software teams improve user experience with session replay, air tracking, and product analytics. Try it free at logrocket. com. I'm Tejas Kumar, and today I'm joined by Mishko Hevery, one of the all time greats in the industry, I should say.

The inventor of AngularJS currently building. An entirely new framework for user interfaces over at Build. io called QUIC. Pioneered the principle and technique of resumability. I mean, Misko's resume is pretty ridiculous, but Misko, it's a real honor to have you back on the podcast. I would be very surprised if listeners didn't know who you are, but for the sake of inclusiveness.

Would you mind introducing yourself to the listeners today?

hi. Hi, so I'm Mishko. So I did this thing called AngularJS and Angular. And now I'm working on QUIC at Builder. io. Builder. io is a headless visual CMS. And I'm just driven about making the web faster. And so that's, what's all it's all about.

I like how you [00:01:00] delineated AngularJS and Angular. 

I think there are two separate frameworks.

I would actually, I think you and everybody else, because there was no migration path. There was no anyway, but we're not here to talk about that today. I want to zero in on our topic and that topic is something that I recently played with.

Um, Let's give context for everyone listening about how this episode even came to be. Mishko and I were in Italy at a conference called CodeMotion in Milan, and I was building my developer relations agency's website. And Mishko was like, dude, just use Builder. And I said, what do you mean? And he showed me something that Steve, the CEO of Builder and Yoav, one of the dev advocates at Builder also showed me, which is Builder AI.

So it's this thing where you take a Figma and you click a button and it becomes a Deployed website on Builder. io I was impressed by the experience because it I had like interactive canvas elements and stuff going on, and it didn't create them in canvas, but it created it made them images. I think if I remember correctly, and there was a code inspector where you can [00:02:00] prompt it to redo portions of your site, and it was really interactive.

So I'd love it if we could dive into that a little bit today and talk about details for starters. Thank you Let's maybe talk about Builder. io itself, because this was my first exposure with Builder. I'd never used Builder before, and as far as I can tell, it's a smart website builder, similar to Wix or Squarespace.

Am I accurate with that?

Yeah. So it's a good way to think about it. And I always tell people that it's like Wix, but there's a one big difference, right? And so with Wix, Wix has to be hosted on Wix. So basically Wix owns all the infrastructure, et cetera. And because it's hosted on Wix, that means you're limited to the kind of code you can ship.

To the client. For example, if you have a site and you wanna hook it up with how many items you have in the stock, so they can tell the user, you know, like, Hey, you can buy this item, but only three are left in stock. Doing something like that in Wix would be hard because, it's not your site, you don't own it.

So the big difference is with Builder is that instead of being like Wix where you go into the site and you kind of own it by them, you NPM install builder io into your existing [00:03:00] application. And because of your NPM installing, you are the owner of the application. You decide where the application is deployed, what servers or technology, whether you're running GreenGear, React, Angular, Vue, Svelte, et cetera. And so you can incorporate Wix like editor. into your website. And because it's in your website, it has one huge difference. And that is that your creative folks who are using the builder editor for, drag and dropping things onto the page, they can also drag and drop existing components from your application into the UI, right?

So you, as a developer you create a new great widget that does the right thing for what you want. And you spend a lot of time and effort on it because it talks to the back end of the database and shows the latest who knows what, right? And as a creative folks, you can, once you registered with Builder, you can drag and drop the component onto the page.

You can set its inputs, what does it talk to, what particular widget do you want to show off or something like that. And so it opens up a whole new world of things that is not possible with other [00:04:00] headless CMS systems.

Interesting. This reminds me a bit of Headless CMS. I recently was... Exposed to call it react bricks, but this is different because it's fully local. Is that correct? I didn't know you could just npm install builder and you get a ui for a drag and drop. 

So the \ part that you end payment install is in your machine, but builder stores, the database in the cloud, right? So it's not fully local in that sense. And like the database for what you drag and drop is in the cloud and it's a SAS in that sense, but it is the part that, renders things for you into your existing application.

That is local.

interesting. Wow this is honestly this i've seen builder io a lot. You have a very comprehensive blog I recently saw I think steve or you wrote a blog post about how to use ai appropriately I had never I this is the first time i've ever heard that's what builder is You can npm install a drag and drop ui That is on top of a specialized database that's hosted in Builder and, give yourself superpowers, really.

That's incredible. How then [00:05:00] does just so I understand correctly, is it, you mentioned, Svelte, Solid, Quick, whatever you want, is it framework agnostic as well? How does it

Yeah, so that's a good question. No, it's not framework agonistic. It's very much tied into the framework. So we have a separate SDK for each framework. And I know what you're thinking. That's a lot of work. Except we have this other thing called mitosis. So we write our SDK once and use mitosis to generate a specific SDK for each of the frameworks available out there.

And that's why we cover the gamut.

That's incredible. Can you then use mitosis to also, I don't know why anyone would want to do this, but to describe UI? Agnostically and then generate like multiple versions of the same app with different UI

Yeah, that's exactly what it is for. And the use case for that is imagine you're a company and you have lots and lots of departments and each department happens to choose a different technology stack for their application, but you want to make sure that they have a consistent look and feel consistent set of widgets, consistent design system across all of the apps, right?

So [00:06:00] you would build your design system in Mitosis. And then mitosis would spit out whatever technology you want, and then you give those things to the individual teams.

That's such an interesting take. So I've heard this problem being solved in the past with web components. I know Vonage, for example, they do similar things where they have an entire design system built with web components as one level under the abstraction. And then they generate, more specific behavior.

based on their UI library using the web components under the hood. , what is the interplay then between a mitosis based approach and a web components based

Yeah. So Web Components wraps the web component inside of either React or Angular or Swelt, et cetera. And the problem with that particular approach is that first of all, you end up with two frameworks inside of your final app, right? You end up with the Web Components framework, and then you end up with your, React, Angular, Swelt, or whatever technology you happen to have.

So that's one thing to look out for. The other thing is. Because you're doing wrapping the two technologies don't seamlessly work together, which means like the web component just [00:07:00] renders whenever it feels like, and your react app has its specific set of reasons under which it renders. And so they're not in sync with it.

And so there's many things like that, that make the web component solution. I mean, It works. But it's not an ideal thing. And this is why something like mitosis actually I think it solves the problem much better because what you're getting is not a web component that's wrapped inside of a React, Svelte, Angular, whatever package.

It is an underlying thing. And let me show you a place where this actually makes a huge difference. For example, web components and SSR don't really go to. The other place where it doesn't really work is web components, force hydration. So if you want to use web components inside of quick, you can, but you're just undoing all of the hard work that you've done by using something like quick so that you don't eagerly have to execute any kind of code to make the application interactive.

Interesting. Those are solid points. You did mention a web components framework. As [00:08:00] far as I know, they're browser native. So why would one use a web component?

it's not right because uh, web browser native gives you a custom element and it gives you a shadow DOM. anD end of the day, the web component probably needs to use some kind of a templating technologies. Let's say you choose lit. Lit isn't built into the browser. It's a JavaScript library written like any other JavaScript library, which you have to install and et cetera.

And you don't have to use lit. You could use other technologies, right? Or web component could actually be written in react. Who knows? And so, you know, people like to say Oh, it's just the browser. That's just the technology, but there's more to it than that. Like what browser just gives you is just a custom element.

And a shadow DOM and that in itself is not a component framework.

That's super helpful. Just, I'm curious if we could expand on the Shadow DOM piece for people who are listening. I know the audience here is a ton of people who work with libraries, so react well to solid, et cetera. And this question of why aren't web components enough keeps coming up what is the shadow down?[00:09:00] 

How does it work and what are its. limitations that keep us from having framework style behavior natively.

basically a DOM that is not in your normal DOM tree, but instead of like in a alternate universe. And the reason for that, the reasoning is good. The reasoning is like, You know, if I have a component such as, for example, input element, the input element internally has a bunch of other things to make it do its thing, but you don't want to see that as a user.

I just want to see an input element and be done with it, right? Maybe date picker is a better example, right? If you say input element, it's a date picker. There's a complex UI that. That comes up when you click on this particular thing, you don't want to see that complex UI in there. So that's one reason.

The second reason is when you do styling, you want to just style your component and you don't want the styles to leak outside of it. So styling is a form of capturing that to make sure that styles don't leak out. But this also creates a problem for SSR. There is no way to produce HTML that internally has.

Oh, this [00:10:00] is, this goes into the shadow DOM. There are approaches and people are starting to solve this particular problem, but it's not solved yet. And the other issue is that if you have a web component, typically, realize it. Not typically. Heavily relies on DOM. And so now if you want to do server side rendering, you have to run DOM on a server.

And if you want to be performant and streaming based framework, then you cannot rely on DOM, right? You want to start and concatenate strings into the output stream. And you can't really do that with web components. So all of those things make it tricky to do. It's possible, but it's not ideal.

And when there's off the shelf solutions made for that specific purpose, things like quick, why would you even let's get back to builder. So I can NPM install builder and I get this drag and drop UI that is somehow connected to the builder database in the cloud. I'm assuming through some type of token, acts API key or something like this. Do I get builder AI locally as well?

no. And all of that is in the client in the cloud. Actually, even the editor [00:11:00] itself is in the cloud. The thing you're installing is just the glue code to make it work with your application. And the way it works is that the builder website, the builder editor loads your Website inside of the iframe.

And then the builder is talking through the iframe to the underlying component and allows you to do drag and drop and editing, et cetera.

And the builder editor that runs on builder. io \ , that accesses my local website, how, like through local host or 

no, through iframe. So the iframe sends post messages, et cetera.

well, Yes, I know that, but is the iframe destination, my local, like it's my website running on my machine, npm

Yeah. So it could be anything that you can access through your machine, including localhost, right? So that's the nice thing about it is you open up a builder editor inside of the iframe, you open up your localhost, which is your development environment. And as the development environment is doing hot module reloading, and it's you're editing things, you actually see hot module reloading actually happening inside of your builder IO editor.[00:12:00] 

that is very cool, man, I really, I feel like an urge to play with it, y'all need to market this, I think, more,

We're trying. This is why we're here.

Like yeah, this is, I'm glad we're here for this, but like your blog posts are cool, but I don't actually see the product in them, and this may be feedback, but anyway, I'm glad we're getting to it here okay so this

Sorry. I'm going to go on a tangent here. The trouble is if I start making a blog post about how cool the CMS system is, nobody cares, right? So what you do is you talk about how cool, whatever you care happens to be about, which is either performance, the websites, SSR, whatever you have. And every once in a while, you're like, gee, I wonder what this people do.

And so you. Click on the builder IO and you're like, oh, they have this cool, visual CMS system. I maybe I can use it

Yeah, I guess the part that I haven't seen yet is the differentiation from things like, React Bricks and Squarespace, et cetera but I guess that's what we're doing that's amazing, how what, how would one register a component? With Builder. io, and I ask this because I've worked with a number [00:13:00] of headless CMSs where, aw man saying this component, is, like registering it with a foreign system requires some type of manifest and like a complicated JSON file, I never enjoyed that, not once with any headless CMS, and you can use your imagination to which ones I'm talking about, but man, how was that process on Builder?

Yeah. No, it's super nice. So because the SDK, but let me just explain why, because the SDK is specific to the particular technology. Let's pick on react. So because you're using a react SDK, the SDK already knows that it's going to be getting a react component. So you're not asked to create some other third party, unknown generic component.

You're using your existing component as is. And just letting the the framework the builder know about this. And the only thing you have to provide is a little teeny tiny JSON. It basically says, what is the name of this component? Because that cannot be extracted manually. What are the inputs of, for this particular, but basically what are the props for this component?[00:14:00] 

What do you want to give them a, human readable names and if they have any defaults. And that is all, that's all you have to provide. So little tiny, object literal that says. This is a reference to the component function and it's called George and it has props called first name, last name, and that's it, you're done.

Is there no maybe this is a very silly question. I apologize, but is there no way that you can have a static code analyzer service slash 

We 

We have that, it's called the dev tools and it will go and statically analyze your application and gives you a nice UI that allows you to pick from it and just say, I want to make this, and this one registered. And then the static analyzer tries to guess, what should the name be and what should the inputs are, what the props are and what the default value should be.

It makes a reasonable guesses. And then it generates some code for you that is inserted into your code base. And then you're free to go and modify that code.

interesting. Okay. So I don't even have to write these [00:15:00] connectors for my components to build. Or I would just, there's 

Yes, that's correct. But I just want to stress that like the simplest possible connector is just a reference to the component and a object literal that says name colon human readable name. That's it. That's the simplest possible connector.

I see something in Builder. io's developer experience story that could work really well is a VS Code extension as well. Like when you open a React component in a Builder project, there's like a little button, turn this into a fucking, oh, excuse me, I can't swear, you'll edit that out.

Turn this into a component on Builder. Yeah, I think that's really the name of the game, is making things as low friction as possible, CMSs that I've used, it's very clear that they aren't so much for developers. building for developers as much as they are for developers building for like content people.

And I personally, I'll go on record to say I've had a very hard time with most headless CMSs and part of me longs for the like WordPress with WP API, the plugin, cause that was perfect. Like I'm in charge of the API and there's other people to handle the visuals [00:16:00] and it, you can handle the visuals in WordPress itself

Yeah but I want to make a point out of this, which is that headless CMSs are usually not visual, whereas hosted CMSs usually are visual. And the reason for that is because if you're headless, you have no idea what underlying technology you'll be having to use it. And the way we solve this particular problem is that we generate an SDK for every single underlying technology, which means that's a lot of work that nobody wants to do, but we do it through mitosis.

Okay so Builder has these innovations, you've got the editor, you've got Builder AI, all of this runs on the cloud, you've got Mitosis, which we've talked about, and then there's PartyTown, not to be confused with PartyKit 

could you talk a little bit about what PartyTown is, why people should use it, and

Yeah. So PartyTown is basically a way of running third party code, like Google analytics, Facebook pixels, and whatever else you happen to have on your website in the web worker. So that the main thread for the main application is free, [00:17:00] allowing for better startup performance.

So it's basically bringing multi threading into your JavaScript application.

Okay it already has multi threading if you're using workers, or even service worker as

Well, Yes, except that , all the third party code has to be on the main thread. And so what party technology to do is take advantage of the fact that you can have multi threading, but nobody takes advantage of it because well, it's kind of hard to do.

Can we speak to the mechanism of that? And we're not going to, look, we have Mishko podcast. We're probably not going to talk about QUIC because there was a previous PodRocket episode about QUIC. There's talks about QUIC. I'd encourage whoever's listening to go explore all of that. And if you want another QUIC episode, we'll do it.

I'm more interested in the Builder. io adjacent side of the world. Party town. It is hard to do because these scripts literally need access to the DOM, right? Like a page view event, a click event, all these things that you send are DOM events. So how, can we speak a little bit to the mechanism of how 

PartyTown accompanies? 

So basically you take the window object in the main thread and you copy it over [00:18:00] to a web worker, right? Now, of course you're scratching your head, like you can't do that. Right? So what we do is we create a proxy on a web worker so that , if you say like window dot at event listener inside of the web worker, we proxy that call over to the main thread.

Now, of course, you can see, imagine that there's a lot of complexities involved. And the biggest complexity is the fact that the web worker and the main thread talk through an asynchronous communication. If you do post messages, asynchronous. And because it's asynchronous, that doesn't work because many API are fully synchronous , on a window API or document API, et cetera.

So the whole trick of party town is to figure out how to talk between the main thread and the service worker thread in a synchronous manner.

So that's the hard part we figured out. And on top of the synchronous communication channel, we are able to basically proxy all of the calls from the worker thread to the main thread. And the result is that we can now run a third party code in the web worker. Even though the third party code is thinking that it's running in the main thread [00:19:00] and it's accessing API on the main thread as if they were fully synchronous.

When you say proxy you don't mean the proxy type in JavaScript. You mean you're

Oh, no that's actually how it's implemented using the proxy type.

Okay. So let's demystify this a little bit because I don't see it yet and I'd like to, and I think the listeners are there with me. So a proxy in JavaScript is you define, getters and setters that intercept access on objects more or less, right?

So do you then put a proxy on the entire window object? Such that when addEventListener is accessed, you copy its arguments and things to a WebWorker. This is where the question is. How

So it's more like the other way around. So on a main thread, you look at all of the properties that are, that exist on window and document, et cetera. And basically you learn about the API surface of the browser, and then you ship this over to a WebWorker and then the WebWorker creates a bunch of proxy objects on the WebWorker side that mimic.

The main thread. So on the web worker side, if you say window. [00:20:00] addEventListener, the proxy on the web worker can then talk to the main thread and say, would you on my behalf call addEventListener with these arguments so that I can, create what the web worker is trying to do.

, You create a proxy inside the worker that implements addEventListener on the window in terms of

function 

Yes. So window doesn't exist in a worker, right? So you create all of the objects that exist in the main thread in the web worker thread, and you create them through a proxy so that you can intercept any access to these objects.

right. So then when something accesses add event list So in a foreign script Accesses at event listener inside a worker you say wait, I'm gonna intercept this and post that message to the browser

Right.

Thank you, Mishko. This is so man. I just it's a very cathartic for me.

So okay you do that Now let's put some more nuance to this async sync discussion because when you post that message to the browser, that's an asynchronous communication. What's

the complexity?

Well, iT doesn't work because let's say worker thread [00:21:00] calls window. name. It says what's the name of this iframe? Right. It says And it expects to get the value synchronously. Right. So you can intercept it. You know that it's calling, but like, how do you synchronously get this value? And so you cannot do it through postMessage because postMessage is asynchronous. But it turns out that if you're clever enough and you have enough insane hackery, you can work around this particular problem. And the way you do that is that you use XMLHttpRequest, and you make a request to a URL and HTTP, XMLHttpRequest for whatever crazy reason has a flag that says make this synchronous. Okay. So you make the request to a URL, and of course you don't want to make a request to a URL. That's not going to work. So instead what you do is you have a service worker that intercepts the URL and says okay, okay, I got you. So let me now use the URL information you posted. To then talk to the main thread through a synchronous means in order to figure out what the name is of this particular iframe and then send it back.

[00:22:00] And all meantime, the WebWorker thread is fully blocked because it's waiting for this particular value, right? But because the WebWorker thread has no UI, it's not an issue because it's fully blocked because like, so what it's blocked, I mean, that's kind of the point. Right.

Wow. , \ you use the word hackery, not me. And that sounds like some bona fide hackery. Oh my gosh. Is there overlap between this? And some of quicks mechanisms. And the reason I ask is because what you're doing when you proxy, something like add event listener, which is a function on window is you're capturing the functions arguments.

And I assume for posting message, you need to serialize the arguments and there is that serialization boundary. And I think serialization boundary. Okay. I think quick and resumability is there overlap here? Or am I just 

like. 

Yes in, in that low level, you have to serialize stuff but the real overlap is somewhere else, which is that. What both quick and party town are trying to solve is the problem of javascript, right? And the fundamental problem is you have too much javascript on the main thread, right?

You want to have a [00:23:00] performant application. It's easy in a sense that Don't ship JavaScript. But it's also hard in a sense is how do you expect me to build anything useful if you don't allow me to ship JavaScript and so what both quick and party time are trying to solve is the fundamental problem of like, how do I not overwhelm the browser?

It's main thread with too much JavaScript, right? How do I move JavaScript off of the main thread? And if you move JavaScript off the main thread, both in terms of third party code in terms of party town, and also first party code in terms of quick, the result is that you have a fast website.

And not just a regular fast website, a blazing fast website. There's two ways we can take the discussion. I think we've talked about QUIC and we could get some somewhat deep there, some more details, or we could talk more about builder AI, because that is the reason we're here. For those listening who haven't used Builder. ai, here's how it works. You have a visual view and I know this because I've used it in Milan with Misko. Is it public yet, or is it

Yeah. It's called a visual copilot.

Okay, [00:24:00] excellent. How it works is you open this visual editor. Think about any visual editor, like Wix or something. But, of course, it's Builder. io. And you have... Your website that you're visually building, but you also have access to the code of it, like the elements panel on your dev tools.

And that has a text field, enter a prompt. And make stuff happen and you do, you say, change this div into an aside element or added details in summary, whatever it may be, and it just magically works. And the cool thing, I think my wow moment with this was, I did see the code update, which you get with Copilot in VS Code, but then I saw the website update as well, in, in real time.

And that was Somewhat mind blowing. So, Can you speak to maybe top contour, how that's built? I think I say top contour, but with a little bit more detail than, oh yeah, we just call chat, we call it, we call GPT and stuff happens. And then we could maybe fine tune the level of granularity.

So I think there's two things you could talk about here. One is the AI part, right? But the second part is the visual copilot, which is something slightly different. It's using AI, but it's not the same thing as the AI you just described, [00:25:00] where you just type in a query into the builder and something happens. So the type in the query into builder is really just that, right? Is that like we take the content of the page and because we have mitosis, we can translate it into other languages, right? Other formats. And we translate it to a format that is understood by chat GPT. And then we just basically say, Hey, could you like change this? And then chat GPT responds with something and it responds something very relatively simple. And then we use mitosis to cross correlate everything back and put it back to where it was. But , there is limited amount of things you can do by querying the chat GPT.

Basically the basic problem is let's say marketing team says I want to make a new campaign. You can't just go to chat GPT and be like, make me a new campaign. It doesn't work. Because you really want to have Somebody who's graphical, and this is where we're getting to the territory of visual co pilot.

You want somebody who is a visual person. That's not me, by the way, I cannot for the life of me make anything look pretty. And you want them to design your next website. And when people design [00:26:00] they don't want to be encumbered by like divs and spans and styles or whatever, right?

They just want to have a freedom to do whatever you want.

And so you want to use a tool such as Figma or Adobe Illustrator, and this tool basically says, do whatever you want, like there are no constraints, and this is great for designing a new experience visual people that as I said, I'm not one of them.

But anyways, they create a nice experience inside of Figma or Illustrator. And the next step you want to do is say, convert this over to actual HTML. And this is where it's usually handed over to an engineer. And the engineer has to go through and like, okay, so like, how do I make divs about this? So that looks like a row and it looks like a column.

And, how do I make this responsive? And, they put all these images together and the images overlap. So really I have to convict. Make a one big image that contains it or something like there's a lot of decisions you as an engineer have to do to convert a Figma design into an actual page\ and so this is where visual copilot comes in. [00:27:00] And the idea is that the visual copilot can look at the Figma. Output and, subdivide it, break it into columns and rows and figure out how to best size the images and how to deal with the images. The fact that they might overlap and all that stuff and extract all of the relevant information and convert it into essentially HTML. And that's a huge thing because companies spend a lot of time converting. Sigma designs into, whatever output you want to have, whether it's a landing page or you want to change the look and feel of your website or whatever, right? That's a huge amount of time that's spent on this.

And so if you can have a tool that automates this process, that's a huge time saver.

Yeah, you know, this is probably a testament to how good the product is. The visual copilot, , so Steve Jobs would say, products that are really good, you don't even notice they get out of the way. And I legitimately did not think about that at all. Like I clicked, I remember my experience in Figma, I clicked build this in with Builder and it just appeared in Builder.

io. And for [00:28:00] whatever reason, It was just completely lost on me that, wait a second, this thing took a Figma file and turned it into Markup. What? And only now it's registering that happened.

Yes. So the hard part is how do you take a Figma file, which is absolute positioning and has no concept of grouping or columns or rows and turn it into an HTML that very much wants to have rows, columns, margins, paddings. And more importantly, how do you do it in a way that is responsive, right?

Because there are different size screens that we have, and we have to make sure that we support them all. And so the visual copilot does this particular thing, both a combination, it's a combination of heuristics. It's a combination of AI that, and it's a combination of model that we trained ourselves. And it's a combination of chat GPT.

It's that all of these three things cooperate together to give you the seamless experience. , 

let's talk about a number of those things starting with the Figma format itself. I'm assuming that's your input, right?

[00:29:00] You get a Figma file. What does a Figma file even look like on the inside? Is it just HTML? 

No,

No, it's not HTML, right? It's its own thing. I'm assuming it's XML. But I think the important thing to understand is that the XML you get out of Figma does not have structure, right? It is just absolute positioning of whatever you want, whether it's text or whatever, it's all absolutely positioned.

By absolute, you mean within the plane of the Figma editor itself, which is pretty massive because you can scroll. Oh my gosh.

Correct. 

So

when you converting this into an HTML. You have to understand Oh, this text is about this image. I think what they really meant is that there's a div around it so that the text and image go together. This is the intent that the designer had when they centered the text over that image, right?

groupings play into this somewhat? Because typically Figma designers will design like a group, a box,

Yeah, so there are groupings to some degree, and I think we use that as an input of things. But you can [00:30:00] design a lot of things in Figma without doing grouping. The problem with grouping is that you can't really go back to a visual person and be like, Make sure you have all these groups correctly done, otherwise it won't translate correctly, right?

It's that's not a thing. You have to explain like rows and columns and like, why, grouping this way versus that way when we can't none of that stuff happens, right? These are not programmers. They don't understand the constraints of margins and paddings and CSS layout and.

responsiveness, right? They just want to lay things out. What looks good to them visually and push a button that says, okay, do your magic. And so the magic is about looking at this unstructured content and figuring out and extracting basically structure through a set of heuristics and custom build AI systems and something like chat GPT.

okay, so three pillars, heuristic, you've mentioned this twice now, heuristics, custom machine learning models, and OpenAI's GPD, you know, 3. 5, turbo, 4, whatever it is. Let's talk about those heuristics. Can you dive a little bit into what [00:31:00] heuristics and where they're used?

Yeah. So in in AI or in, you know, machine learning, , you basically have this thing that yes, you could just take raw data and throw it at the machine, but chances are you won't get a really good response. So what you really want to do is you want to massage the data in a way that like boosts the correct signals.

For the machine learning, right? And so if you just take the content of a page and just throw it at ml, it's not gonna have a hard time because fundamentally, MLS don't understand math. And so they don't understand that this is 37 and this is 38, therefore 38 is after 37. Like they're relationships in there or something of that sort, right?

So what you wanna have instead is a set of heuristics that translates this absolute layout into. Groupings and columns and rows and something that kind of makes more sense. And these heuristics are reasonably good at the beginning, but if you want to get to last, 10 percent or last, the 90, 80 percent rule, right?

If you want to get the last 20%, if you want to get [00:32:00] that really good, you really want to have AI involved. So basically AI looks at it and says Oh, it looks like you're building a hero image. Would you like me to put this in a separate component and label it hero image? And you say, yes, please do that for me, Looks like you're building a menu. Looks like you're building a footer or something of that sort. And so the AI, looks at the structure after all the stuff is gone through, and then can give additional semantic meaning to things that you cannot really extract manually.

Interesting. So you take this Figma file, you iterate through its whatever XML structure that's full of absolute positionings, and you somehow , I say sanitize, I don't mean for security, I mean you adjust the data in such a way that you can then feed it to your machine learning model and you get back.

What do you get back

So the intermediary machine learning model that the purpose of there is to get good row column information so that you can then use that to to lay out the stuff, right? 

by row column? You mean like nesting structure or

Yeah. So if you see things next to each [00:33:00] other let's say you have four things, right? Like upper left corner, upper right corner, lower left, lower right. So like the AI has to make a decision is it two rows inside of columns or is it two columns inside of rows? And that will have implications.

In how you will have responsiveness, right? So when the responsiveness is happening, what's really happening in the responsiveness is that you have columns and these columns end up underneath each other, right? So we basically are turning columns into rows inside of a responsive design. And so the way you break the logical structuring of the application of the layout will have implications of whether or not you will correctly have a responsive design.

If there's a text and the text clearly goes together with an image, if you put it in a separate columns, then if you go to a responsive design, they will end up in wrong locations with respect to each other. And so you want to make sure that logically, this makes sense, what the designer.

And that's, again, it comes down to a [00:34:00] combination of heuristics and what AI has seen before, what has been done before. And based on that, it says okay, I think what the designer meant is to make these two columns instead of these three rows, and so when we convert it over to HTML and we provide a correct set of margins and paddings and all this stuff to lay things out, it will have a reasonable behavior for responsive designs.

man, I'm smiling because this is all just a testament to the enormous amount of work that human beings do trying to parse figmas into actual designs because we all do this, but we do it instantly do it rapidly just by looking and there's so much more that our brains just do automatically.

Silence.

then you probably hand it off to chatGPT and you go okay, cool. Can you give me a mitosis component out of this or something similar? And then you get, okay, that, that pipeline is, I think it's somewhat demystified.

It makes it more approachable to me as an engineer except the part about the custom machine learning model. [00:35:00] What is that? Is that a TensorFlow JS type of thing? Do you all have like powerful GPUs that you use to

Uh, That's a good question. I don't, I haven't worked on that project, so I don't specifically know what exactly they did over there. But yeah, fundamentally you really want to just answer the question of, as I said, again, the rows and columns and groupings, you want to know what logically goes together.

And that is a, aI system that we built and it's significantly cheaper than chat GPT to execute. And this is important. Otherwise we won't be able to offer it at reasonable price. And because it's custom trained for this particular thing, not only is it cheaper, it also gives better results than something like chat GPT.

I like that you said reasonable price and it's actually free. I probably

It has to be, cheap enough for us to run so that we can offer for free, right? Chat GPT is not cheap at all. Like it's fricking expensive.

Yeah. And if you use chat GPT on the free, UI app, you're probably paying for it with your data. So nothing, nothing really is free we're honest about it. . That's so useful. I want to ask what is the future of this feature [00:36:00] is it's now publicly available, how is it being used?

Are people actually benefiting from it? What parts of it do you monitor and. , how is it going to evolve in the coming weeks?

yeah. So if you look at a CMS systems CMS systems are typically used in marketing and typically the way the life cycle starts is, somebody has an idea of what they want to market, et cetera, and then you hand it over to design people to say Hey, make this. Whatever site look good so that we can sell whatever we want to sell.

And this is where the Figma comes in. And then once you have the Figma, then you go and translate it into HTML. The HTML somehow gets into a CMS system and then the CMS system that gets optimized through A B testing, like maybe the color will make a difference, maybe the, font or whatever, you go and change these things to see if the conversions improve. And if you have a CMS system, that's nice, but if you have all of a sudden this thing that goes upstream and says, actually, you can just start with the Figma instead of starting with the HTML that's a huge advantage for it.

And it's a competitive advantage with respect to other [00:37:00] CMS systems, right? So if you're most other CMS systems, most other headless. CMS system, meaning headless, meaning you are the one serving the data. You own the application. You're not hosted like Wix, right? If you're

Can we, 

Can we just tell me if this is appropriate or not? Can we consider it headless CMS is just like specialized databases?

Yes. Headless CMSs are specialized databases, or as I sometimes say, they're glorified Excel spreadsheets, 

Yeah. Okay, good. Yeah. I think this makes the distinction headless versus head full

But what it makes the distinction is really about like,, who owns the infrastructure for serving the technology, right? With headless systems, you as a customer own the infrastructure because you need to serve other things than CMSs.

When you are hosted, really the only thing you can serve is CMS and nothing else.

And that seriously limits the kind of things you can do. Right. So, If you're target. com, it's hard to imagine how you can run on top of Wix because Wix is the one hosting it. If you're target. com, you're like, no, I need to host this stuff [00:38:00] because there's a million other things other than. Just the CMS part, right?

And so this is where you need headless. And when you have headless, the problem is it is really difficult to make a visual editor. And so while hosted CMSs visual editors are common, headless CMSs visual editors essentially don't exist. And so we are oddity. In this particular space. And so not only are we unique in this fact that we have visual editor, now we're also unique that we have this technology that allows us to start with Figma and convert it into the HTML.

So what's next, I guess, is to continue developing this feature. You know, I actually have a feature request. Since you have the visual editor anyway, and you're inferring data from Figma, I wonder if there's room to learn from the Figma what a website's brand identity is and looks like.

And then \ builder, the visual editor itself could have A legitimate bonafide copilot, similar to github copilot, where you can start building UI components [00:39:00] in there itself.

So you can actually do this today. You can go to an AI and say build me a landing page for this product and make it look like Everlane.

And It will spit out a whole bunch of stuff where you're like, yeah, that kind of looks like Everlane.

Yeah. So because I think about, for example, v0. dev. I don't know if you have access yet. I don't. I want to. V0. dev is this thing from Vercel where it's like mid journey for react components. That's the marketing they're using. You, you write down, I want to, I want an avatar card and you get it with react code.

That in builder IO, I think would be tremendous like, because then you don't even need it. An engineer, really. You can just have a designer go I want to, I want a card with a picture. And it appears, but it appears as React code registered with Builder. io somehow. Wow, that'd be the dream.

That is literally what we are aiming for. Yes.

Wild. Maybe I should come work at Builder. io. That's nuts. That's a future I want to definitely see realized because yeah, headless EMSs are an absolute pain. With that, Misho, it's man, every single [00:40:00] time I talk to you, one thing that happens consistently is I keep learning. Whether it's about carbs and letting your meat sometimes go blue whether it's about plants trying to kill us, whether it's about the downsides of hydration, whether it's about.

Proxying the window object from a worker thread. Tremendous. You are a fountain, my friend, and it's an absolute honor to, to be able to talk to you and learn from you on this podcast. On behalf of me and everyone at PodRocket, thank you so much for coming on board.

I love you too, man.