Temporal wiith Ryland Goldstein
===

~Have you guys been, um, doing a bunch of like, uh, I haven't actually seen, um, the recent episodes. Like is there a lot of AI content recently? Um, with all the L L M stuff, Yeah, I mean, we do ai, I'm in charge of AI stuff at temporal and like I actually am investing in some AI companies as well. So I have, I definitely have some like, uh, experience and understanding of that space, but it's, Uh, either way it's completely fine.~

~Yeah, see if we get into it.~

~Cool.~

~Test, test, test.~

~Test, test, test.~

~Okay.~[00:00:00] 

Hi there and welcome to Pod Rocket, a podcast brought to you by Log Rocket. Log Rocket helps software teams improve user experience with session replay, error tracking and product analytics. Try for free@logrocket.com today. My name is Paul and joined with us is Ryland Goldstein. He's the head of product over at Temporal and we've actually had some folks on, including Ryland from TEMPORAL last year.

~Um, ~but we're gonna be digging back into the product a little bit and talking about some of the new features that have come out.~ The new features that are going to come out. Just about temporal in general. Cause it's a great thing and we, we should all get down and, and learn how it works. So~ welcome to the podcast, Ryland.

It's exciting to have you.

I'm super happy to be back again, Paul. It's, ~uh,~ it's a great, it's a great podcast and, ~uh,~ really happy when good podcasts want me on. ~Uh, you know,~ when the bad. Podcasts want you on. That's only lightly exciting.

It's nice to be wanted. That's a good thing.

It is. It is.

How long have you, ~um,~ been head of product Ryland?

~Yeah. Uh,~ so I think a little over three years now actually. So I joined TEMPORAL in April of 2020, ~um,~ and pretty much immediately was, ~uh,~ head of product, ~um,~ [00:01:00] was actually head of developer experience for a short stint there. And then, ~um, you know,~ transitioned it ahead of product pretty quickly. So yeah, it's like a three years and some change now.

At three years, you're gonna have a lot of perspective. You're a good guest to have on the show and really, ~you know,~ crack into the nut of what temporal is because, ~you know,~ we've had other guests come on the show and talk about ~like,~ oh, under the hood we use this big thing. We use this like we use a sledgehammer.

Let me tell you, we use this thing called temporal and we go, ah. We've had them on here before, ~like,~ okay, so it might seem like it has a use case ~all over,~ all over the map, and that's because~ it,~ it really is an ergonomic tool. So for those who aren't familiar, can you start to intro us into what is temporal at a high level?

~Sure. Uh, yeah. So~ Temporal is ~a,~ a platform for developers, ~um,~ to build, uh, like highly reliable and scalable applications. ~Uh,~ and so there's this, ~uh,~ concept, ~you know,~ in computer science ~of,~ of durability, ~um,~ which. Basically means that, ~you know,~ like when something, ~uh,~ loses power or ~you know,~ is ~like~ cut off from a network, that the data and the things that you've, ~you know, kind of~ stately state, fully stored with it, ~um,~ remain there and ~are~ are safe.

~Uh,~ and [00:02:00] so temporal is ~sort of, um,~ like foundationally built on this concept of durable execution, ~um,~ which combines that basic concept of durability with ~like, you know,~ the concept of a computer programming,~ uh,~ program executing and moving forward. ~Uh,~ and so the idea is that you wanna have this computer program that you know is moving forward and it's.

Progressing. But if at any point in time, ~like, you know,~ the network cuts out or the server the program is running on, ~you know,~ fails, ~uh,~ the program is able to continue on as if, you know nothing had happened. As if, ~you know,~ when ~the,~ the light switch got turned off, all the state that was relevant for the application was stored somewhere in, in insured to be safe.

It's, could I say I'm like thinking about breaking down my program no matter what language it's in, if it's in the support, because you guys, it's supported language, right? I have to use a supported SDK to run a temporal, ~they're,~ they're vast, but provided I use a supported language, it's like breaking it down into atomic steps that you guys keep track of.

Absolutely. ~So, um, the,~ the nuance there is that, ~you know,~ we're not doing it, ~uh,~ every line ~of,~ of your code. ~Um,~ that's like a lot of people when they first see temporal and they understand how it works at a high level. [00:03:00] They ~sort of~ assume, oh, every line of code you're breaking and, ~you know,~ sending to your server or something like that.

~Um,~ in reality we've just, ~uh, kind of, you know,~ realized,~ uh,~ over a long period of time that there's these very specific points in a, ~you know, um,~ executing programs lifecycle where, ~uh,~ things are being done state fully, ~um,~ in a way that can't be reproduced. And so a great example is, You write a piece of code that calls, ~you know,~ to, ~um,~ an external server over H T T P, ~um, well,~ there's a point where like your server itself can't just reconstruct the arbitrary response of another server because it's not that server.

And so you need to store that somewhere because if that other server is down and return something different in the future, ~uh,~ your program will act differently than you, ~you know,~ intended it to. And ~so, um,~ these specific points, like these, ~um, sort of, you know, um,~ inflection points in your code, ~um,~ where stateful things happen.

Temporal has been designed to automatically ~sort of~ capture those, ~uh, you know,~ stateful events and send them to this server, which, ~you know,~ stores them durably in, in fault tolerant, ~um,~ fashion.~ Uh,~ and that way if your, ~uh,~ application crashes at some point in the future, ~you know, we,~ we can, ~um,~ reconstruct its state.

And ~so, you know,~ going back to what you originally asked, ~um,~ we, we do have to do this at a language per language level. ~Uh,~ and we actually ~kind of~ make a [00:04:00] unique choice there, ~um,~ where a lot of ~you know, uh,~ companies that provide an API and they wanna provide a language experience in a bunch of different languages.

~Um,~ they tend to write these SDKs, which amount to being. Wrappers around an HTT P A API or a G R P C api. ~Uh,~ and ~you know,~ other than the fact that it's in go or it's in Python, it doesn't really look like a GO API or go sdk. It looks just ~like, you know,~ this product's API that has been surfaced into Go.

And so with Temporal we chose a very different route, which is we do things very idiomatic. And ~so, um,~ if there's some core primitive of the temporal platform that says, ~you know, um,~ you can have a concurrency or ~like, you know,~ parallelism in a workflow, ~um,~ that means we need to go and ask ourselves for every language we support.

~Like,~ what does it mean for a developer to, ~you know,~ use concurrency within, ~uh,~ Golang? ~Like ~what would be the intuitive ~and,~ and ~kind of~ natural way that they expect that? And ~like,~ then we have to kind of surface that exp. In ~that,~ that fashion, ~uh,~ so it stays idiomatic and true to what developers expect.

And so we do that ~for,~ for every language we support. And so there's quite a bit of work, ~uh,~ and a really brilliant team behind making sure that our s d K experience, ~um,~ stays top-notch.

And you can see [00:05:00] that when you go to the temporal docs,~ I mean~ the docs between even the Go S D K and the TypeScript sdk, the two that I have just very lightly perused, like ~they're,~ they're vastly different. And at first I was scared cause I was just like, oh crap. Cuz you're used to looking at ~like,~ you're used to looking at an SDK that's.

Just a wraparound in api, ~like~ I feel like that might be the status quo. So when I'm looking at SDKs and they're wildly different, I'm like, did the team just ~like~ pivot? ~But what?~ But ~what,~ what you're telling me here is really driving home, ~like~ the ethos ~of,~ of temporal, which is ~like,~ we wanted to make it ~No, no, we wanna make~ the best it could be.

So you made it idiomatically in line with the language and focused in on that language. ~Um,~

~Yeah.~

~I,~

~Oh.~

~go ahead.~

Yeah, so the specific thing, ~um, you know,~ TypeScript ~and,~ and Go are great example because they don't just represent ~like sort of ~the, ~you know,~ idiomatic approach we take to each language. But it also ~sort of~ represents the evolution of the way that we think about designing these language experiences in general.

~And so, Um, you know, when I,~ when I joined, ~uh,~ temporal, there was only two languages that we had support for, which were go in Java. And so one of the first things that,~ you know, I,~ I pushed for when I joined the company was~ like, you know,~ we need to add a new language. And specifically I felt ~like, you know,~ [00:06:00] TypeScript and JavaScript were ~kind of~ like ~the,~ the place that development trends were going in general, and, ~you know,~ We could get a lot of, ~you know,~ users and a lot more growth ~from,~ from that language specifically.

~Uh,~ and so one of the things at the last company, ~um, I,~ I started a company before temporal, ~uh,~ and we were trying to, ~you know,~ build a competitor to AWS Lambda. ~Um,~ and so one of the things that we had spent a lot of time, like what I had been tasked with looking into,~ um,~ very early on was like, These alternatives to ~like, you know,~ container based isolation.

~Like, you know,~ you had virtual machines and eventually now everyone started running things ~like~ in docker containers. ~Uh,~ and there was this idea that if you could have ~like~ an isolation mechanism that was much cheaper than like a, ~you know,~ a Docker container, ~um,~ it could be really important for ~like~ a cloud business's growth and ability to have good margins because, ~you know,~ we could run all this stuff, ~um,~ in a lot higher scale than you would be able to do ~with,~ with containers.

~Um,~ and so unfortunately that didn't. Pan out, ~um,~ that sort of direction. ~Um,~ but the research, ~uh, like kind of~ had me focus on this concept within, ~um,~ v8, which is Chrome's, ~uh,~ JavaScript engine. ~Like, you know,~ the thing they use to ~like~ browse the internet. ~Uh, ~and in, in v8 ~uh,~ there's this concept of ~like~ isolates, ~um,~ which are ~like~ a mechanism that, ~you know,~ this JavaScript engine provides [00:07:00] for running a piece of isolated JavaScript code, ~uh,~ that is totally sandboxed from everything else.

And this is the way that like, Tabs work in your browser, right? Why you can't just have one tab maliciously take over another tab or something like that. ~Uh,~ and so from this work that I'd done, ~um, you know,~ before I joined temporal, I knew about this isolation mechanism. It's like ~a,~ a virtual machine, ~um,~ for lack of a better way of putting it.

~Uh,~ and so when I, ~uh, you know,~ showed up at temporal, like R c o Max, he had been very, ~uh, sort of~ enthusiastic about web assembly, ~um,~ because we have this, ~you know,~ Kind of requirement in temporal that developers need to understand determinism and item potency. And those are very scary concepts. ~Uh,~ and so when you're programming, for example, in our Go sdk, these are like very, ~um,~ in your face things that you need to be aware of.

~Like,~ oh, if I write this line of code,~ um, you know,~ maybe my, ID won't tell me it's wrong or anything, but because it's temporal, it's not against the, or, it's against the rules. And ~like,~ this is gonna cause my, ~um, you know, uh,~ application to run into issues. ~Uh,~ and so because of this, ~uh, you know,~ like V8 isolate concept that's part of, ~you know,~ this, ~um,~ Chrome JavaScript engine.

~Uh,~ I ~sort of ~realized that even though ~like, you know,~ our c o thought web assembly would be the solution for solving all these determinism problems, like web assembly was [00:08:00] not in a place at that time to be practically useful to us. ~Uh,~ but specifically for JavaScript, ~this,~ this VA thing did a lot of this stuff We were hoping web assembly would, ~um,~ and it was available and it was, ~you know,~ widely adopted and ~like, you know,~ Part ~of,~ of no Js as an example.

~Uh,~ and so that's what I pushed and like eventually we ended up hiring, ~um,~ one of the most brilliant engineers I know, and like ~a,~ a best friend of mine, ~um,~ who ended up building our TypeScript SDK on top of, ~uh,~ this VA engine, which is like the first time we'd ever tried something like this. And ~so, um,~ the reason ~I,~ I say all this is that, ~uh,~ the outcome, like the reason it's so exciting is that when you.

Program with TypeScript and temporal, ~uh,~ you don't actually have to think about determinism at all. ~Um,~ we have actually created an environment where it's impossible to write an non deter, deter, ~uh,~ deterministic code. Like the, ~uh,~ compiler, for lack of a better way of describing it, ~um,~ actually won't let you do it, ~right.~

And ~so, um,~ we actually fundamentally removed that whole. Class of ~like, you know,~ concerns and things that you have to worry about traditionally as a temporal developer, ~um,~ as part of, ~you know,~ this TypeScript, S D K. And so now when we've started adding even new languages, ~you know,~ after TypeScript, ~um,~ this has really inspired us and really made it hard for us to ever wanna add a new language that doesn't [00:09:00] have these capabilities and doesn't just, ~you know, kind of~ solve this whole class of problems ~for,~ for our users.

~So, um,~ there's been quite a bit of evolution even within the way that we developed these SDKs, idiomatically, ~um,~ in the last ~like, you know,~ few years.

That's fascinating. So you tried to look at web assembly and it was, the idea was there, ~the, the,~ the landscape wasn't, but there was something, so it was the V8 isolates and that really pushed you guys in that direction. So ~when you,~ when you draw that, like mar that winning flag, you're like, you don't even have to think about determinism really.

Like ~what,~ what is determinism Ryland, if we were to like, I don't want, I don't wanna read ~the,~ the amazing, by the way, documentation, which is like paragraphs long on the site, like determinism in my code. ~Like~ what does that mean?

It's a very, ~uh,~ existential or philosophical, ~uh,~ concept almost, ~you~

I know we're getting at like the halting problem here. ~Kind of~

Or even like hearkening to ~like, you know,~ religion and predeterminism and all of that stuff. ~Um,~ so I think it's, it is rightfully scary to a lot of

to philosophy with Ryland.

Yeah, I don't know if you'd want to stay tuned for that one. ~Um,~ so yeah, I think, ~you know,~ determinism, ~uh, it's,~ [00:10:00] it's, it sounds really scary, but I think, ~um, you know,~ when it really comes down to it, ~it's,~ it's a pretty straightforward concept, which is that there's certain.

Types of code that you can run, ~um,~ that no matter how many times you run them, they'll, ~you know,~ produce the same output. And there's other types of code where you can run them one time and get one thing and run 'em another time and get ~a,~ a different thing. ~Um,~ even though you probably passed in the same variables or, ~you know, ~everything about your configuration when you called it ~was,~ was the same.

~Uh,~ and so that's really the difference between, ~um, you know,~ determinism ~and,~ and non-determinism.~ Um,~ I think one thing that, you know, maybe, ~uh,~ it helps me because I'm like someone who's. Spends a lot of time thinking about computer science and temporal, but ~uh, Math, math, like,~ you know, math, there's functions.

~Um,~ and every single function that you can write in math is deterministic. ~Um,~ and the reason why that's, ~you know,~ like true is that well, you can write it down on a piece of paper, ~um,~ and if you can write it down on a piece of paper and I give it to someone else, there's no way that I can like influence that.

The outcome of that function is gonna be different. Like once I write it down, it's hard coded. And so I think that's ~a,~ a decent way to think about it, which is like anything that's math related and is inherently deterministic. There's no randomness, right? There's no chance that, ~you know,~ part of a math [00:11:00] function fails or that, ~you know,~ like it returns a different number than it did before.

~Um,~ everything is very explicit input out in front of you. And ~so, um,~ for me, that's ~kind of ~the easiest way to understand what ~is,~ is and isn't deterministic. I go in. Think, ~well,~ could I write it down on a piece of paper and hand solve it? ~Uh,~ and if you can't, ~well~ then, ~you know, it's probably,~ it's probably not deterministic.

~I,~ I love the piece of paper thing. I've been trying to wrap my head around it thinking ~like, uh, ~like from the react point of view, is there a side effect? That I need to pay attention to here and my use effect ~and,~ and in my state cycle. ~Um,~ but the piece of paper thing is really neat because that's like material and it sits in my head real nice.

I really like that. ~Um,~ so what if somebody ~is,~ is listening to this and they say, all right, I'm writing a function and ~well,~ I write a lot of functions and it's a cli, right? And you give some input and you give some output. They could give a different input every time. ~That's a,~ that's a non-deterministic piece of code, right?

So,

No. So that's, ~um, I,~ I should have mentioned that's like a key, ~uh,~ facet of determinism is that for the same inputs that you always get the same outputs, right? And ~so, um,~ just to ~like, you know, kind of like~ follow that through with the previous way ~of,~ [00:12:00] of framing things, if I, ~you know,~ write down on a piece of paper a math function for summing two numbers, ~um,~ I can give, ~you know,~ two different numbers, ~um,~ between ~like, you know, uh,~ executions of that math function.

~Uh,~ but if I give the same numbers, I always get. The same, ~uh,~ result back. And so that's what inherently makes it deterministic, ~like, um,~ obviously can be based on the inputs that you get to ~the,~ the code. It's just that only the inputs and what's defined in the code are what matters. And nothing external or, ~you know, um,~ outside of that function itself.

A and that sounds like amazing as a developer to be able to say ~like,~ my code is like deterministic and it'll always run. ~And it sounds,~ and it sounds like a very difficult, ~like~ that's what we engineer for. That's what we want our code to do. ~And then,~ and then you're out here heavy slinging saying, Hey, you guys, if you're in TypeScript, you don't even need to think about it.

~And,~ and the way you do that is, To my understanding with some abstractions, right? So we, we have ~some,~ some things called workflows and activities, and you guys really molded. The SDK to force you to code in the right area. And you also, I, and before we jump into the section, I also wanna think about the compiler, cuz you mentioned Okay, the [00:13:00] compiler for the lack of a better term.

~I,~ I know it's not really compiler, but when people hear that might go, oh gosh, there's a compiler. ~Like,~ I don't wanna deal with that. ~Um,~ so yeah, what, what's really happening under the hood with your S D K and how it forces you as a developer to sort down these two buckets?

Yeah, it's a great question. So I think, ~you know, um,~ we will start with the, ~uh,~ the go example because it doesn't protect you as much. It doesn't protect you, ~um,~ almost at all. ~Uh,~ and ~so, you know, uh,~ the basic idea is that we give you guidance and go. So like ~the, ~the thing that's really stopping you from, ~you know,~ not writing or from writing code that's non-deterministic is you,~ um,~ when you're a go developer with temporal, ~um,~ you have to know that if I'm making a call to, ~um,~ an H D T P service that's non-deterministic and I need to put that in an activity, which is ~sort of~ our container, ~uh,~ for non-determinism.

~Uh,~ and you have to know, for example, if you're doing a sleep in your code, you can't use the, ~um,~ Language, native sleep. You have to, ~you know,~ use, ~um,~ either the temporal sleep or, ~you know,~ call sleep in an activity. ~Uh,~ and that way you can ~kind of, you know, uh,~ encapsulate this non-determinism. And ~so, uh,~ in go, this is just something that, ~you know, in a,~ in a few of our other languages that you just have to, ~you know,~ know the rules about and you have to watch out [00:14:00] for.

~Um,~ there's various efforts. There's an amazing talk recently that, ~um,~ Chad from our s d K team gave. ~Uh,~ about the different approaches we use for, ~um,~ helping with non-determinism ~in our,~ in our various languages. ~Um, but you know, for,~ for go as an example, there's not much we can do for you. And ~so, um,~ one of the things that's ~like, you know,~ really, ~um,~ specifically funny about go, ~uh,~ is that in go, there's, ~um,~ non-deterministic iteration over a map.

~Um,~ which might even sound weird, but basically if you try to iterate over a map ~and,~ and go, ~um,~ most people would think that, ~you know,~ if I call that. For loop over a map, it's always gonna run the same. ~Uh,~ but going intentionally made a choice that they're not gonna keep the iteration order the same between executions of the program.

And so it's not even, ~um,~ actually legal to use an iterator over a map directly in temporal within a workflow because it actually is a source of it. And so that's ~like~ one where, ~you know,~ I remember I ran into that the first time and it's like I understood determinism even at that point. And I'm like, yeah, there's this.

Really well understood set of things. And then it was like, oh, apparently because of the way they design go, this is one of those set of things that I need to, ~you know,~ specifically think about. ~Um,~ and ~so, uh,~ to contrast that, if we go back to, ~you know,~ the TypeScript [00:15:00] example, ~um,~ so yeah, when I said compiler, ~um,~ that was a bit misleading.

Really what's going on is we have this, ~you know,~ v8, ~uh, you know,~ JavaScript engine ~under,~ under the hood. It's essentially providing like a. Deterministic virtual machine for each of~ your,~ your workflow executions. ~Uh,~ and so one of the beautiful things about like the concept of a virtual machine is that it's virtualizing all of the underlying, ~you know,~ like IO and hardware that the computer is actually exposing.

~Uh,~ and when it comes down to it, pretty much every source of like non-determinism in this world is. From io it's from, ~you know,~ having to, ~uh,~ connect to a hard drive, which is, ~you know,~ going over some sort of bus and that could fail. And ~so, like,~ you can't guarantee that, ~you know,~ a read from DISC is always gonna work.

~Uh,~ it's from, ~you know,~ calling over ~like, you know, uh,~ a network and,~ you know,~ going over the network card and, ~you know,~ making a request. ~Uh,~ and that could fail. ~Um,~ or even just things like, ~you know,~ random number generation, which is based on~ like~ some timing of ~like, you know,~ cycles of the processor, which you can't necessarily count on, ~you know,~ between any, ~uh,~ runs ~of,~ of a given program.

And ~so, um,~ the beautiful thing is that because,~ you know,~ TypeScript V8 ~sort of~ surfaces itself as this virtual machine, ~uh, you,~ you can plug in what a. What io you want to be available for the environment of [00:16:00] the user, ~um,~ when they're writing their code. And ~so, uh,~ basically what ~we,~ we did is we just said none of it's allowed, ~uh,~ you, you can't do any of these things that, ~you know,~ would typ typically cause you to create a non-deterministic behavior.

~Um,~ and, ~you know,~ we'll just not let ~the,~ the sandbox recognize it. And so it doesn't even let you write the code, like when you're typing it in, it will just be like, this isn't a valid call from this context. ~Uh,~ and there's absolutely no way that you can get yourself into trouble. And now, Obviously sometimes~ it's,~ it's surprising for people because they're like, why can't I write this code?

~Uh, you know,~ normally you could write that code in any place that you want. ~Um,~ and so then they still have to go and read the docs, but at least it doesn't let them foot gun themselves and ~like,~ run it in production and create a problem. ~Uh,~ they just have to go and, ~you know,~ do some education before they, they fully grasp what's going on.

It feels very, and correct me if I'm wrong, but it feels akin to ~like~ the TypeScript compiler when I'm working in VS. Code. It's your guiding hand to bring you along and tells you what can be in each area that you're writing. Would that be fair?

~Uh,~ yeah. Yeah. ~I mean,~ we're using, ~um, you know, like, uh,~ God help us, we're using like Webpack, ~uh,~ and we're doing a lot of stuff, ~you know,~ like at the translation level because ~we,~ we do need to do some fancy stuff in JavaScript to make all of this work. And ~so,~ yeah, it's very similar to [00:17:00] like, you know, I will say that myself and, ~uh,~ Roy, who was the main person buying the sdk, ~uh,~ I think we really liked TypeScript in terms of the tradeoffs it made as a language and the way that, ~you know,~ it ~kind of ~surfaced like a typed experience, ~um,~ having used a bunch of other, ~you know,~ strongly typed languages.

~Uh,~ and so I think there was a lot of inspiration that we took for the SDK, specifically from TypeScript, which in the world of temporal is a great way of doing things because it makes it even more idiomatic. ~Um,~ so yeah, I think that there's a lot of parallels between the way TypeScript itself works and the way that ~we've,~ we've implemented these things in TypeScript.

And you're speaking to the right audience here.~ I mean, we're,~ we're in a web development podcast talking about. Lots of, there's probably majority type script developers like listening to this that are probably like, yes, Ryland, you're speaking my language, man. So yeah, ~you,~ you were definitely correct about ~like~ the, assuming that you were gonna bring in a big developer base with that one.

Yeah, I mean I, ~uh, you know,~ in all fairness, I have a Stack Overflow blog post where I'm like, the article is, ~uh,~ the best ways to write modern JavaScript. And the number one step is write type script instead. So,

[00:18:00] I mean, looking back, I feel like it'd be crazy to ~like not,~ not use TypeScript and because it's all about reducing your errors, reducing the amount of work you want to do. ~Um, and, ~and Ryland, I'd love to ~like~ step into some of the cool features that not only I've seen you. Put videos about on YouTube about temporal and like schedules, for example.

That's super exciting. ~Um,~ before we do that, I just want to talk to our audience of web developers listening and remind everybody that this podcast is brought to you by Log Rocket. And Log Rocket will help you and your software team find. Errors within your application and surface them quickly using the powers of AI to find patterns that you might not pick up on.

Error tracking and product analytics. You can spend more time building the app and less time in the console, less time where Ryland's team is and like breaking open V8 for us so we don't have to chase as many things down in the future. So head over to log.com today to try for free.

~That~

~So, right.~

~great,~

~Sorry.~

~so~ it does sound pretty great.

Yeah, ~it's, it,~ it definitely will help you develop [00:19:00] against edge cases in ways that like you just ~like~ didn't think about, ~which is,~ which is really neat.

And ~the,~ the heat maps ~are,~ are fun to play with, especially, ~um,~ on new features for temporal one that. ~I,~ I can't wait to talk to you about, cause it saved me like weeks of programming is the schedules. ~Um,~ you've, you guys have had scheduled workflows in the past and now we're looking at the new schedules feature.

So I, anybody listening, you're like, what is the scheduled workflow? Is this thing where you could ~kind of like,~ make this deter this deterministic piece to code, maybe run it on a chron job or something. Now Temporal's coming out with schedule, so Ryland, could you tell us about what was and what is. And why it's a big deal.

~Yeah. Uh,~ yeah, schedules is like a very interesting microcosm of temporal, ~uh,~ in a lot of ways. ~Um,~ so the basic idea is that, ~you know,~ there's ~this,~ this kron concept, which is one of ~like~ the most, ~um,~ prevalent, ~uh,~ concepts ~and,~ and pieces of technology, at least in my experience ~as,~ as a developer.~ Um,~ but I think for most developers it's true.

And ~so, you know, uh,~ Kronis is ~sort of like, uh, you know, the~ like. Redheaded stepchild in a sense where ~you,~ you [00:20:00] always need it. And it's never like this thing that you think about primarily, it's always like ~a,~ a secondary concern. You realize, oh wow, we probably need to run something like on a timer. ~Uh,~ and so ~like, you know,~ there's always ~a,~ a places I've worked, like some Linux server that you have, like just running some set of your CR jobs that, ~you know,~ like probably are really important to the business, but you ~kind of~ treat it as like this random server that you just maintained for, ~you know,~ doing this ad hoc stuff.

And ~so, Um, you know,~ I think there's this very common problem that, ~you know,~ you need to run things ~on a,~ on a schedule. ~Um,~ maybe you need to be doing basic monitoring, pinging a website. ~Um,~ there's~ like~ a million different reasons that you might need it. ~Uh,~ and it's actually like surprisingly difficult, ~um,~ to go and do this ~in a,~ in a ~like,~ say professional way today, ~you know, so, um,~ outside of ~like ~hosting your own server, ~uh,~ and running, ~you know,~ Kron just from, ~you know,~ like the command line, ~um,~ which then brings a whole set of questions of ~like, you know,~ Like, how does that server stay up and what happens if it goes down and is it stateful?

And like, how does Theron behave in those cases? And ~like,~ who's gonna be watching that? And ~so, uh, you know,~ just like doing that in a reliable and ~sort of, um, you know,~ trustworthy way is actually a very difficult problem historically. And even today, I think, ~uh, for,~ for most companies and for most developers.

~Uh,~ and so this is something, ~you know,~ from very early on, even before. For, ~uh, you know,~ the team left Uber, ~uh,~ that they had [00:21:00] identified as being like a real pain point for, ~you know,~ d developers who were working with distributed systems is that they needed this, ~you know,~ a reliable way of running scheduled jobs and running things ~on,~ on a timer.

~Um,~ and ~so, you know,~ very early on in ~the,~ the history of the technology, they added this, ~um, you know,~ like ~sort of. Uh,~ Kron capability to the system. ~Um,~ which, ~you know,~ in all fairness compared to the other solutions that are out there are probably much better because it's built with temporal. So you know that it's gonna be like, ~you know,~ durably executed.

Like it's not gonna go away if the server crashes. Like all of those stateful questions that you would have to ask about, ~you know,~ a traditional Kron job that you're just running as like a. A process, ~um,~ that no longer is ~sort of, you know,~ holding true even with ~our,~ our initial solution. ~Um,~ and ~so, you know,~ that had always been like a pretty popular feature in the product.

~Um,~ even though the, ~you know,~ like DX around it was severely lacking. ~Um,~ it was just solving such a big. Gap in like the ecosystem and what people, ~you know,~ needed, ~um, you know,~ out of software that people were flocking to it. And ~so, um,~ I think that, ~you know,~ from my experience at least, there's two really major reasons that people adopt temporal.

~Um,~ one is that they need like some sort of general purpose or service orchestration, microservice, orchestration, whatever you want to [00:22:00] transactions. ~Uh,~ what something that falls into that group. And then. The only other real category that's like top level to me is like people want it for doing Chrons.

~Like~ the surprising number of people adopt temporal simply just because they need to solve this, ~you know,~ durable cron job problem that they have in their company. ~Um,~ and so that's ~sort of~ where things were. ~Uh,~ and ~so, you know,~ as we saw how popular this feature was over the last couple of years, ~um,~ and then realized how actually not great.

The developer experience was around it for how popular it was. We started feeling ~like, you know,~ not super great about that. And so we, we said to ourselves, ~um, you know,~ there's probably some ways that we could like iteratively get this, ~you know,~ into a better place, but it's really misaligned with the way that, ~you know,~ we want this experience to, to happen.

~Uh,~ and so that's when we decided to ~like, you know, kind of, um,~ rewrite things and produce a fundamentally new, ~uh,~ version of the experience that we call, like scheduled workflows. ~Uh,~ and so that's something that we started over a year ago. ~Um,~ the idea was that, ~you know,~ from the, ~uh,~ like mechanical point of view, like a lot of things about the existing solution were great.

~Uh,~ but when it came to ~like~ the actual APIs and the capabilities and ~like, you know,~ the way that you interact with it,~ um,~ it was just ~really,~ really lacking and honestly pretty unusable in certain cases. ~Uh,~ [00:23:00] and ~so, you know, um,~ outside of like things that just. Flat out didn't work with the old solution.

It was also missing some capabilities that I feel like make the new one super powerful. ~So, um,~ one example is,~ uh,~ you have this situation that can happen with a Aron job where, ~uh,~ you say have it run every 30 minutes, ~um,~ and it fires one time and there's some piece of code that starts running and maybe that piece of code takes a pretty long time, ~um, you know,~ to finish.

~And so, um,~ 30 minutes later, ~uh,~ the Crown job is supposed to fire, but the piece of code that it started, ~um, you know,~ on the last, ~uh, you know,~ in invocation hasn't finished running yet. ~So,~ so what do you do? ~Uh,~ so with the previous, ~uh,~ Kron solution, the system made an opinionated choice and you didn't get any say in it.

And ~like~ that was just the way the world was. And ~so, um,~ one of the awesome things that you can do with, ~uh, you know,~ the new scheduled workflows is you can actually specify the policy for what should happen when an invitation gets fired, when an. ~Uh,~ an existing invocation is already running. ~Like, um, you know,~ should all of them be able to run in parallel?

~Like~ should it, ~um,~ be queued up in a fashion? ~Like, you know,~ should it only allow one to run and the other one simply won't happen? ~Um,~ so that's like something that, ~you know,~ was just like fundamentally possible now that, ~you know,~ wasn't possible before. ~Um,~ and like another example there is, there's this concept of a catch up policy, ~um,~ where, ~you know,~ imagine that you have ~a,~ [00:24:00] a crown jobb set to fire and the temporal server itself goes down.

And so a few minutes later ~you, you know,~ Re restart your temporal server. And now there's this question of ~like, well,~ there's this crime that was supposed to fire, but the server was down. Do we still fire it or not? ~Uh,~ and so this catch-up policy now also lets you to con control the behavior in that case where you can say, oh, ~well, you know,~ I wanna fire anything that would've fired in the last five minutes.

Or, I wanna fire everything that would've fired, or I don't wanna fire anything at all. ~Um, ~and just ~kind of~ pick up where we are now and not think about what we missed before. ~Uh,~ and so just things like this truly transform the experience and, ~you know,~ make this like a. In my mind, like a world class, ~uh, you know,~ like Kron solution, durable kron solution ~on, ~on top of that, ~uh,~ that most people would probably prefer to use over all the other Kron solutions, at least I know of.

And I, I think it's interesting when you really highlight these deep dive differences that you just double clicked on. Like specifically like an overlap policy, a catch up policy that could be. A behemoth to tackle ~in,~ in just a regular chron? Or if you're reaching for an open [00:25:00] source solution, does it implement it?

How does it implement it? Then how does your sister systems then talk to it? Having it in one spot is ~you're,~ you're right. ~It's,~ it's so powerful. It's fun to play with. ~Um,~ so was schedules, if could somebody realistically ~like~ have made schedules by themselves a bespoke implementation using. Workflows somehow to then,~ you know,~ cuz under the hood, I guess I'm wondering does it run on a workflow, the schedule?

It does. Okay.

It does. Absolutely. It's a great question. ~So, yeah, um,~ this is a big debate that we had at the time when we, ~um,~ were trying to design it, ~um,~ initially. ~So, uh,~ initially the idea was that we would do it in a much more, ~um, you know,~ like ~kind of.~ Like bespoke in, ~um,~ a specific way and ~sort of~ build it into the actual state machines of ~the,~ the system, ~um, ~itself, ~um,~ in a more integrated fashion.

There's still actually some custom state that we have to do with even the way that we, ~um,~ we did approach it, ~uh,~ but we ended up deciding that we wanted to do it a lot more like. The dog food, ~um, you know, kind of~ the same way that~ a, ~a user would have to do it. And that meant that, ~you know,~ we were basically going to be,~ uh,~ running this workflow to actually manage [00:26:00] your scheduled workflow that you define and, ~you know,~ you ~kind of~ set up ~as,~ as a crown job.

~Uh,~ and so what that means is that, ~you know,~ within the temporal cluster, ~uh,~ in the temporal service itself, ~um,~ we actually have to run, ~you know,~ workers and, ~you know,~ we have to run these, ~um,~ workflows that manage the scheduled executions. ~Um,~ now in, ~you know,~ reality, we already did have like workflows that run as part of temporal itself.

So that's part of ~the,~ the way the product runs. And so there was already some ~like,~ kind of infrastructure, ~um,~ using that term loosely, ~uh,~ set up in inside ~the,~ the architecture, the product to allow us to run workflows. And so it wasn't like we had to go and, ~you know, ~solve that problem ~in a,~ in a novel way.

~Uh,~ but we did have to, ~you know,~ make some changes and some improvements in, in how all those things work. Because of how much people will want to use, ~um,~ scheduled workflows. And~ so, uh,~ there might be a lot more of these workers in these workflows running internally than we would've had in the past. And so we definitely had to, ~you know,~ take that into consideration when we design the feature.

And ~you,~ you mentioned ~the, you know,~ you guys have to run workers earlier, ~you know,~ using the verbiage of~ like,~ okay, the temporal server is running your schedule, or what be, it makes me think about where is temporal running? And I know in the last [00:27:00] podcast we talked about ~like,~ yes, it's ~like,~ just to remind our listeners, it's an open source technology, which is fantastic.

~Um,~ and there is temporal cloud. Where you can, you don't have to host your own. You can go ~like,~ sign up, you can get a namespace, make your own cluster.~ Um,~ is what we're talking about right now, schedules, is that available? Ga ~uh,~ and if not, when is it gonna be available for people to use?

Yeah, so it's been, ~uh,~ it's like in a production state for, ~uh, ~the. Open source and for ~the,~ the cloud environments. ~Um,~ for the cloud right now it's in ~a,~ a state that we call public preview. ~So, um,~ it's totally, ~you know,~ like usable. ~Um,~ you're, we're happy to, ~you know,~ have people build stuff on it.~ Um,~ like I'm very proud of where it is at this point.

~Uh,~ and so yeah, people should feel totally free to use it. ~Um,~ and I know lots of people who have already heavily adopted it on ~the,~ the self-hosted side. ~Um,~ and I've already built some stuff within myself on the side, ~so, um,~ I can definitely recommend it, ~um,~ from the user experience point of view.

Is, are schedules now built into temporal this, ~uh,~ the single binary that you guys have? ~Um,~ and really quick for anybody who doesn't know what the single [00:28:00] binary is, what is the single binary,

Yeah, ~so, uh,~ it's now ~the,~ the single binary is like ~the,~ the OG name. ~Um,~ we're now referring to it as like the temporal c l I. ~Uh,~ so the first answer is, yeah, ~uh,~ schedules are definitely available. Pretty much any part of temporal is available through the, ~um, you know,~ temporal C L I. And so temporal c l i is really, ~um,~ we're something we're really excited about.

We think that in a way it's small, but in another way ~it's,~ it's super huge because it, it makes it so much easier to get started with the technology, ~uh,~ and just develop with it locally even after you've already, ~you know,~ gotten started with it. ~Uh,~ and ~so.~ It's a, ~uh,~ distributable binary, ~um,~ something that you can install via brew.

~Um, you know,~ download off of GitHub is like a zip, ~um,~ and put into ~your,~ your bin folder. ~Um,~ and it allows you to run all of the stuff that you would wanna run with temporal, ~um,~ without any other dependencies or any other files or any other resources. ~Um,~ and so in this one, ~uh, you know,~ compiled binary file, we have the, ~um,~ temporal server.

So like all the services that go into that, ~um,~ we. Allow you to basically use ~a,~ a database because we, ~um,~ have a, ~uh,~ integration with SQL Light. ~Um,~ and so SQL Light right runs as like an in-memory database for temporal [00:29:00] in the temporal cli. ~Uh,~ and all of that side of things, including like the initial sort of compilation of the server into the single binary is something that we were like gifted by Datadog, actually who was a super close partner of ours.

~Uh,~ and ~so, um, you know,~ there's an amazing engineer, ~um,~ over at Datadog named Jacob. ~Uh,~ who was the one who initially, ~um,~ gifted us the contribution of the sequel light driver and also liked the initial thing, which he called temporal light, which eventually became temporal c l i. ~Uh,~ and so temporal c l I became this way that replaces our, ~you know,~ docker based, ~um, you know,~ getting started experience and just allows you to install this.

Thing with a command. ~Uh,~ and then you have your ui, you have your temporal server, ~uh,~ you have a database and, ~uh,~ you even have ~like, you know,~ the entire temporal c l I itself, like the, what we used to call the T ctl. ~Uh,~ and so you don't even need a separate tool to ~like~ administer the cluster once it's started.

It's all just contained within this one package, which is really convenient. It makes life a

R really convenient. ~I mean,~ when you think about booting up a distributed stack to do local dev, it's never ~like, I,~ I hate if I look on ~a,~ a d, ~um,~ some GitHub and there's a docker composed of ~like~ 17 services. I'm just like, oh my God. [00:30:00] One of these is gonna crapshoot and then I'm ~gonna~ gonna not develop and I'm gonna troubleshoot Docker.

So ~it's,~ it's beautiful to have the single binary experience. So you said this available in brew? I could just like brew install. Is it cask?

~Uh,~ no, it's just so straight up, like top level. ~Uh,~ you can just brew, install. ~Um,~ I think that right now we don't like the. From having known this from personal experience, the path to getting on apt, ~uh,~ and like the more Unix, ~uh, like,~ sorry, Linux, ~um, you know,~ like flavored packages, ~uh,~ they're just harder.

~Uh,~ so we've actually started that process, but it just takes longer to get into ~like~ one of the main distros, ~um,~ in terms of. Getting your package there. ~Uh,~ and so that's something that we're working on. So for now, if you're like, ~you know, um,~ using proper Linux, you can just use like the curl. We give you a command, I think on ~the,~ the getting started.

~Um,~ but yeah,~ for,~ for Mac users, I don't know if we have Mac ports yet, but we definitely have brew. ~Um,~ that's how I keep it up to date. ~Uh,~ so it works. Works great.

~I,~ I love this. So we talked a little bit about what are, what is durable execution, A little bit about the SDKs, a little bit about the single binary and temporal cli, [00:31:00] and then some bleeding edge stuff. What is coming Ryland that you can tell us about either in cloud ~that's,~ that's already in the open source.

Or something that just hasn't dropped entirely that you might want to ramp people up for.

Yeah, I think, ~um,~ so ~there's,~ there's a few things. So I think, ~um, you know,~ one of the, there's ~like~ two things that are coming pretty soon, at least in ~like~ an initially usable state, which are workflow versioning, ~uh,~ and workflow update. ~Um,~ so workflow update is, ~uh,~ the. More straightforward to explain for sure.

~Um,~ right now in temporal, ~uh, you know,~ we have these workflows, ~um,~ that are ~sort of~ these, ~uh,~ stateful entities or actors, you can think of them ~kind of~ like that. ~Um,~ so they're, ~you know,~ staying alive for a while and you may wanna ~like,~ deliver messages to them or get information out of them. And ~so, um,~ in order to do that, we provide two separate.

Functions mechanisms right now. ~Um,~ one are called signals and that's a way to send data into a workflow like, ~um, you know,~ send events to a workflow, ~um,~ but in a single direction. ~Uh, ~and then we separately support something which are called like workflow queries, ~uh,~ that allow you to extract data in a read only fashion from a workflow.

And ~so, um, you know,~ if you have a running workflow and you wanna have ~like~ a request [00:32:00] response style, Interaction with that workflow. ~Uh,~ today, what you need to do is first send a signal, ~um,~ and like also make sure that within the payload you give in the signal that there's some sort of like correlation id.

~Uh,~ and then separately you need to query that same workflow after you've sent the signal. And like then get whatever the extracted result ~that~ that, ~you know,~ event caused the workflow to, ~you know,~ generate is, ~um,~ out of the query. And ~so, um,~ first of all, from a developer experience point of view, this is not fun.

~Um,~ like even if it works, it's just. Not an ideal way of approaching this problem. You have to be polling with this query to make sure that you get the result back from your workflow. ~Uh,~ and also from like a performance and usability point of view, it's very limiting because, ~uh,~ the latency of making two separate calls is much higher than if you could do it in ~like ~a single round trip.

~Uh,~ and so you end up, ~you know,~ like basically limiting, ~um,~ temporal for a lot of use cases, ~uh,~ visa latency is prohibitive for doing say something like, ~uh,~ realtime payment processing. ~Uh,~ and so there is like a real company that sort of. Inspired us to, to prioritize this feature that is trying to do real time, like swipe a credit card, run a workflow because of it.

~Uh,~ and they're like, look, the latency ~is,~ is too high. Like, how can you guys,~ you know,~ help us [00:33:00] solve this? And~ so, um,~ the idea that ~we,~ we had and we've had for a while is this concept of workflow update. And so the idea is that it's a request response style primitive for a temporal workflow, ~um,~ where you can basically send an event and synchronously wait.

For the result of whatever that, ~you know,~ event, ~um,~ changes in ~the,~ the workflow state to come back, ~um,~ before you basically, ~you know,~ finish the call. And ~so, um,~ this is something that, ~you know,~ is coming, ~uh,~ it's coming in 1 21, which, ~um,~ at least for the open source, we have ~like~ an initial buildup now. ~Uh, and,~ and that, ~uh, you know,~ workflow update, it massively simplifies a lot of, ~you know,~ communication patterns that users.

Tend to do in temporal, ~uh,~ and also enables this entire set of use cases where you may be latency sensitive and wanna use temporal for something like, ~you know,~ doing, ~uh, ~payment processing. ~Um,~ so that's the first one. ~Uh,~ the other one, which is also as part of that same release, ~um,~ which I think is. If I had to guess, it's the biggest single barrier of entry to like succeeding with Temporal today, which is workflow versioning.

~Um,~ and so the idea is you have these like workflows in temporal, ~um, you know,~ they're running potentially for a very long time.~ Uh,~ and you have this issue where you wanna update the code of ~like~ a running process. And that in itself is like a weird thing because most [00:34:00] software you don't really think about, ~like,~ I wanna update a running, ~you know,~ piece of software.

~I usually think, well,~ Yeah, it is funky. ~Uh, it breaks,~ it breaks my brain still, and I've been thinking about it for ~like~ three years straight. ~Um,~ and ~so, you know,~ right now we have a, a solution for this need to version your workflow and ~like, you know,~ change the code of a running piece of software. ~Uh,~ but it is ~like,~ if it, tenal is like a very, ~uh,~ complex and computer science, ~you know,~ nested thing.

This is ~like.~ The temporal of temporal, it is like one of the most ~like~ complicated, ~like~ hard to grasp aspects of the entire product is ~like~ the, ~uh,~ legacy versioning, ~um,~ which we call in workflow versioning. ~Um,~ and the reason we call it that is that, ~uh,~ the way that we basically, ~you know,~ have solved versioning historically for users is that if you have a piece of code running and you wanna up.

Data it, ~um,~ you literally have to instrument your code to basically say, Hey, if you started running and you were started running on this version, take this branch ~like~ literally with if statements and if you were started on, ~you know,~ this version, take this branch. And that way, ~like~ within the code, it can handle whether the process was started with ~like~ one version of the world versus another.

~And so, um, you know,~ even if you understood what I just said, it is. Even harder to follow when you're actually looking at the code and trying to understand ~like,~ which of the stuff [00:35:00] matters anymore. ~Like~ which versions do I have running? ~Uh,~ and so it is just seriously one of the biggest headaches and frustrations for people ~who,~ who adopt the technology.

And so,

get diffing running code.

You are basically to get e Exactly. Yeah. It is, ~uh,~ it is wild. ~Um,~ it is pretty cool. ~Like I think even as a person who, like, if I never had use it, I would like to learn about it and then like stop and not have to use it. But, uh, it is, it is deeply interesting. Um, and ~so what we've done is we've,~ um, you know,~ inspired by a lot of people in the community,~ including some, some folks from Datadog.~

~Um,~ we are now, ~uh,~ Providing a second approach, ~uh,~ it doesn't replace because there's some cases where you really do want that legacy versioning. ~Uh,~ but in a lot of cases, in the most cases, ~um,~ this new versioning will be ~the,~ the default choice. And the reason it's so much better is that what you realize when you spend enough time developing with temporal is that while some workflows do really need to be changed in flight while they're running, ~um,~ most of the time it's okay for you to let your workflows on an old version finish before ~you know,~ you start workflows on a new version or, ~uh, you know,~ like ~kind of ~doing a more traditional drain that you would.

Find like a distributed system. And ~so, um,~ what this new, ~uh,~ workflow versioning approach, ~um, like,~ takes what ~the,~ the approach it takes is that it says, okay, ~um,~ when you wanna make an update to this workflow, we're gonna go ahead and ~like, you know,~ make sure that there's a worker running and ~like, you know,~ ability for it to receive these new,[00:36:00] ~uh, you know,~ workflow tasks on this new version.

~Um,~ but until there's no old versions of the, ~you know,~ previous workflow code running. We're not gonna get rid of these, ~you know, like, um,~ basically older workers and we're not gonna, ~you know, like, um,~ touch anything about those old executions. And ~so, um,~ now instead of trying to ~like~ change the code of a live workflow, we're just saying~ like,~ you just need to sequence, ~you know, the,~ the versions of your workflows and say, okay, like this is the legacy version.

We're not gonna start anything more on this. We're gonna let them run and finish. That way we don't have to mess with them in flight. ~Uh,~ but now any new workflows to get started, we're just gonna automatically start in the latest version and ~like ~so on and so forth. Forth and you can continue doing that and have, ~you know,~ quite a few versions running at a single point in time, ~uh,~ if that makes sense for you.

And ~so, um,~ it just generally, you, you don't think about it like almost at all, ~um,~ compared to what you were having to do before in most basic cases with temporal, unless you're running a workflow that runs forever or something like that, ~uh,~ this new versioning will just allow you to make changes and not really have to think about the way that, ~you know,~ you're, ~um, you know,~ changing the workflow code as much, which is~ really, really,~ really nice.

Right? Cuz before you had to think about the atomic pieces of your workflow code and now you're just changing the damn thing and you're saying, [00:37:00] Hey, this, ~you know,~ run the o What's your overlap policy? ~Well~ think about schedules, right? ~What's,~ what's your overlap policy here? ~How,~ how are you gonna. How are you gonna run?

Stop that. So that's fantastic. You can ~like~ have a zero downtime. Realistically, you could ~like~ switch over to the new version.

Exactly. Yeah, that's, yeah. And we're, a lot of the things that we're planning to, ~like,~ expand and build on top of that are things like, ~um,~ what we call, ~you know,~ safe rollout. ~So, um,~ when you deploy new version, what we can even, ~uh, you know,~ potentially do is take some of the. ~Uh,~ traffic that is, ~you know,~ reaching your, ~um, like, you know,~ workers, ~um,~ and basically put those on like ~a,~ a separate, ~um,~ set of infrastructure workers, ~uh,~ that isn't actually ~like, you know,~ doing the real, ~uh,~ work for your workflows, but it's just testing to make sure that, ~you know,~ the workflows are compatible with this new version, ~uh,~ that you've ~kind of~ deployed out.

And so what that would allow us to do is say you put out a new version, ~um,~ before you even start actually directing live traffic to it, we can ~kind of~ run this shadow traffic on it and say, Hey, ~Um,~ does this break the compatibility or does this create any problems with what we expect from, ~you know,~ this workflow execution?

~Um,~ if it does, it never promotes that new version to be like the latest version, ~uh,~ lets you ~kind of~ go and fix the changes before it actually rolls out. [00:38:00] And if it looks good, it just automatically rolls it out for you. And then you don't even have to think about things at that level either.

One, one thing I love that you guys are doing is you're keeping the primitive of the workflow and then building on it. And for the community too, it just makes it digestible and palatable to build because it's a men, this is a mental model that takes time to build. It's not something that you just read and overnight you're good with.

So ~it's,~ it's fantastic to see the team. Moving in that direction. ~We're like already over time and I wanna talk to you more Ryland. So that means we're gonna have to have you on a third time in, in the future and learn about the new updates that are happening. But until then, if people wanted to keep up to date with you and your musings, do you write, do you blog?~

~Do you tweet mastodon? Where? Where~

~I should, oh, yeah. I don't even know. There's like a, a great schism in like social media, at least the ones that I use right now. So I don't even know where people, people exist anymore. Reddit's going away. Twitter is in, in, in, in, on fire or something. Um, so yeah, I generally, uh, I don't. I don't actively blog all that much right now.~

~I do have a Twitter, uh, tale logs, um, but most of the time you can see me from the stuff that I post through, uh, temporal, either on my LinkedIn or on Twitter or on uh, you know, like the various, uh, platforms we have, like blogs in the community, um, you know, community forum for example. Uh, so yeah, those are the best places to find me.~

Ryland, it's been a pleasure having you on. Thanks for taking the time talking to us about temporal, not only the basics, but what's coming. Cause ~that,~ that's exciting. Especially the workflow version.~ I,~ I can't wait for that one for my brain. It'll be great.

My brain too. I appreciate it. ~No, it was, it was lovely.~ It was a great conversation. I appreciate you having me.

Thanks again.