Anna: So today we are talking with Dan Robinson, a research partner at Paradigm, and also someone who co-wrote a recent blog post called "Ethereum is a dark forest". It's been doing the rounds. It's about the mempool. And we're going to be talking a little bit more about that. So welcome to the show, Dan. Dan: Thanks for having me on. Fredrik: Yeah, Welcome to the show. Great to have you I guess before we dig into the mempool stuff, which is an interesting story, I think that, that looking forward to, let's talk a little bit about what you do, what Paradigm is. So, you know, what is a crypto VC, I guess, and what is unique about Paradigm? Dan: Sure. So paradigm is a crypto asset investment firm. And here, I should mention that I'm here speaking and representing my own views and not those of the farmer of the fund and nothing that I say on this will be investment advice. Paradigm are a crypto asset investment firm, and we act kind of like a hybrid between a VC and a hedge fund in both making private investments in the cryptocurrency space and buying public tokens and holding them. And it's like bitcoin and Eth. We are entirely crypto focused, so everything, not only is every investment that we make crypto focused, but we really sort of everyone on the investment team tries to be as technical and sort of as deep into it as possible. And we think that gives us an edge in terms of the kind of investments we can make Anna: What makes it the crypto VC versus a traditional VC? Whatdoes that mean? I know you mentioned. Okay. So you're also partly a hedge fund, but just on the crypto VC side of things, how is working in this space different? Dan: Yeah, so this, this is my first job as a VC. Previously I was I was a lawyer actually, and then I became a crypto engineer at a company called Chain, which was acquired by Stellar. So I can't necessarily compare it to what it's like to be a generalist VC. I think it would be pretty different in that a lot of what we do revolves around very sort of technical, either due diligence or value add. So the reason I think we were so focused on crypto in addition to thinking that it's a really exciting space for investment is just that personally for that, for the whole investment team. So the firm was founded by Matt Huang from Sequoia and Fred Ehrsam who was one of the co-founders of Coinbase, and we've got a few other researchers and investors on the team now everyone is just sort of personally passionately in the crypto. And we think that that kind of obsession and that, and that amount of technical knowledge kind of gives us an edge when investing in these companies and helping them succeed. And that's something where a generalist investors, they sometimes invest in crypto stuff, but they sort of don't know what to make of it. Fredrik: Sort of a non-sequitur, but it's something I've always been curious about in your your intro triggered it for me, who needs to say that they're not giving investment advice? Because I always feel like it's either people on YouTube who aren't investment advisors and feel like they seem more legit by saying that they're not giving investment advice or it's people who are investment advisors and like, wants to make sure that it's like, this is not investment advice. Dan: Yes. In this case, that's the latter. So it's Paradigm which is a registered investment advisor. I do think in general, it's kind of like when someone launches a smart contract and then it says your smart contracts are all insecure, you're going to lose all your money, this token, one of any value everyone's like, Ooh, that means I really should invest in it. So I worry about the, about just sort of the reverse psychology of it, right? Anna: Yeah. Georgios who works with you was on the show also said it recently. And we, we haven't said it. I mean, I'll say it now to anyone who's listening. Nothing we say is investment advice, I hope that's obvious. Fredrik: I wonder if we can really give, get in trouble. Like if I were to give someone investment advice, like on this podcast now that I ever have, but if I did that, could I actually get in trouble for it? Dan: That's a good question. If it was, if it was fraudulent then yes. And I think a lot of what people, when people do that is they're trying to cover their own ass in case something they say is wrong either legally or I'm not, I'm not actually sure about the law around. Yeah. I mean, there's not really a law that says you have to have a license to give, to give investment advice, like legal advice, but it's but yeah, I think they're trying to cover their own asses. Anna: So I think now we should move on to the topic at hand or one of the topics we wanted to cover in this episode. And that is the article that you wrote called Ethereum is a Dark Forest and you wrote it and published it like a week after I think Dark Forest version four was released or like right when it was coming out. And I obviously, I mean, good, good piggyback there, but tell me in your case, you're using the term Dark Forest. Like it's, it's a similar, I don't know how to describe it. You're describing something vaguely similar, but as quite different from the video game zkga.me, Which by the way we've had on the show a few weeks ago, if you haven't heard it, we'll put the link in the show notes. It's great. Dan: Yeah. So I think we both, I've talked to that to the Dark Forest guys, and we're both just fans of, I guess, Chinese science fiction or particularly this, this one series and the whole series is great. I recommend it: The Three Body Problem. The whole series is a very rich sort of fertile ground for metaphors. And there's a lot of other ones that I use in day-to-day conversation that are actually poor obscure, but the "Dark Forest" metaphor, which is from the second book in the series is a really powerful one for thinking about these environments like crypto, where it's kind of a war of all against all. So Dark Forest and the context I was using and which is also the inspiration for the game is an environment in which there are such advanced predators out there, such monsters out there that if you're visible, you'll be kept, almost guaranteed. And so the only solution to this is to try to hide. And then if somebody wants to destroy you, all they have to do is reveal position publicly. And so in Dark forest and the game, they depend on these snark proofs, right. To, and the sort of the grinding to basically have private information about barriers of space. And so that's a way to basically maintain this fog of war without which otherwise you're just, you just sort of have like constant, constant battle. Anna: Totally, by the way, did you play this version? This last one Dan: I have not yet played Dark Forest. So I'm really looking for, I have some friends who just spend all their time on it now. Anna: So I, I did it. I did play last weekend for the first, I mean, I had heard about all the other versions and I had not yet jumped in and then I did. And then it turned out to be a dangerous game for me because I love things like Civ and I love map games. I love resource management games. I have been addicted to like these grinding games, these games where you have to like, you know, mine something for a long period of time and then wait. The timing. And then on top of it built with the ZKPs. And I think, I don't know how many people were playing maybe 400, but like it's the zero knowledge proof community mainly there. So it was amazing. And except I only played like for right near the end, so I didn't, you know, rank, but next time anyway, you guys should totally try it. Fredrik, have you tried it yet? Fredrik: I am not. I have not gotten the time, but I I'm. Yeah. Like it's one of the things, you know, obviously I want to play it, but it's also one of the things like I want to have the time to play it because I know that once I start playing it, I really want to like hack around probably by a super powerful machine, start grinding out hashes and like actually just go gung ho on it. But I, I can't really do that. Dan: One of the best things about my job is that I will be able to credibly say that I'm playing this game for research. Anna: I felt the same way. Very cool. Now, going back to what we were just describing, you know, you're using the term Dark Forest based on this book to describe the Mempool. And we did do an episode on the Mempool, which is quite technical. We'll put a link to that in the show notes, maybe worth listening to, just to get some context for what we're talking about here. And we want to kind of hear the story of what you tried to do to basically like, you know, save, save a transaction, or save someone from losing a ton of funds. But before we go into that story, let's do a few quick definitions. So what is a Mempool? Dan: So the Mempool on Ethereum or on any blockchain is the set of transactions that have been published. And they're known by all miners or stickers or whoever's constructing blocks, but haven't yet been confirmed into a block. So the Mempool is a technical detail of how nodes are implemented, but it's also a necessary one, at least naively, because you don't know where the next block is going to come from. So you have to basically tell a lot of people in order to make sure that your, that you're a transaction gets included. And so one of the known issues with the Mempool is that your transaction is public and it's possible for somebody to create a transaction that gets in before your transaction, with the knowledge of what's in yours, that's the Dark Forest environment that Anna: There's a way for someone to front run it or someone to take what's in that transaction and sort of use it themselves. I don't actually understand how that would work. Like I do know that there's, you could maybe kill a transaction maybe by like throwing up the same one with the same nonce. I don't know if that actually works, but like with a higher gas price, but like, is there, like, how would you take what's in it? Dan: So by default Ethereum miners order transactions in a block based on the gas price and descending order of gas price. And so if you see a transaction that's pending, someone has published it. It's an it's in the Mempool. And we use the term the Mempool, although of course, every, every node has their own as their own individual one. But basically by that, I mean, it's just out there in the world waiting for someone to add it to it. And everybody's knows about it and is waiting to be added to a block. Once it's out there, somebody can submit a transaction with a slightly higher gas price. And then if, if a miner sees it, they'll give precedence to that transaction. And so for most transactions, possibly this won't matter. So for example, in a Bitcoin transaction, if I'm sending Bitcoin to somebody, there's no way for you to get a transaction submitted early. Dan: That takes advantage of that, or that prevents mine from being executable. And this is true for a normal Ethereum eth.send as well, is that, you know, that's my, if you can't, you can't actually stop me from doing that. When it becomes an issue is when there's more complex, what's a term called miner extractable value, which is when you have something sort of going on in this transaction where somebody knowing that you're doing this means they can get in and cause you to get a worse price on a trade or possibly cause a transaction to fail. And the fact that you've published a space that gives them valuable information and doing this. So the classic example is front running on Uniswap. So if somebody makes it really large trade on Uniswap, you can do, what's called a sandwich attack on this trade, where you put in a transaction before, which pushes the price up. And then, and then they end up buying it a worse price. And then you're putting a transaction afterward, which sells and basically locks in your arbitrage at a higher price because, because the user has pushed the price up for a large enough user trade, you can make a profit just sandwiching someone's individual trade is a pure arbitrage profit. Anna: And because it's public, you can actually see the transaction. You can see what they're trying to do. You can look at the price right now, the way to do this would be to do something, to push the price up and make the gas more expensive so that it happens earlier? Dan: On your transaction. You set, you submit the transaction with a higher gas price. So you choose the gas price to have your own of your own TX. And so, yeah, so you just basically submit a transaction that has a gas price that ensures that better transactions included before the one you're trying to attack. And then you include one that's after. Anna: Are there tool like, is that happening? That must be happening all over the place. And I can't imagine that it's like someone individually going and looking at this thing, but are there bots that just do this? Dan: This happens all the time. And I think on Uniswap, it is one of the most popular applications on Ethereum. Aren't good. They're generally, there's a lot of, really specialized ecosystem around Arbitrage and Uniswap specifically. And this isn't just doing like the sandwich tax. Dan: It's also getting the first trade in the block after the price has moved in the real world. So you can get an arbitrage on it or arbi between Uniswap and some other on chain trading trading platform. But there are all kinds of bots out there for extracting all kinds of MEV Dan: But then there are other ways in which, in which it can be a bad thing. So these are, there's a lot of, you know, a lot of really smart people. I kind of picture them as sort of like Russians or just sort of in a, in like a, yeah, like a basement somewhere, just sort of like hacking on this. Like you don't, you don't have to be in the ecosystem. I probably don't know the best black hat hackers are meant for people because yeah, I'm sure they, and they, I think they made crazy amounts of money different from what I can see on that and what's happening in the Mempool, but there is a more generalized form of desk. This, this is what I find really fascinating on Ethereum. There's a particular strategy you can do, that doesn't it for a particular type of MEV. It can like always take it on any contract and the programmer doesn't even have to have any idea what the contracts are doing. That could be a new contract with new code by, for particular kind of MEV. This bot can detect it and pick it up. And this was what, this is what led to this story that I, that I published with Georgios. Not that a half ago, where, in which we called Ethereum a Dark Forest. Fredrik: We have an episode, a previous episode that explained sort of the technical aspects of Mempools. But back then, I mean this, this was years ago, this episode, and even back then, we were seeing something like a thousand transactions per second, just from bots, outbidding each other. Right? So that's also an interesting thing is like these bots get into war with each other. So like, if there is miner extractable value, then one bot will detect it and inject their transactions. And then another bot will detect it, say, I'm going to bid one way higher gas than you and inject their transactions. And then the bot replaces there. So like a node is actually turning through thousands and thousands of transactions per second, sometimes just between like two bots fighting each other to get the final thing in. And then eventually one bot gives up because they say, well, this is not profitable enough for me anymore. Dan: That's right. So there's this thing called the priority gas auction, which is when a bunch of sophisticated parties see some MEV or one of them sees it and has triggers other ones to start to go after it. And, you know, I don't think I'd be able to successfully participate in that in a priority Gas auction. I think these are highly specialized predators here, figuring out how to, how to win these against, against other bots. Absolutely like them and how to minimize their costs if they lose. Yeah. So that's part of what makes a Mempool really scary, I think, is that not only are these, are they picking off individual users, but even they aren't safe from other, from other predators. Fredrik: And there's another interesting angle too, is I would assume, at least, that basically all of them use gas tokens. So they're not actually paying the same gas price as you are. So they have a completely different playing field as well in how they engage in these transactions. Dan: That's right. So they can, they can set a high gas price on their transaction, but they don't pay all the gas that the transaction would cost because they're exactly they're redeeming a gas token in the same way. Anna: And by gas token, do you mean like gas token the thing, or is it a type of thing? Dan: There's a few, there's a few instances of gas token and the original one was designed by Phil Daian. Who's also one of the researchers who coined the term MEV and is one of the experts on the Mempool and front running and all these things. But gas token, for those who don't know, it's a trip that takes advantage of how Ethereum measures gas costs to be able to basically to get a refund on gas at exactly the time that you want it. So you can basically prepay for gas by storing some junk state on Ethereum blockchain, and then later get the gas back and the transaction where you want to use it, potentially one with a much higher gas price. Anna: Is it kind of like an averaging product? Like, it sort of means that gas prices won't spike, but you may sometimes be paying a little more or you always paying less. Dan: You typically probably wouldn't use a gas token. If you could buy one at the, at the same price that you'd be paying for the gas. So it's sort of like almost an option on gas at, at a particular price, right. And you haven't timed. So one way to, to think about it and imagine pricing it. But I think it does probably tend to have a leveling effect on the price of gas. So I'm not sure if that's played out in practice and because it's mostly used by these, by these weird games, like possibly could have the opposite effect. Fredrik: I think it averages the, the box costs for gas, but for the general public is probably not that big of an impact. So I think we've, we've walked through the basics here and the primitives. So tell us what the experiment that you did was what was the, what was it that you did in the blog post and, and what did you find? Dan: So this was a situation that I found myself in which I'm not usually in because I'm, I'm mostly not a white hat hacker. I don't spend a lot of time trying to break other people's contracts. So what happened was I heard about somebody having sent Uniswap liquidity tokens into the Uniswap contract itself. And this is one of the most common mistakes that people will make on Ethereum is sending ERC 20 tokens to the token contract itself or to some other ERC 20 contract or some contract where they can't recover it. And so this was basically a fat finger error. And usually this is unrecoverable, but I realized, after thinking about this for a bit that in this very particular instance, the tokens that they'd lost, who were worth around $12,000, could be recovered, not just by them, but by anyone. There was a function on the Uniswap contract that they could call, which would actually withdraw the liquidity for them and give them all the, all the liquidity that's worth $12,000. Dan: And since anyone could call this, I could call it and I could just be a white hat hacker and, and pick up this money and give it to them. But I knew because of the nonsense that goes on in the Mempool, that this actually wouldn't be that easy. So I was lucky that I I'd heard about this from Phil Daian a few years ago, the same researcher who created gas tokens and MEV. He told me a story over beer about an encounter with something that he called a generalized front runner. So a specialized front runner is the kind of front runner that we were talking about before, which is written by a programmer to sandwich who trade on Uniswap and make a profit. And this depends on some human intention around looking at it mechanism, figuring out that it's, that it's broken in a particular way and constructing a pattern matching bot that will automatically take some action in response to saying something. A generalized front runner, doesn't just look for one big of the kind of MEV. It looks at every transaction in the Ethereum Mempool, and then simulates what would happen if they sent that transaction instead of the user. Anna: Wow. And I guess for most of them, they're for most of them, they're just like, this is useless. It wouldn't do anything? Once in a while, right? Dan: So for most transactions, this will fail for some transactions that just run this and they see, Oh, my ERC 20 balance in this token increased and I didn't pay anything and it was worth, it was worth it. You pay for gas. And when this happens, they just front run your transaction. I just submit a transaction with a slightly higher gas price that gas is generally guaranteed to get in before yours. And so this is only an issue for a very particular kind of MEV. And this is where there's like a function that anybody can call. And that's, that's not every kind of MEV by any means. You know, it's, I probably have MEV. But it was the situation I was in because there was this function on, on the Uniswap core contract that anyone could call and get this money out. Anna: Do you think though, is it, is it always mistakes? Like those things that one could front run, is it always an error or is it sometimes purposely? Dan: I would think it would usually either be a mistake or resulted mistake as it was here. Yeah. Or a bug in a smart contract. So like relatively often when there is an exploit, one of the problems is that exploiting it yourself could potentially get it front run by something like this, because it's in the nature of an exploit, not every exploit, but in a lot of exploits that anyone can do them. Yeah. It would probably be a bad idea to design a mechanism on chain. That really depends on people not being able to do this because it's, it can, you know, anyone can call this, could've called this function at any time and pick up the money. So you shouldn't really design it so that like anyone can pick it up because then anyone might. But the funny thing is, in this case, the money had been sitting there on chain for eight hours, before I had this thought that I, that I could go pick it up and I went and checked and it was still there. So there was this function, the documented function on the contract. Anyone could just call and get and get the money out. No one had. But I knew that about these generalized smart contract frontrunners in the Mempool. And as a result, I knew that if I tried to pick it up, we're just like transaction sending to it, it would get snipped. And so the Mempool is even scarier than the Ethereum smart contracts state, which is pretty scary already. Anna: Wow. Okay. So you, you were there, you saw this problem. You realize there's a solution, but then kind of, luckily also realized that if you just went for it, blindly, someone would see it and then front run you on you attempting to save it. Fredrik: Yeah. It's sort of like, like you as a human, see this, right. But the bots only like they, they're not going around analyzing contracts. They're not actually smart. They're not actually finding these things. So no one else was looking at like the actual contract and the, this in-depth, but you have a bunch of these predators are just looking for like, how are you moving? And then based on how you're moving, like determining where you're going and then going in the same direction. Anna: Damn I'm so seeing some similarities now to the Dark Forest game anyway... Dan: Exactly, exactly. And so it's a really fascinating situation, but then it was, it was somewhat scary. One to be in, there was also a ticking clock because if anybody removed liquidity from this pool, the normal way, then either they would get the liquidity that they'd get this money out or the front runners would see that would see the call and they would act, and the front runner would end up hitting it out either way. It wasn't likely to get back to the person who accidentally sent it in. So in order to try to recover this money, I talked to some smart contract security researchers, including Sam Sun, Scott Bigelow, some experts on the Mempool and on smart contract security. And we formulated this plan, which should involve basically a lot of unnecessary steps, needlessly complicated scheming, to try to hide from the bots, the call that we were actually trying to make. Dan: And so I obviously couldn't call the contract directly with my transaction, but I also couldn't just put up a contract and call the contract, that would, that would make the call. Because, you know, you wouldn't necessarily be able to tell that just by looking at my transaction, that I was even interacting with Uniswap, but if you ran my transaction and simulated it, you can see every internal call that I make. And what these bots can do is run the transaction, do an execution, trace of it and simulate, not just if they sent the transaction, what would happen, but if they did every call internally, what would happen? And so that wouldn't be enough obfuscation on its own. Basically they could extract that I had done this call, simulate it, and then just do a transaction that makes the call doesn't even go through my contract. Anna: And there's a belief that these generalized bots would be doing that they would actually run every call. Dan: I didn't know exactly whether they would, but basically in this, in this scenario, we had to sort of think like an attacker and say, okay, if we built a bot like this and we had a lot more resources and we're a lot smarter, like these people are, this is how we would do it. And so we figured that wouldn't be enough to prevent, to prevent this, this call from being, from being front run. Our step actually involves deploying two smart contracts and splitting the transaction that activated them into two transactions. Would you try to submit into the same block? And the idea was hopefully a generalized front runner would simulate these transactions separately. If they simulated them in, in sequence, then they would see that the second one actually did the rescue call. But if this is unrelated, then separately then the transaction, the second transaction would fail and they wouldn't act without calling, the call to Uniswap. So that was the hope is that the bots weren't smart enough to actually detect us. Anna: And was the, the individual, like the first entity calling, it gets your address or something like, did you also split that or did you just split the contracts? Dan: Yeah, we called from two different and externally owned accounts. Got it. And made these calls to two separate contracts. One of which would activate the contract that would actually do the rescue and the other, which we would call a contract that would then call that contract, which would then do the rescue. And we hoped, you know, these transactions don't look at all related. The different gas prices and everything, but the different centers called different contracts, but yes, if you ran them both, then you would see this. So unfortunately we didn't manage to get both transactions in, in the same block. The strategy was not itself, fully tested. We didn't see whether it would have worked if we'd gotten both transactions into the same block. The second one ended up in the next block, or maybe a couple of blocks after that. Anna: I can understand the architecture of the action that you're trying to perform, but like why in the same block? Do they have to be sequential? You don't want anything in between or...? Dan: Well, so what actually happened was we got the first transaction in one block, right? But as a result, the second transaction, when run against the current blockchain state, when some generalized front runner was trying to look at what it would do, the second transaction would make that internal call to Uniswap. When they, when they ran it against the current stage. And so the reason we wanted it in the same block was on the hope that the generalized front runner wasn't running all of the transactions in that block, which would repeat, which would be somewhat more computationally intensive, the running all transactions of the Mempool in sequence or constructing candidate blocks and running them and seeing what happened in each transaction in them. We were hoping they were running the transactions individually. So this didn't work. We actually got front run. Um... Anna: Which part got front run though? Speaker 3: The second transaction, the second transaction was basically, it was a naked call. It was, it was routed through a two contracts. So it was, it was like a nested internal call. But nevertheless, when you ran the transaction against the current blockchain state, it resulted in the Uniswap burn call being made the call that could, that could be used to pick up the free money. And so we were basically sitting ducks, would that transaction in the Mempool by itself. And so the transaction did get, give you a front run. Some crazy bot came in and burned a bunch of gas tokens and picked up the picked up the money. Anna: Did your first one go through correctly? And then it was the second one that they front runned and then they, because the second one was the one where you actually got that. Okay. Bummer Dan: That's right. So, you know, this was a strange experience because while we were, we were spending like hours trying to, you know, write this contracts and, and write the scripts that would submit them and test it. And this whole time, you know, we, it felt a little like being paranoid or like, we were like, why are we does it actually makes sense for us to be, to be doing all this? And we knew intellectually, this is a huge risk that, that, that this, that this would happen, but it wasn't until it actually happened that you could really believe it, you know, like in the, in the beginning of "A Quiet Place" where the kid gets eaten by the monster and like in like the flashback or the, I should have the, for yesterday. So I think the reason they do that in the movie is otherwise everyone's going to look really stupid, being super paranoid about saying anything. Anna: I don't know. I don't know the reference, but... Dan: Okay. This is a movie where aliens are on earth, that, that detects sound. And so you have to be really quiet and you can't talk. So there's like very little dialogue in the whole movie. But like that it's a situation like this where basically being detected means death. But until that happens, you can't see the monsters around. You just sort of know they're there. Anna: Wow. And I guess you, were you not sure exactly. Like where are you exactly sure. On how the bots work, is there a deep understanding about how these work, or is it sort of a mystery monster, like a big unknown Dan: When this post was published? I think it surprised a lot of people. And it surprised me when I first heard about these bots before, because this is a case where this end like White Hat hacks are sort of the cases in which this particular kind of MEV up and where this kind of generalized frontrunning happens. And most people don't encounter that on a daily basis. There's there was some lore in the white hat hacker community around what these bots could do. It's not all public, they're not even everything that's known about how they work is public in part, because that would leak alpha, basically where if White Hats actually do after one of these transactions included the tricks that would work. In this case, because our trick didn't work, we felt comfortable publishing, publishing and thought that it was worth scaring everybody. But we've, we've considered and we actually have, we have a research budget that hasn't yet been used, to kind of map out this Dark forest and figure out what these bots can actually pick up on and what they can't. Fredrik: It will be interesting to see, because like, it, it's sort of the same thing as in the hedge fund world, or like the general trading world, where there are a bunch of open-source bots that can do automated trading in, in various ways, but they get published because they don't work anymore because they're not profitable anymore. And then, but you know, that whatever a real hedge fund is using is this at its basis. But with some variation that now makes it profitable again. And so you kinda know what the scene looks like, but you don't know exactly. You don't know what exact models they're using, because then they wouldn't be profitable. Dan: Right. And to be honest, I think this obfuscation tactic is doomed. There's just going to be no way that, that just even like with a sequence of different transactions and hash precommits and all kinds of ways to get it, to get in the way of these being detected, it's ultimately it's unsolvable. And like the bots are going to win this arms race. And the reason you can say this is because for our rescue to be successful, the transaction had to be included in an Ethereum block that made this call at some point internally to Uniswap. And that is the necessary condition for us to, for us to win. Right. And as a result, because we were spending this to that, to these miners Mempools, some miner had to be able to construct a block that did this, and if a miner can do it, then the attacker can do it. And so if, if these, you know, I'm not sure whether they have enough computing power right now to really be like, to be, to be constructing blocks fast enough to always, to always get this. But that's sort of inevitable. Fredrik: Yeah, I think so. Like it, they'd certainly pretty close. So, I mean, going back to what you were running your, your whole thesis was that if we have these two separate transactions and they both get included in the same block, the theory is that they're not constructing and executing a whole block because then again, they would see that call being made, but just running one transaction against the current state. And, you know, if they did this for each of them individually, they'd run the first one, see, Oh, this only activates a contract. It doesn't give them, give me any money. The second one, Oh, it just calls this contract. And nothing happens from that. So it doesn't give me any money, but constructing an executing, a whole block takes on the order of like a hundred to 300 milliseconds. And the block time is like 15 seconds. So you have a lot of time to construct and execute full blocks. And this is with like mining clients. And there are many clients and, and like EVMs that aren't in production, but are way, way faster. So if you were like on a hyper experimental, like, I just want to simulate shit, you could do it way faster than that. Probably down towards like 10 to 50 milliseconds that gives you a lot of simulated blocks per like block round chance. Dan: That's right. And you can imagine this running as like a cluster of, of huge high, high memory nodes on, on AWS. Like there's, there's really sort of no limit on how parallel, for example, this could be right. And so, so that's, that's why I say, like, I don't think, I don't think there's a way actually to obfuscate your transaction when ultimately at some point an internal call has to be made that is that you're trying to hide. There's no way to hide it because ultimately yeah. They can just do whatever a miner does and do it maybe like 10 times in parallel with different combinations of, of transaction to the Mempool, who knows. And I think that ultimately is why that's a dead end, the way to do it, or right now the cutting edge on this is submitting the transaction directly to a miner. So it gets included in a block without ever touching the Mempool or being published to anyone else. Anna: Yeah. I was actually, I was going to ask, like, if the miners run this themselves, then that would skip this problem, I guess. I don't know if that's actually how it works, but like... Dan: Yeah. So, so, okay. So yeah, so there's, so first I'll talk about, I think White Hat miners and like miners being able to be, to be really helpful in actually being able to do this. And this is something that actually happened subsequent to subsequent to our posts. And then I'll talk about that, the future and why that, why it might be worrying if miners start to essentially try to extract this value themselves. So the best way to ensure that your transaction doesn't get sniped by any of these bots is just not to show it to them. And to have it be included in a block without ever being published to anyone with only being in the Mempool of this, of like a single mining node. So if you haven't to be an Ethereum miner, or you're like close friends with one, you can basically get them to include this manually in there. Dan: And that blocks without propagating it. When we were in this situation, like we didn't know any miners. We, so we were, we had to make do in the, in the public Mempool a couple of weeks later, Sam Sun, the person who Paradigm just hired to do contract security researcher actually found a bug in a Defi contract that put him in a similar situation, except with $10 million on the line instead of $10,000. And the way he got out of it was pulling an all nighter with some other people in the security community and miners to be able to, to actually get this included in a block without ever being published. And so this, this is a great story. It's a, follow-up sort of, kind of sequel to my, to my Dark Forest posts. This post is called "Escaping the Dark Forest". And that's actually what they did. They got a transaction included that skipped the Mempool and skip therefore skipped all these monsters. Anna: Is that actually published? I don't think I saw it. So we'll definitely include that then. Cool. Yeah. Fredrik: This is in general, a very controversial topic because it's, it's been often discussed should miners have like their own private network among, amongst each other, because that would lower uncle rates and we could increase blocks and, you know, a bunch of other things, but then you're obviously centralizing between these, just these miners. Yeah, exactly. Like you're forming this, this strange thing, but in, in an interesting way you can sort of have the best of both worlds if you go proof of stake, because then the validator set is known. It will change by the protocol, but the set is at least known. So you could submit something to just this set. Dan: That's true. Although, although then you have to, so in that case, you have to trust that the stakers that you're submitting it to, aren't going to take the MEV themselves. And this is true of miners too. If you give this to the Mempool, in this case, it was SparkPool and they were honest and publish a transaction in the block without, without sending it to the Mempool. If they're honest, then you're, then you're finding this is a great situation. If they just chose to extract it themselves, there's nothing you can do about it. And so you do have to reveal this to somebody. Now, I think it's a, it's a better situation than having to reveal that to everybody. And generally, I think miners so far have tended to act in a kind of rationally altruistic way with respect to, to not kind of extracting any MEV themselves. In the future, I think that's likely to change in part, because if you're a miner extracting MEV, you can get so much value potentially from that MEV that that can offset your computation costs and make you a more competitive miner. So then miners that don't try and do this can get competed out by those that do. Anna: Oh, no, but wait, when you say miner, like it can't be just one miner can actually do this better. Right. Cause you need a lot of miners to accept this too. It's like, so I don't actually understand, like you have to have a pool. How does it work that you'd actually get these miners to do it? Fredrik: Well, it's always a, it's a, it's still a valid block. So like by the protocol, the other miners would accept it, sure. They could run their own analysis and rejected, but that's, then you're breaking the protocol. Dan: Right. Right. So exactly. So it's one mining pool that we, that, that they submitted this to SparkPool and, you know, they, they mine, I don't know what percent, maybe 20% of Ethereum blocks. So that ensured that this transaction would get included like relatively soon. And then once it's included in a block, yeah, the other miners built on it, same way they normally do. But this is a risk. And it potentially one that could destabilize the consensus process itself because if one of those other miners was really malicious, what they do is they roll back. They fork out your block, they'd look at your block, see the MEV and fork back, built on top of its parent block and include that transaction and extract the MEV from that. And then, and then continue. And this means that potentially you'll have this war happening now not between front runners, but between miners, where they're all trying to fork each other out and all trying to just try to get there. And that, that the, the war between front runners is mostly only shows up for the end user as being like this weirdly high gas costs. And sometimes, sometimes transactions do weird things. This would show up as quite often having, having rollbacks and get that. So that would be quite troublesome and we're not in that situation yet, but it's possible that MEV gets, gets too big without, without other ways of dressing it. Anna: Is there any way to prevent this? I guess somebody's gotta be doing some research in that direction. Dan: A lot of people are, if you are working on this please reach out to Paradigm. You can reach me at dan@paradigm.xyz or find me, Dan Robinson on Twitter, because we love, we love talking to people who are interested in this space and we're interested in making investments. And there's a few of our portfolio companies are working on ways that could, that could potentially that could mitigate this. Fredrik: I think there's a lot of, I've heard that a couple of different proposals. I mean, there are ZK versions of these proposals to avoid front running, but there's also like some, some companies have reached out to us to use like the Parity Secret Store, which is like the miners could deploy a distributed key generation mechanism so that you can actually encrypt the state of transactions, but miners can still decrypt it, run it, make sure that it's valid, included all this stuff. So you're, it's kind of like this idea of just sending straight to the miners, but you're sending it to the public just encrypted and the miners can decrypt it. Dan: So of interest to your listeners. There's a lot of ways that potentially cryptography could be used to ensure to ensure that transactions can't be read by the miners and they won't know what's going on in them until after they've been included in a block. In addition to encryption, another way you could do it is VDFs / Verifiable Delay Functions. If, for example, every transaction itself had to have a VDF on it. That was like 15 seconds, that would mean that nobody would be able to create a transaction in response to one that's already in the pool. If every transaction, it was production had a Timelock encryption on it, that prevented it from being seen for 15 seconds. Then it would be included in the block, potentially not decrypt until after you don't know what the state of the block has until after the block has already been constructed. But these are generally ways of either separating transaction inclusion order from transaction execution or blinding whoever's executing or sequencing the transactions from what they're actually doing. Fredrik: And they all have various trade-offs in terms of like centralization, complexity, you know, UX, if you have a VDF on everything. And like, I it starts getting really complicated. Yeah, it's, I think it's an interesting space. I haven't dug super deep into it, but there's a lot of interesting stuff that you can do there. Anna: Is this an Ethereum problem? Is this a only for Ethereum problem? Is this a problem that every blockchain is going to have, or has? Dan: Bitcoin has mostly avoided this problem, because there isn't all that much MEV on Bitcoin. So if you, like we talked about before, if you submit a transaction, just spending your Bitcoin, no one else can submit that transaction. Nobody else can, can basically do anything to interfere with it. As we start to see more things built on Bitcoin, like Lightning, for example, a payment channel provides a way to get MEV. It requires quite a lot of work for the miners, much more than just front-running a transaction. It requires censoring for potentially a long period of time, but that's one way in which miners could potentially extract value by manipulating transactions in the pool in a way that could ultimately both mess with Lightning and mess with Bitcoin consensus generally. Another way is just the transaction fee itself is a kind of MEV, including a transaction with a very high fee is, is obviously Miner Extractable Value, right? Dan: And actually there's a paper call on the instability of, of Bitcoin in the absence of the block reward, which talks about how transaction fees are the main way that Bitcoin miners are paid. Then it becomes much more profitable to do weird strategies where you roll back other people's blocks in order to steal the high transaction fee transactions that are in them. And that this could potentially, and then, then you get into games where leaving you want, it might want to leave transaction in the pool, so you don't get rolled back. And ultimately there's, there's not really an equilibrium to this. And so it results in this instability of the consensus system itself. So I do think as miners get more sophisticated and competitive, even on, on simple chains like Bitcoins, this would be a problem. And it certainly will be a problem on any smart contract chain that uses roughly Ethereum's model of, you know from that the contracts are public transactions are public in a Mempool. Anna: I mean, I guess this will carry forward on to any sort of ETH2 or sharded blockchain as well. Fredrik: Yeah. And, and I think the, the only thing that we've talked about on this show that I think starts to addresses is like Zexe, where contracts are private. The calls to those contracts are private and like everything done that line, but then yeah, completely different model of programming and interacting and everything else. Dan: It is also in theory, the miner could just say to users, you have to unblind your transaction to us. So the user is capable of unblinding the transaction. So this is kind of a hard problem. Maybe you could do something with deniable encryption. I don't know, but if it uses PayPal, if I'm blinding the transaction of showing the, the miner and the miner potentially extort that out of them, and this is just a really hard problem. Tt's just like game theory and what's in some ways an iterated game, but in some ways is a free for all. And this competitive dynamic... What scares me most the issue that it really doesn't matter how honest and prove work like miners are because ultimately the least honest ones, the most extractive ones potentially will win and will win. Not just like when these races, but also just like, just like out-compete the honest ones. And so that's, that's a very worrying subject. This is all under the general metaphor of Moloch, which is a popular metaphor in Ethereum of basically coordination, failures and competition leading to what ultimately is kind of like, like a really scary environment like the Ethereum Mempool is now. Fredrik: Yeah, because like, even in a, if you paint an extreme scenario where if I'm submitting a transaction to a DEX, then I also have to be running a bot to make sure that that gets in at the price that I want and et cetera, even in that world, the miner still has the advantage and can just like, he doesn't have to pay a fee because he would just pay himself. Right. Anna: Would this also affect POS systems? Cause like their miners are taken out. And I know you just mentioned, you gave this example of like validators, but like has there been, has that been considered in a POS context? Dan: POS is almost the opposite problem where in proof of work, because anyone can show up and compete. That's one of the best properties of it. It's also one of the scariest because it means ultimately that, you know, some malicious party might actually end up being the best able to pay for electricity with their MEV profits. Right. And be able to essentially take over the chain. Proof of stake is is sort of the opposite. It's much more entrenched in that the large owners of stake and those are the they've delegated to ultimately, they're harder to display that's, you know, you can't just have some random party come in with a, with a better strategy and beat them because they don't have all, they don't have all the stake and it's hard for someone new to come in and buy up all that stake from them, They'd have to sell it. Dan: But as a result, this does mean that if ownership of your, of your coins does get captured, that's potentially the whole chain is basically locked down. And ultimately if that, if that turns out to be a corrupt oligarchy, that's running the chain... Anna: Then trust is lost completely, I guess. Dan: Yeah. Yeah. And decentralization is lost and potentially, you know, and censorship resistance. And so I do think in the, in the medium run, proof of stake will likely work a little better than proof of work. And it's because it doesn't have the competition just isn't as fierce, but ultimately, I don't think it's an ideal situation to have basically trust that these, that these large whales have the chain's best interests at heart. Fredrik: No, I think, I mean, they, they still have to be considered malicious. Right. I mean, that's how you always have to build a protocol. You don't have to consider all validators malicious or yeah. Like at a base level. And then you hope that two-thirds of them aren't. But it's still interesting, cause like, I don't know, in the miner world, the miners are known like if spark pool gets a bad reputation, cause they're doing this bad shit. Yeah. You'll have the same kind of effect or like the people who use that pool is going to move away. And so same thing with validators who are getting a lot of nomination, like if they behave badly, maybe people nominate someone else. Although we aren't really seeing that. We're not seeing nominations being that sophisticated. We don't see nominations that have the chain's best interests in mind. And so I wonder if we'll get to that point or like, if it's currently all nominators are just basically people who seek higher interest rate, then their bank will give them. Dan: Yeah. So I think that that, that's one of the big structural risks for proof of stake long-term. It's also a risk for, proof work. And I mean, right now, proof of work partly works because the mining pools tend to be yeah, long-term rational where basically they, they know, you know, this is true in Bitcoin and Ethereum, they know that doing things like, like double spending attacks and rolling back the chain or really aggressive extraction of MEV, could basically break the chain itself and mean that like their, all their mining equipment is worth us. Right. And this is similar for proof of stake where, yeah, there's, there's probably some abuses you can get into at the edges. But if the consensus set gets too dominated by someone and they start expecting too much from it, then it kills the chain and ultimately their coins are worthless. And so like, arguably, this, this has maybe happened to EOS where I think that the perception that block production is, is centralized or cartelized and run, run by small group has meant that the, the chain itself has been less competitive. Anna: So it sounds like this is a space that, you know, there's no conclusions yet. There's still a lot of research to be done and we have to keep our eye on it. It's been kind of fascinating to even visit this part of the blockchain world. I had not spent much time thinking about it until your blog post and even more so now in this, in this interview. To sort of switch gears for this last point, you're in this role where you do research for this VC firm, you're, you're spending a lot of time kind of like communicating this outward. And I wonder, like what your feelings are about that, like, do you find it, is that, do you sort of see that, like this blog post that you wrote, like creating the story out of it, do you find this like something that's important for you to be doing or your firm? Dan: Absolutely. And it's really what made me excited about joining a firm in the first place is, well, first there's this intake step where basically that to a very sort of broad overview, because, you know, we have people coming and pitching us. I'm looking at them out there and always looking for potential new investments,uare developments that are, that are meaningful to our existing investments. And so I is there, I feel like I get this kind of perspective where I get to see a lot of ideas and then process and pick out some of them and then try to communicate them to a broader audience, basically in order to make it easier for more projects to form that are working on these things. So this is a lot of what I've done in DeFi is sort of publishing out papers with like DeFi ideas. And this happened, I published a paper called The Yield Protocol, and then ultimately a entrepreneur came, they wanted to build it and we, and we're actually incubating them now. Dan: And so this has meaningful effects for the fund, but with the MEV post also, we were basically trying to see the startups out there to be like, look, this is a serious issue. It's potentially going to threaten crypto. We think there could be promising investments and potentially really important ones to be made in this area. And so it's kind of a call for startups to do this. And so trying to, to some extent, set the agenda on it or, or get these ideas out there, we think is really valuable. And in these cases much more valuable than like keeping this outside to ourselves would be because ultimately Paradigm really depends on the crypto space working. That's the most important factor for us being successful. And so anything we can do to kind of advance that, including inspiring these new projects is worth it. Even if we don't capture for the gains, even if we don't actually make investments in them. Anna: Do you sort of see though the strength in adding that emotional side of it? The storytelling? The metaphors? Like, it takes it a little bit away from the technical, but I actually think it's incredibly useful. It's not done that much. I feel like it's kind of missing. Dan: Absolutely. And I, you know, I used to read this, there's a fantastic book called the Cuckoo's egg, which is about tracing the hacker in like late eighties, you know, using the computers... "Ghost In the Wires", by Kevin Mitnick, there's like great sort of hacker fiction or sorry, hacker nonfiction. It's actually quite thrilling to read, even though it gets into technical stuff that, you know, you have no idea about, but when you explain it really in terms of this story, you don't really have to. And in fact, it's much more interesting than like the fiction about hackers is where, you know, they got a million screens and they're just like typing on the keyboard right Anna: Numbers blowing down the screens vertically... Dan: And I think, one of the most rewarding things for me of publishing this story, was the people who didn't know anything about crypto at all. Maybe wasn't even interested in crypto even after reading the piece, but thought it was a fun kind of cyber thriller. Fredrik: Yeah. I think it's one of the few crypto blog posts/pieces that I saw on hacker news actually ended up there and get somewhat popular where normally hacker news is a really crypto averse. Dan: Yeah. How can you say it's crypto? And most of the comments were like, "Crypto is stupid, but this is kind of fun." so I thought that I was like, I'll take it Anna: Well on that note, maybe we wrap up and say, thank you so much for coming on the show, Dan. Fredrik: Thank you very much. Dan: Thanks so much for having me. Anna: And looking forward to your next story of visiting some part of the Dark Forest that is Ethereum. And to our listeners. Thanks for listening. Fredrik: Thanks for listening.