Anna (00:00:05): Welcome to Zero Knowledge. I'm your host Anna Rose. In this podcast, we will be exploring the latest in zero knowledge research and the decentralized web, as well as new paradigms that promise to change the way we interact and transact online. This week, I chat with Kristy-Leigh Minehan, previously of Core Scientific, Genesis mining and the co-author of the ProgPow EIP. She now works on Symbol protocol. We discuss her road to crypto mining, mining on Ethereum and how the hardware industry around this developed. ProgPow or EIP-1057 and the subsequent controversy that it generated in the Ethereum community, the challenges of off-chain governance signaling and more. Anna (00:00:54): But before we start in, I want to let you know about the upcoming ZK Jobs Fair! It's happening on July 29th. We'll be hosting some of the top zk projects who are looking to build out their teams and hire. Most of the jobs will be technical, focused on software engineering and cryptography roles, but there are usually some other positions as well. So if you're looking to work in blockchain and the zk space, this will be a great way to meet the teams and get to know them as you apply. We host this on Gatherer.town, which is an awesome video game style, interactive online space. It's a lot of fun! This will be our third edition. If you're interested and you want to come check it out, uh, I've added the sign up link in the show notes. In the meantime, you can also check out the ZK Jobs Board, which is a list of open positions in the zk community. Anna (00:01:41): As the last related bit of housekeeping, the ZKValidator is actually hiring right now. ZKValidator is the other project that I spend my time on these days. We are currently looking to fill a Dev Ops position. So if you have the skillset and want to start working with us, exploring new PoS and L2 blockchain networks, do get in touch! So I've added the link to the Job Fair, to the Job Board and to this particular job at the ZKValidator in the show notes. Anna (00:02:09): I also want to thank this week sponsor, EY blockchain. EY is one of the founding members of the Baseline Protocol group, which is an industry-wide open source initiative to build privacy tools and scalability technology to make it easy and secure for enterprises to do business with each other on the public Ethereum blockchain. Find out more about it at blockchain.ey.com or visit the Baseline Protocol site at baseline-protocol.org. So thank you again, EY blockchai! Now here is my interview with Kristy-Leigh Minehan. Anna (00:02:47): This week, I'm here with Kristy-Leigh Minehan known on Twitter as OhGodAGirl, a long-standing participant in many crypto ecosystems. I want to say welcome, Kristy! Kristy (00:02:56): Thank you so much for having me, Anna. Anna (00:02:59): And I think in today's episode, the idea is to explore your background. We're going to talk a lot about mining. We're going to talk about ProgPow, maybe how that started and where it's at now. We're also going to learn about what you're working on today. So I'm very excited to jump into all of these topics with you. Before we start it, though, this is the first time we're meeting. So I really want to hear, where did you get your start in the space and what led you to working as a miner? Kristy (00:03:26): Oh, geez. That is quite a story. You know, I say that I officially got started in 2010. So we all have that moment where we either brought Bitcoin or we heard about Bitcoin and decided to mine a little bit. We all have that exact moment where we got captivated. And for me, I had a few chat groups I hang around in and a few friends had told me about Bitcoin and then dropped the whitepaper, and I was too young and too uneducated at the time to understand what any of that meant. But all my friends were getting super excited. So I naturally got super excited as well. And it wasn't until 2010 when the first, initial release had been dropped that I actually started tinkering and my friend group at the time made a huge game out of it. We were all competing for the highest hash rate. Hash rate to me was almost like a bit of a video game high score. And in the early days, when there wasn't a lot to do, you started optimizing and hand-tuning the memory on your laptop or the CPU, and overclocking it to get a higher score. It was incredibly addictive. And there's something, I don't know quite how to express it, but there's something... It's a bit like bragging rights, when you get to take a screenshot of your current performance, your current hash rate, throw it into a chat group and everyone's like, "Oh my God, how did you do that?" It strokes a little bit of your ego, but it's also incredibly addictive in the same way that you'll take screencaps of a fantastic pentakill on League or your high score in some other video game and just share it on the internet. It's bragging rights. And that was my addiction point. That was how I got hooked in crypto. It wasn't the tech, it wasn't the economics. I was way too young for any of that. It was all around the high score of hash rate. Anna (00:05:13): Were your friends at the time gamers? Was it like a group board gamer types? Kristy (00:05:18): Yes! Anna (00:05:18): Okay. Sounds a lot like that. Kristy (00:05:21): Not even that. The friend group, we were all security enthusiasts, it's what I would call us. There's a lot of overlap between the cyberpunk communities and the security communities and the video game communities. And it's something that you naturally do online to bond with people. There's not a lot of ways where you can build relationships with online communities, really close relationships, and spend time together outside of chatting. You can sometimes work on projects together, but games are just a natural thing that a lot of us default into. So it was a diverse group. And we had started competing until one day, my best friend at the time drops a screenshot, and he's got double the performance of what any of us have been able to achieve. And we're sitting here, the rest of us are sitting here, trying to think, "Okay, how is this possible? This isn't at all possible, we've all maxed out our machines. What what's going on?" Turns out he's rewritten part of the client for performance. And he won't tell us what he's done. He won't tell us how he's achieved it. He won't share. And he has the hide to say, "I'll give it to you for $200". And at that moment I got so outraged, so angry, just so furious that I was like, "You know what? I'm not going to pay for this. I'm going to teach myself how to do what he's done, just to go in and one up him!" And so that set me on a three-month rabbit hole of teaching myself everything there was to know about C and becoming incredibly obsessed with the C programming, everything there is to know about bitwise optimizations and hand optimizing for dedicated processors and how you target x86 architecture and just staying in a little box and working on this until finally, I had an optimized Bitcoin client that was 2% better than his, and that was all I needed. And that was, that whole experience was so addictive that it's what I've carried on with me for close to 11 years now as a career. And then finally, in the tail end of 2013, I actually got recruited to do the Golden Code, which is the optimized C code for an ASIC. And that kicked off my entire career in learning about floor planning and ASIC and hardware development. And that was incredibly addictive. So I had that nice little progression path of software - CPU - GPU - FPGA until finally I hit the hardware side of things. Anna (00:08:01): The ASICs. When you talk about this Golden Code, was this optimizing for existing ASICs? Or was there actual hardware changes being implemented through something like this? Kristy (00:08:13): Yeah. So your Golden Code is basically your algorithm optimized in Assembly. So it's got all of the inefficiencies stripped out of it until it is the most performant iteration of an algorithm. So in the case of Bitcoin, it's your SHA-256 algorithm with all of your rolls and unrolls baked into there and any inefficiencies removed. And then for a hash your Golden Code is separate, so on and so forth. That becomes the basis for the actual hardware design. And from that, you go into the architecture and floor planning stage where you now break that down into actually circuits, which is really, really cool. And so Golden Code it's the hand-optimized code you use to build a crypto ASIC. It is what sets you up for either success or failure. And in those early days, when everyone was neck and neck competing, and there were still a lot of secrets to be uncovered, Golden Code was incredibly valuable. So nowadays, however, everyone basically has the same Golden Code, SHA-256 has been studied. And now there's enough reports out there, there is enough optimizations out there that everyone has the same floor to compete with. And now all of the optimizations are actually just at silicone level. So now it's at your process level where you're now tweaking and tuning the five nanometer process and getting your simulations right. And then taping out to your multi-project wafer and optimizing, hand-optimizing and refining there. It's more a physical optimization versus a software optimization, like it was originally. Anna (00:09:54): Cool. What you're talking about here, was this still very freelance, kind of you're contributing, but in your own capacity? Or had you already joined a company professionally? Kristy (00:10:05): No. I actually did not join, I was entirely freelance and in fact I was entirely anonymous. So I had a male identity I used online. I mean, I didn't outright claim it was male, just people assumed that it was male. And so I just ran with it. And it wasn't up until 2015 that I actually joined a company full time, which was Genesis Mining. And I decided to just be myself, which was a shock to a lot of people who had thought, "Nope, this is definitely a dude, this is definitely a dude. There's no way!" And along with that came the moniker of OhGodAGirl, because I was like, "You know what? If everyone's so shocked about my gender, I might as well just own it and throw it out there. So it's not a shock to anyone else." Anna (00:10:54): Did you get a little bit of a kick out of that? Kristy (00:10:57): I did. Of course. Anna (00:11:01): I like being underestimated and then keep showing them. Or in your case actually I guess it was more like masked identity and then reveal. Very cool. Kristy (00:11:09): I think that US talks a lot about gender discrimination, but I feel it's not highlighted as much in other parts of the world. It definitely was prevalent in the early 2000s. And so I got a huge kick out of that because to me it didn't really matter what gender I was. How how does that have any sway? How does that have any influence on the skills I have as an individual? Anna (00:11:34): And in a way you proved that. Kristy (00:11:36): Yeah. Anna (00:11:38): You removed that from the equation and were able to compete properly, therefore you just proved that there was actually a need for this gendered view. Kristy (00:11:48): Yeah. Did not expect to dive into gender politics in this discussion, but... Anna (00:11:55): Well, actually let's bring it back to your story. So the first job that you had, what was it? It was Genesis Mining? Kristy (00:12:04): It was. I originally joined as a cryptographic engineer, which meant I was responsible for optimizing, doing a lot of things, actually, mostly hand optimizing a lot of their mining infrastructure. So a lot of their mining software, which at the time was predominantly Ethereum and Zcash and Monero. And then I think, three weeks into the job, my boss rang me and he was like, "Hey, you know hardware, right?" And I was like, "Yeah..?", He's like, "Great. We've got something we want you to look at in China." So I was on a plane to China to go and see [breaks up] to spend there. And we'd been trying to build an evolution to our current GPU rigs. So Genesis Mining, for those who aren't aware, is a cloud mining provider. And so cloud mining is essentially where you can purchase a hashrate contract, which gives you access to so much hash rate over a period of a year. And from that... You pay upfront. So you pay, say, $300 and now you get 300 GH worth of hash rate for the rest of that year. So it's almost like an extended cloud computing contract. It's a pretty gray area, because there's still a lot of laws being built up around this on whether or not it's a security or not. But nevertheless, it was an incredibly big business, I would say pre-2017. And it was a major way where people actually got exposed to mining. So not everyone had the capabilities nor access to the capital, nor even the knowledge of how to buy an ASIC or a GPU, but everyone could go to Genesis Mining and purchase a hashrate contract. And so, in order to be the most efficient in that business, you have to ensure that 1) you have the latest and greatest software that is hand optimized, that gives you an advantage over the competition, but also ensures that every unit of energy you're expending is giving you an equivalent hash rate, or an optimized hash rate output. So ensuring that you're better than your competition in that regard. The second thing, however, is optimizing for density. So you want to ensure that you have as many GPU cards as possible in one rig or as many ASICs as possible in one physical footprint. So optimizing for density is pretty important. And so GM had been experimenting with a new design and there were plenty of vendors in China that were also looking into this problem and that kick-started my 2 - 2.5 year stay in China. And it was an experience. I lived and worked in the middle of Fuzhou, Jianxi, and we built close to 850 MW worth of power infrastructure. And when I finally left China, we had close to 370 MW of that GPU infrastructure running. And that that's an experience, living in China during 2015 to 2017, that is quite an experience. Anna (00:15:23): So those numbers, I don't know what that means exactly. But would you be able to put that in node numbers or something that maybe would help me to better understand that size? Kristy (00:15:33): Yeah. Not quite node numbers. So megawatt, it all depends on what GPUs you're using, but 370 MW was close to, oh, this is going to be a scary number for people, that was close to about 8-9% of the Ethereum hash rate at the time, probably more. And then we progressively added to it at one point, we were close to 17-20% of the hash rate before the rest of the market kicked in. And it was a lot of GPUs, a lot of GPUs. But that was Genesis Mining's business. They didn't necessarily mine for themselves. They threw out this infrastructure and then people would purchase hashrate rental contracts. And then that hardware would be mining for the individual. Anna (00:16:26): This is sort of a random question, but can you run multiple nodes on a single GPU at the same time? Is that usually what is happening? Or is it per GPU, you tend to... I know, I'm probably focused too much on the node client thing, but that's always how I think about this. Maybe help me understand how that actually connects. Kristy (00:16:44): Of course. So imagine you have 8 GPUs in a rig and this is essentially just hash rate, it's a hashing machine and all it does is it gets work packets from a pool and then sends those work packets to the pool. And what we think of as nodes don't really exist in mining per se, we do have the concept of mining nodes, which is an individual miner. So think of that like that 8 GPU box. Anna (00:17:10): It's still an individual that could just be one node in a way, you'd have one node running at some point, but it's the hash rate of all eight. Kristy (00:17:18): Kind of. Because it's not really the concept of nodes. So what we think of in Ethereum with nodes, we think of a full client. So a node holds all state of all smart contracts. You don't actually have that in mining. It's not required to participate. All you need to have is mining software, which essentially gets work packets and then sends work packets back to the pool. And all it's doing is crunching through numbers, essentially, that's its whole job. And so all of the consolidation into blocks that actually happens at the pool level, which is where the nodes are run and maintained, at that level. And so in the case of something like cloud mining infrastructure, it's not quite that everyone has an individual node that they maintain, it's just that everyone has a certain amount of computing power that's allocated to them. And then they get the rewards for that computing power and the pools actually maintain the nodes. Now we do maintain nodes as farms, in order to make sure that our transactions and our blocks are actually picked up. It's more important if you are a large percent of the network, if you are just an individual yourself and running a machine at home in the US, chances are your transactions [are] probably going to be picked up quite close, but you only need about one node per 1200 rigs. Anna (00:18:46): Whoa. Holy shit! Yeah, you're right. You don't count in nodes. I get it. Kristy (00:18:51): Yeah. You only need one per facility. So a good rule of thumb is you have one per physical data center facility, and that just helps propagate your transactions and communicate with the network. That's about it. If you're running your own pool, that's an entirely different setup. And then you do need to maintain quite a few nodes in diverse regions. But for the average miner, no, all they're focusing on is the hardware itself. And this is because nodes don't provide any sort of incentive to running. Miners are very cost-sensitive. That means that we try and optimize for the lowest cost possible to squeeze out the most profit. And so in a lot of cases, running a node, it's a goodwill, it's just something that you do for the health and the benefit of the network, but a miner isn't necessarily incentivized to run out large collection of nodes. Now that being said, many miners in many pools do run lots of nodes for the benefit of the network, because they do deeply care about Ethereum or about Bitcoin. But definitely it feels like we should have put some incentive for running and maintaining a node in there. And that was the original design ethos of the first crypto miners: your wallet, your node, and your miner was all coupled into one. And we kind of killed all of that off, and I think about 2012, when we started having light clients and independent wallets and then independent miners. Anna (00:20:21): Got it. That project that you're talking about, where you went to China, was that an Ethereum-focused project, that specific mining thing, that factory that you just described was that mostly Ethereum? I know you mentioned it was Ethereum, Zcash and Monero generally, but what were you working on exactly? Or is it shared between all of these? Kristy (00:20:41): Itì's shared between all coins, it's whatever clients have purchased in terms of hash rate contracts and it's distributed over a wide variety of coins. Normally, cloud mining facilities would predominantly maintain a large percent in Ethereum, so in Ethash, and then they do a percent in ZEC and a percent in Monero, which were the hot coins to mine at the time, and then a small percent in alternative currencies. But it was whatever was most profitable at the time of where the demand was. In GM's case, yes, it was almost purely Ethereum and yeah, it was a very fun time to be a miner back then. In 2016 and 2017 we started having all of the memory shortages and the chip shortages, and that can be directly attributed to the huge crypto boom that was happening, especially in 2017. It was insane that memory was actually being purchased from the grey market, by some of the OEMs in China, just in order to have enough memory to fill all of their demand that they had. And miners just flocked to purchasing GPUs straight from OEMs in China versus what predominantly had been done, which was purchasing GPUs in say, on Amazon or in store or in a mom and pop store. So it was definitely a turning point into that kind of industrialization. And part of that was also that AMD was starting to get really interested in mining, and you can't blame them, they see these clients that have never showed up on their retail radar before suddenly purchasing 3000, 4000, 5000,1000 GPUs in one stroke. And they're like, "What is going on? Are you running a gaming cafe?" And you have to sit down and explain to them the process of crypto mining and "Nope, this is actually just we're mining this magical internet money, it's called Ethereum, here are all the cool things you can do with it." Well, at that time, you couldn't do a lot with Ethereum, but here all the cool things in the future you could do with it. It was a whole new market and hardware OEMs were completely mesmerized by this. And when we think mining, a lot of people still think Bitcoin. So they still think ASICs. And the fact that GPUs at the time could be utilized for crypto mining, it was mesmerizing. So it was a very exciting time. And there was just a lot of innovation that was also happening around this point, spurned by Ethereum mining innovation in hardware. So for instance, your actual server design, how could you get the most efficient server with all of the high cost components stripped out of it and something that would allow for optimal density. And so these designs that were pioneered in 2016 and 2017 actually are still utilized today and they're utilized in markets outside of crypto, which I find incredibly cool. Anna (00:23:48): What was the next phase? So you did Genesis Mining and then you also did Core Scientific. Was this a similar role or similar job? Or was this a big shift? Kristy (00:23:58): Core Scientific was bit of a shift for me personally. So Genesis Mining was run and maintained by a bunch of passionate cryptocurrency enthusiasts, but Core Scientific was the complete opposite. You had traditional corporate pedigree. And I met the Core Scientific team when they were called MineCo when I was actually looking to, I was running my own little startup at the time and I was looking for hosting space and I came across these guys in my search and they were in North Carolina. They called themselves something different and they were very passionate about the crypto space and Bitcoin mining in particular. But I could tell that they had just started to get into it and they didn't really know what they were doing quite, but I loved their enthusiasm. And I fell in love, honestly, with the characters and the individuals at the company. And I met Kevin in early 2018 and we had a discussion and it was right after I'd signed a hosting contract, actually, with Core Scientific with my startup at the time, that I was offered an opportunity to be a CTO and come grow Core Scientific. And so Core Scientific specialized in blockchain infrastructure. So specifically in data centers, in building, deploying and maintaining data centers, selling out hosting capacity for interested participants and really hyper-focusing on the crypto side of things. And then we also had an AI business, which was specialized in high-performance compute. And that's because there's a lot of overlap in the designs, not necessarily in the thermals, but definitely in the initial physical design of a data center, between HPC and crypto. Most importantly, they both favored density, we want to cram as much heat and as much output into a tiny form factor as possible. So we had these two sides of our business and the entire team at Core Scientific was traditional Fortune 50 executives. You had Kevin Turner, former COO of Microsoft and CEO of Citadel Securities. We had Bill Humes, former CFO of Ingram Micro, one of the largest electronics distributors and components distributors on the planet, among other things. We had Christy Barwick, one of the most fantastic treasurers I've ever had the pleasure of working with from Intellectual Ventures. You just had this absolute rockstar team that had come together and said, "You know what? We believe in this cryptocurrency thing, we want to come together and build the infrastructure to support it." And it was a big growth in both my personal career, getting to learn from that rockstar team, but also being able to teach them all about how it actually works in this space. How do you market to cryptocurrency miners? How do you talk to them? There was still that dynamic between centralized corporations and decentralized autonomous groups that clashed heavily in 2018 and 2019. It's only in 2021 that we've kind of become okay with the industrialization of crypto and the enterprises coming in in a big way. So it was very interesting, similar businesses, just different teams and different approaches to building out infrastructure. Anna (00:27:33): You also just mentioned a startup that sounds like it was in between those two roles. What was that? Was that just something you did for a shorter period of time? Kristy (00:27:41): Yeah, I had a small startup that actually started in Ukraine, because I was living in Ukraine at the time. Anna (00:27:47): From China to Ukraine? Or is there some other stops along the way? Kristy (00:27:53): There were quite a few stops. I had been actually recruited to do some CBDC work before CBDC became a buzzword in Ukraine. So I was camped out in Ukraine for a good few months. And during that time I had wanted to try my hand at actually doing the startup. So I formed the startup with two other good friends at the time, one who I met in Ukraine, one who's actually based here in Seattle and went on to try and build a cloud mining company that was run entirely through Ethereum smart contracts. So instead of a business that was selling hardware and selling these contracts, it was all entirely run on Ethereum, everything from the GPU, GPU's actually old hardware has something called ASIC IDs, which are these unique identifiers that are actually stamped into the silicone, but are also read by your machine and are used as a unique identifier. And so we tied these to smart contracts, so that your ASIC ID actually became almost your unique token. And it was uniquely yours as a user. That was your GPU. You could check it, you could modify it. It was yours. And the thing to understand about miners is a lot of us care deeply about what memory our cards have, because different memory gives different performance characteristics or different power characteristics. And we care about what our GPU card looks like and what fans it has and what are the manufacturers of the individual components on the PCB. And so we took all of this, we built tools to scrape all of this data and wrap it up into a smart contract so that miners could pick their perfect GPU card and get it at a cost plus a small amount of profit, which contrasts how things are actually sold in the crypto space. And it was fantastic. It was a great business early on. And I learned a ton as both the founder and the CEO. And it was really great to build a successful business on crypto. Anna (00:30:03): But what made you walk away from it? Kristy (00:30:04): For me as an individual, and it wasn't a decision I made lightly either, it took me about close to 2ish months to come to terms with this, getting to learn from industry giants about how to properly build a public company is what was needed to serve the next generation of infrastructure for blockchain. We'd gone through this evolution, wherein up until 2013, it was all scrappy individuals in their homes and then some small businesses. And then 2013 to 2016, that was when you got your semiprofessional businesses starting to industrialize, but it was all very hush-hush and secret. And this was incredibly true of the cloud mining space. We never talked about how much hash rate we actually owned or operated, we never talked about machines. It was kept very much under wraps. It was almost dirty in a way. Then in 2017 and 2018, you had the big ICO boom, which was full of scam artists and charlatans and people who claimed they were blockchain consultants and blockchain experts. And once the crypto winter hit, all of that got rustled out. Anna (00:31:21): Which is nice. Kristy (00:31:22): It is, it was very nice. And all that was left was now your professionals and the companies that wanted to be completely transparent, wanted to be out in the open, wanted to, go out there in the space and just do good. And so it was a hard choice for me, but it was the right choice to move forward with that. And it taught me a lot of things that I needed to know, in order to be successful in my own career. But most importantly, it was the opportunity to get to be part of a big company that for the first time in history was going to go and take a company public and actually build a public company around blockchain infrastructure. And basically be like, "Look, we're here to be a partner to clients with no politics involved, with no bias. We're just here to do what is right for cryptocurrency, what is right for Bitcoin, what is right for Ethereum." And that, to me, fit perfectly in with my moral compass and I was like, "I'm on this train". Anna (00:32:28): Cool. One of the things that I really wanted to ask you about was, around that time... So I feel like we've done a great history of where you started to 2018, 2019, maybe. So I remember on Twitter and to be honest, I didn't follow this very closely, but there was a lot of conversation around ProgPow and later a lot of conversation around EIP-1559. I gotta be honest, I never fully got inside those. And I am curious to hear, maybe you can tell me a little bit about what's the origin story for ProgPow. Let's start there. And what were you doing at that time? And I know it's probably a very big topic, because that's actually where I remember first seeing you on Twitter. Kristy (00:33:19): So ProgPow, it's important to understand the context at the time. When the first Ethereum ASICs were leaked, there were a lot of discussions on the Ethereum forums, as well as on the Ethereum Reddit about do we change the proof of work algorithm? In the whitepaper one of the defining characteristics at the time for Ethereum was that this was a proof of work chain that was optimized around GPUs. And it stated that ASICs were a bit of a scourge. They were a plague on a network. And it went into the design reasons and the design rationale for Ethash. And so Ethash, as an algorithm, it's the hashing algorithm for Ethereum, was designed to target GPUs and it was designed to be efficient on GPUs. And so there were a lot of proposals coming up about an alternative proof of work algorithm that would either break the ASICs or make ASICs impossible. And so myself and two other friends, we'll call them Def and Else, so we became IfDefElse, were sitting around, reading these proposals, and we were realizing that a lot of people had no idea how hardware actually works. And a lot of people don't think about how the hashing algorithm actually translates down to hardware, because the easiest way to explain how you make an ASIC, it's actually that you start with GPU and you strip off all of the components that are unnecessary until you're left with the base silicone that you need. And then you go make an ASIC out of that. It's a very "explain like I'm five" explanation, but it helps to visualize. And so in the case of Ethereum, there was 40% of the GPU card that was left idle actually in the Ethash algorithm, and everyone was going about it the wrong way. And so we decided, "Well, let's actually just go and contribute something. Let's go put up a proposal of how you should do this right and see if they're interested in it". And at the time I was actually between my startup and an offer for Core Scientific. So it was kind of in the middle. Anna (00:35:30): Okay, got it. Kristy (00:35:31): And we decided to do that. We actually approached Vlad Zamfir first for a review. And he told us, because we had no idea what we were doing, it's one thing to go and to do something in the Ethereum community, like be a miner and talk on forums. It's another to actually go and make an EIP. And the rules around that process, aren't quite clear, or they weren't clear back then. And so we honestly thought the way that you make proposals is you go pitch them to Vlad and to Vitalik, because in our minds there's your CEO and CTO, even though that is not at all how Ethereum works and Vlad was incredibly patient with us and actually guided us through the appropriate path, he was like, "You've got to go create an EIP, here's how you go do that. You've got to make sure it meets all these requirements." Everyone was incredibly supportive about and helped guide us around making sure that our ideas were formulated in the right framework to be classified as an EIP and help get us to the point where it was actually up for discussion. And so we did that. It was EIP-1057, and it was essentially a very small tweak to Ethash that optimized the operations to completely saturate the GPU. And so when we talk about saturation, that means it's trying to utilize as much of the silicon of a GPU as possible, in order to prevent someone from coming along and actually making an ASIC. And while you can strip components of a GPU away, you're also competing with the cost basis of a GPU and you're trying to say that you can get those components far cheaper than a company like AMD or Nvidia can, who have global supply chains. So you're doing one part protocol design, one part hardware design, but you're also tapping into the economics of the supply chain around GPUs and using all of that to your advantage, to create a highly optimized proof of work algorithm, still at hash base. The change is actually minor. It's 5 very small changes. And the entirety of ProgPow it's actually only 37 lines of code, maybe 42 now. So it's very, very small. It went largely ignored up until Devcon 4, when someone had reached out, I forget who now and said that, "Hey, Kristy, there's been a lot of discussion about ProgPow and a lot of the community is really excited and it would be great if you would come and present it at Devcon." And so I'd taken my role at Core Scientific and I was like, "Well, I have to go tell my boss about this." So I explained it all to my boss, my CEO K.T., and he was really supportive and he's like, "Go on, go do it." So I went to Prague and we did our presentation, which is now on YouTube and everyone watches it and likes to take screen caps and make memes out of it. And it was meant to just be a quick primer over the changes that we'd done to Ethash, why we had designed them, as well as the behaviors and incentives that certain hardware types create. So it's really important to understand that when you're designing a consensus algorithm, you need to be thinking about how all of the users in your system are going to engage with this consensus algorithm and about the kind of behaviors and incentives that are created. And the one thing we don't think of, as protocol designers, is we don't think about hardware, mostly because that is a big gap in most designers' understandings and knowledge. How does hardware work? How does the supply chains work? How does that entire space work? And so ASICs have an unfortunate history up until probably 2020 of causing centralized forces. It's important to understand the context again. So I'll reiterate that: it's about the time period. So up until 2020, most large ASIC purchases were done under the table. You went directly to the company and you were able to purchase the entire supply for yourself, and you would leave null to the smaller operators or the smaller miners. It was a first come-first serve kind of thing. And it was where money talks. So you could pay a premium to completely lock out your competition. Manufacturers didn't really have restrictions or regulations like they do in the traditional space. So as a vendor, I cannot come in and just buy out all of Intel supply of CPUs, no matter how much I pay, cause they actually have contracts and obligations to other vendors, retail stores, and other clients in other parts of the world. So I can't just use money and throw my money around and centralize that situation. Whereas in the ASIC space, you easily can, in crypto, you can. So that was a big issue. And then anytime that ASICs had felt threatened, historically, operators of ASICs would 51% attack a chain. And so we have a huge history of altcoins suffering 51% attacks the moment that there was even a discussion of the proof of work algorithm changing. So operators of ASICs at that time and what the evidence has shown us were incentivized to 1) protect their investments, 2) incentivize to stay as small and as silent as possible. And 3) were incentivized to do whatever would make them the most profit, which was keep Ethash running at the expense of the network. And there was a lot of interest and excitement and it wasn't until, I think, the community started rallying for it to actually be adopted, when the next generation of ASICs had been announced and released, which was E3 started to be distributed at large, that the community started to lobby for ProgPow. And so then we went through that whole process of getting it going through the different stages, getting it accepted and championing it. Anna (00:41:38): This is some great background into why it existed and like who the groups were that championed it, but who was against it? Who was the group? How would you define the group that had issue with it? Kristy (00:41:50): So amusingly at the start, for about a year, no one was against it. And part of that was, because a lot of people did not have an understanding of why Ethash, the hashing algorithm, was designed in the way it was. So it was hard for people to form unbiased decisions around ProgPow and be able to have a logical debate about it. The second reason was it didn't really affect dApp developers or anyone, except miners. So the mining community was the only one who cared about it. And the GPU miners cared quite a bit about having this and getting ASICs off the network. And so did some of the Ethereum core team, per the whitepaper's reasoning, and per the initial ask for community contributions to get a proof of work change. What happened was an ASIC company at the time called Linzhi Mining started, what I will say is one of the best lobbing campaigns I've ever experienced, to try and get this EIP shut down. And they were quite voracious in their efforts and that caused a lot of chaos. And it did a few things. It taught the Ethereum community about how we need better tools, in order to actually get real sentiment, because the tools we had at the time could easily be sybil-attacked. Your only tools were Reddit posts, which it's really easy to spin-off a Reddit account or a few thousand news articles. It's really easy to buy a news article. Twitter, it's really easy to make fake accounts on Twitter. Those aren't good tools for sentiment collection. And then you think about, "Well, hang on. What if we have voting through blocks through the extra data field?" And so we actually did a vote for ProgPow, Yes/No, with blocks and an overwhelming majority of the community voted "Yes", but then you've got to think about the dApp users who don't actually mine, how do they vote and how do they contribute? So now we've left a part of our ecosystem and our community completely void. What about the people that are still holders of Ethereum, but aren't technical enough to either engage in dApps yet, because the UX hasn't been built up, or people that are still participants of the network and contributing in their own special way, but don't hold a lot of Ether, how do you get their sentiment, their views? And so a few things happened at the time. There was the formation of the Ethereum Cat Herders, which helped actually get a lot of this sentiment into some digestible format. And there was people that built out other kinds of governance and sentiment tools to help collect all this sentiment. And ProgPow ended up becoming a topic on every single bloody ACD call that I think it just burnt everyone out. Not enough people cared, except for the mining community, and the mining community felt they weren't heard. They kept voicing. They kept trying to almost lobby for this, but a lot of miners aren't able to articulate why ASICs are bad beyond "ASICs are bad." And that's okay, not everyone has the best communication skills. And you had, at the end of it, what ended up happening is all of these groups were pointing fingers at each other saying, "you're only saying this to protect your profits". Miners were saying, "Well, you're only saying this because you don't care about us. We're also participants in the network." It was politics, awful, awful politics. Anna (00:45:29): I have followed a few of the very, very open governance, I don't want to say processes, cause they're like events, let's say, governance events or attempts to sway governance in one direction or another. And it's very interesting, because there does seem to be this attitude of like, in order to win, you have to convince the community. But if you try to convince the community, you're like a scammer. We don't like to be convinced. And so some people will, say, present very rational arguments, and then people can make a decision. You do that. And then maybe the Redditors, they're not paying that much attention to it and they can still manage to sway or scare some of the actual decision makers. And I think we've seen that over the years. I believe years ago, I did an episode actually looking at governance from various different perspectives in Ethereum at the time to try to understand how could this be better, what's needed. And I think a lot of times there was this drive for a signaling tool. I actually worked on a signaling tool while ago, a little bit, for a short amount of time, but that also got squashed. The signaling tool itself became politicized in a way. So that was a very... I mean, I was really just helping on the side. So I wasn't in the thick of it, but what I could even see was like, "Wow! So everyone is saying you need signaling!" And then when somebody tries to build signaling, there's this feeling that the signaling is going to be corrupt somehow, even before it exists. And it doesn't matter how open you are to feedback. And that's where you wonder, what are the forces at play really? Who benefits from maybe not having any signaling, no change, I don't know. Kristy (00:47:17): Exactly. You hit the nail on the head there that. As champions of an EIP, the onus is on us to educate and to convince people why they need this. But with ProgPow in particular, we were just contributing a solution to a problem that there obviously was. And asks for contributions had already been put out in the space. And this was our contribution. To be honest, we just got frustrated that a lot of people who didn't know what they were doing were proposing solutions. And I think that is the same for any developer, at some point you just get frustrated that someone is talking shit and you want it to be like, "You know what? Here's how you should properly do it." But the onus is on the champions to actually go and, well, champion their proposal, go out and communicate. But not everyone wants to do that, because it is interpreted incorrectly. And so because ProgPow was associated with mining, it gave an easy opportunity for the opponents, in this case the ASIC manufacturers, to hyper fixate on my background, who I was, what I was currently working on. "Oh, is Core Scientific somehow incentivize to push or lobby this?" You ask anyone at Core Scientific, what ProgPow is. Their eyes will just glaze over. There'll be like, "That's that thing Kristy was doing in her spare time, right?" There's just a lot of skepticism and naturally we need healthy skepticism in our community, but at some point it turns to toxic skepticism. And yet it culminated in just so much drama that ultimately I was unmotivated to continue. The core devs were unmotivated to keep having this damn discussion every week. Everyone was unmotivated. And at that point you just want to walk away, but ProgPow, now it's a meme. It can not be killed, because to this day that keeps being brought up every now and then, and it still makes its way into cause. I stepped down as champion, I think in, officially in 2020, right before I got COVID actually. So I was hands-off, said, "No, not doing this anymore." Greg Colvin actually stepped up to champion it. And to this day it's still there in accepted status. And every now and then it still gets brought up. As either a meme about governance, or as a meme about controversial EIP, or, heaven forbid, when we're talking about the Merge or about phasing out proof of work, ProgPow is still there as a bartering tool about what we can give this to miners, in order to make make the Merge easier. So it still has not been completely killed off yet. And it's being adopted by a few altcoins. So Ravencoin has their modified version, which they call KawPoW, because their mascot is a Raven, so "kaw-kaw". And a Zcoin, which recently rebranded to Firo, has adopted it as well. There is a small privacy coin called Veil, which has adopted it. So a lot of altcoins are adopting it. And one of the design parameters around ProgPow is the ability to have programmable almost-tiles, which allow you to create your unique flavor of a proof of work algorithm, optimized with specific GPU type. So all of these coins can actually benefit from the same security an ASIC would provide by tuning to a different GPU card, which is kind of cool. Anna (00:51:01): So just to recap though, what the difference was, cause that I never really got. So this is the first time I actually heard. I knew it had something to do with ASICs, something to do with GPUs. But what you were saying is ASICs come along and optimize in a very narrow way to do something. But in the case of Ethereum, most miners are using GPUs, but they are not fully used. And the option here was ASICs are going to come along or potentially you could increase the usefulness of the GPU that has already been used. And your argument was, since miners already have these GPUs, it would keep that mining community in a way healthy. Whereas if you only do ASICs, then those are hard to get and could be almost front run in the real world, with big entities purchasing a lot of them. Kristy (00:51:48): Correct. And also a majority of the infrastructure for mining at the time, it was just GPUs. And the whitepaper specifically called out ASICs as a plague. And the GPUs themselves had been purchased knowing that this was going to be a GPU-friendly chain. So miners felt that there was a social contract that they had entered into and they didn't want that to be broken. Now the whitepaper has been modified now, it's on ethereum.org, to remove a lot of those nuances, but the original whitepaper at the time was a source of almost truth as well as controversy. Anna (00:52:26): And you mean the whitepaper for Ethereum client development? Kristy (00:52:28): Correct. Anna (00:52:28): Got it. Kristy (00:52:30): And specifically even, I think there was a part in the yellowpaper that talked about Ethash and the design rationale of the algorithm. So there was almost a social contract between between miners and the community. Miners were going to provide this hash rate, it was going to be on GPUs. Even on the website at the time it specifically called out "Do not mine with ASICs, ASICs can't exist on Ethash. If you see an ASIC, it's probably a scam", which was hilarious. Anna (00:52:59): I remember hearing, and I don't know if this is later, maybe it's a different thing, but I thought that there was sort of talk of creating some way that the proof of work would always change slightly, so that ASICs wouldn't work well. Is that a separate proposal that came later? Or what was that? Kristy (00:53:16): That was inside of ProgPow, because it has a random math element. So one of the things of ProgPow, it uses KISS99 for its randomization algorithm. And every X amount of blocks, and you can define what it is, so in the case of Ethereum, it was every minute the algorithm changes, so that an ASIC that gets created is forced to have some form of programmability. Anna (00:53:42): And that makes it a little bit less like an ASIC, right? It moves it further on towards the GPU. Kristy (00:53:47): The best way to think about it is the GPU is the ASIC. ASIC just means it's the optimal hardware for that task. And so GPUs are ASICs in and of themselves. When you buy the little core individually, it's called a GPU ASIC. And all we've done is said, "Okay, well, if GPUs are the ASIC, we're going to go and design an algorithm that targets that as the ASIC." Anna (00:54:13): So what happened? You mentioned that there were a batch of ASICs that were released right around that time. So, given the state of Ethereum as it was, without ProgPow being included, did ASICs just arrive on the market and then have they been taking over share of the network? Where are we at? Kristy (00:54:34): Yeah. So what happened was the E3s, for a time, were close to 30% of the network and they were silently mining, so no one knew where they were, who they belonged to. What did happen was COVID and the supply chain and the hit to our supply chain. So it was quite an opportune moment. You had at the time three participants in the ecosystem who were in the business of building Ethereum ASICs. You had Bitmain who had just built their E3 and was looking to optimize that. And specifically the E3 was built because they had an oversupply of memory. Because remember I mentioned that at the time I was in China, we were going around and purchasing memory? Bitmain was doing the same thing, they were one of our competitors. So they were also hoarding and stocking up on all of this GDDR5 memory and DDR4 memory and DDR3 memory. And they had lots of excess surplus. And so they used a lot of their old DDR3 RAM to actually build out these ASICs, because if you've got spare components, what are you going to do? Just go sell them? No, turn it into a machine. So that was why the E3 started to exist. And then research had been continued on other Ethereum ASICs, but then Bitmain was having some internal politics. Entering Linzhi. Linzhi CEO was the former CTO of Canaan, which is another large Bitcoin and altcoin ASIC manufacturer. She left to go and build an Ethereum ASIC and they were in the research stage. And what they proposed was to do this fantastic heavy duty, massive die, connected to a lot of small memory chips. And it would've outperformed any GPU on the market by a factor of 20 to 30x, but it had incredibly high power consumption, which they weren't transparent about and they never made it to market or mass distribution. So there was that issue as well. And then you had Innosilicon. So Innosilicon has the Ethmaster and a few other ASICs. And they're the only competitor still in the Ethereum ASIC game today, alongside Bitmain. And they were in the process of building and researching and innovating, because their main business is actually in producing memory PHY, which is utilized in GPU cards. So they already have a lot of the innovation in the IP and the knowledge necessary to compete. But all of this discussion about ProgPow had everyone skiddish, combined with discussions about when we're going to be transitioning to proof of stake. So the reason we never upgraded to ProgPow was ultimately because, well proof of stake's right around the corner. I kid you not, it was said in March, it's going to be deployed in July this year, July, 2020, it was hilarious. So it was always a case of "we don't need it because we're going to get the Merge soon". ASIC manufacturers don't want to go and invest and do any research or development, and then COVID hit. And all of a sudden the world stopped and everyone stopped doing what they were doing and just sat in their homes. And this is why there's such a constraint on supply even to this day. So there was a time and place thing, a very opportune time. And now maybe we'll get proof of stake and the Merge this year. And eventually those GPUs will be able to transition off into other altcoins and ASICs will just be stripped for parts. And the only part good in most of the Ethereum ASICs is the memory. Anna (00:58:15): What do you think the miners themselves are going to do? Do you think that they'll fork over to something? Do you have any sense for their behavior? You think they're going to sell? Kristy (00:58:28): No, miners might threaten to fork, but honestly a lot of miners do not have the capabilities to self-organize, fork a chain and then maintain that chain. So at the end of the day, value follows where the dApps and development are. And that will always be the Ethereum main chain. And so miners might fork initially, but it won't fizzle into anything. And for it to have value, it has to be listed on exchanges and maintained on exchanges. And there has to be development that happens. So you would need a lot of miners to self-organize into that. And the mining pools themselves have said "No" and will follow the longest chain. Anna (00:59:12): What about something like ETC though? That's a chain that has a very similar thing. The switch would be easy. Have people thought of going there? Kristy (00:59:21): The problem with ETC is partly in culture. They lobbied hard against ProgPow, so they've already left a sour taste in miners' mouths. The second part of that is they're pursuing a proof of work change called Keccak, which favors ASICs. So miners wouldn't flock to that chain, because 1) it has to be profitable. But 2) it's right back where they started, going into an ASIC-favored chain, where their days are numbered. As an investor, you don't really want to invest a ton of capital into something that you know only has a 2-3 months lifespan. That's the best way I can explain mining. You want to make a 2-3 year investment, not a few months investment. So it's really, really difficult for miners. So a few of them will flock to altcoins. There's Ravencoin, there's Firo, there's a few others, but a majority of the hardware in China, especially now, in the current condition, is just going to sell or liquidate. Or it will be repurposed into the HPC space, which again is going to put pressure on the OEMs, because now they can't continue to sell out their orders at their current price point. Anna (01:00:36): What's HPC? What does that stand for? Kristy (01:00:38): High Performance Computing. So anything to do with cloud computing, rendering, artificial intelligence. Anna (01:00:45): I had a question as you were telling the story of these mining communities, but what is the mining community like? And how are they feeling about this? Are they sad? Is there anger? Is there threats? What's going on? Kristy (01:01:01): They're diverse. So at the end of the day, the mining community is just as colorful and vibrant as the dApp community, which is, it's a bunch of really diverse opinionated people that have come together in love of mining. And there's natural skepticism over when the Merge will happen and if it ever will happen. And that is fair skepticism, because we've been told that it will happen since 2018, so that's pretty normal. But there's also... The ProgPow debate made it clear what position miners were in and EIP-1559 has reinforced that. So a lot of miners feel they aren't actually heard and that their voices aren't heard. And unfortunately that is a reality of a chain that wants to move to proof of stake. It is a whole community is being cut out, there's nothing that we can do about that. What we can do is encourage them to come join the staking community and run validators instead and transform. And a lot of the miners will actually do that. And so pools have started to build staking businesses and I've started to prepare for that transition, and the larger infrastructure and the larger, more professional firms have actually diversified into that. But your individual hobbyist or smaller miner definitely there's that bittersweet anger. And I think a lot of people, when the dust has settled and when the Merge has happened, a lot of those miners will still be users of the Ethereum ecosystem. Ethereum is more than just mining, it is development. It is ETH, it is stores of value, it's this vibrant melting pot of communities and ideas and applications now. And so the Ether that they've worked so hard to create has a purpose and has a use. And I think a lot of them will transition over into the NFT space and participating in that way, transitioning into the next step. Even a lot of miners are starting to get into DeFi through way of MEV, miner extractable value. And that is the onboarding into the decentralized finance space for miners. Anna (01:03:22): Wow, totally. I didn't think about that, but it very much is. It's like onboarding them into this more financial ways of playing in blockchain. Oh, that's so cool. Kristy (01:03:35): And to be honest, Flashbots has done something phenomenal in that they've managed to self-organize the mining community around a drive, a common drive and vision for profit, and they've shown them a different way of utilizing Ethereum, outside of just mining. And it's great. And yet Flashbots right now is doing a fantastic job for the Ethereum ecosystem and it will continue to exist when we go into the Merge as well. So MEV will still be a component of ETH 2.0, and I have no doubt that a lot of the people that have been onboarded into MEV through Flashbots will continue to maintain some sort of presence in ETH 2.0. Anna (01:04:20): Cool. I mean, this has been such a journey from your start through these mining and the communities and the companies and then these controversies. What I want to hear about now is what are you working on today? What are you interested in? What's your focus these days? You said you stepped away from being a ProgPow champion in early 2020. What have you been up to? Kristy (01:04:43): So I took a bit of a sabbatical, very short one, from the cryptocurrency ecosystem to go learn how mobile phones were manufactured and actually go and launch a mobile phone. So that was cool. I did that, check that on my bucket list. And then I'm back in the crypto space and doing a lot of advisory and consultancy on the expert witness side. So actually being an expert witness for legal cases and also working on something called Symbol, which is a blockchain protocol that's seeking to disrupt markets. So every cryptocurrency seeks to disrupt something. Bitcoin's disrupting payment infrastructure. Ethereum's disrupting cloud computing. Arweave disrupts cloud storage and Helium disrupts internet connectivity. Symbol's all about disrupting markets. And it's a learning experience for me, because Symbol is a proof of stake coin. They use a modified proof of stake called "Proof of Stake Plus", which aims to reward users of a chain preferentially to hoarders of stake. It's really cool. It's been a learning experience. I've been working on that a little bit in stealth mode and just mainly getting back into the ecosystem. Still definitely a big part of my advisory is still in the mining space, but I'm very curious about protocol design in and of itself. And proof of work just happens to be the largest. But I think that tide is going to tip in a few years, with Ethereum moving to proof of stake. Anna (01:06:25): Is there some zk in this Symbol project? The reason I ask is this is a bit how we've crossed paths. I know that you've started to be a little bit more in the zk space. So tell me a little bit about that. Kristy (01:06:38): So in 2019 and 2020, I was actually doing a lot of consultancy around optimizing the matrix multiplication of zero knowledge zk-STARKs on FPGAs. And I put all of that aside, while I went to run off and learn about mobiles. And then finally with the Symbol, one of the things we're actually exploring is zero knowledge proofs and zkRollups, in order to ensure that our chain can support all of these different markets. And so the last 3 weeks it's been nothing but research on the current state of zkRollups and zero knowledge proofs and zk-STARKs, and understanding the various implementations, be it Zcash or Ethereum and all the innovation that's happening there. Anna (01:07:25): Are you focused more on STARKs or SNARKs? Kristy (01:07:27): STARKs. Anna (01:07:27): Oh, interesting. Okay. Cool. Kristy (01:07:31): I am a STARK maximalist, you can blame Eli for that. So here's the thing, if you're going to use zero knowledge proofs, it feels like you should go through a trustless environment and not have any of that toxic waste. And while I understand that SNARKs are where both innovation as well as development is, at the end of the day, when you're looking forward, STARKs are the superior technology there, while just most of the development hasn't been focused on that path. From my perspective, it seems like if we're architecting this from the start, we should go the STARK path, because that is going to be the successor in 5-10 years out. But it's all still in the research and the architecture phase. And there could be a nice way to eventually transition, but yes, personally, I'm a bit of a STARK maximalist, because trustless environments for the win. Anna (01:08:33): Nice. I think the StarkWare crew might enjoy hearing that. So I have one last question or thought I want to explore with you, which is after all of this journey and now working more in the zk side of things, did you feel burned out or annoyed by all of this stuff? Cause it sounds rough, the fact that you yourself got held up and you were attacked. So how did you go through that, survive it? Maybe if you have any advice to people. Kristy (01:09:06): Oh, goodness. Yes. Anna (01:09:08): And the advice should not be, "Don't try!", right? That's not good advice. Kristy (01:09:12): Yeah. That is not good advice. Definitely, I got burnt out. The burnout was twofold. 1) It was that there was such pressure on revealing the identities of Def and Else that I wasn't comfortable with that, because, as I mentioned, having pseudonymity is an important part of my own ethics and morals and allowing people to have their private lives is incredibly important. And who cares who you are in crypto? I mean, we're all here because some dude called Satoshi Nakamoto decided to drum out his thoughts on a piece of paper. We are born of anonymous identities. So that was a big part of the burnout. And then the politics. And the best advice I can probably give to my younger self is just keep focusing on the technology, ignore what people say about you, ignore what people say about your character or the company you keep, just ignore all of it, because at the end of the day, as much as we don't like it, blockchain is a technocracy and we are all here for the tech. And that's what we need to focus on. If your opponents can't have a logical technical debate with you, then you've already won that argument. And you just need to keep that in your back pocket and keep telling yourself that EIPs are political. Not all of them are, but ones that affect anything to do with fees or profits or the monetary policy will always be political, just like it is in the real world. And it can be incredibly draining, but just keep focusing on the technology and keep articulating and standing your ground. And that's all that is needed. And finally have a good group of friends you can vent to. Anna (01:11:07): Kristy, I want to say thank you so much for coming on the show and sharing this journey with myself and the listeners. Very nice to meet you and get to know you a little better. Kristy (01:11:16): Absolute pleasure. Thank you so much for having me. Speaker 1 (01:11:19): Cool. And I want to say thank you to the podcast producer, Andrey, thank you to the podcast editor, Henrik. And to our listeners. Anna (01:11:27): Thanks for listening.