Ben: Hello and welcome to PodRocket. Today. I'm here with Nick Thompson, who is the creator of a really cool audio tool called Elementary Audio. And this is a special episode because it's our first ever video episode. We're going to be posting video from this episode on YouTube, but of course, audio will continue to live wherever you get your podcasts normally. And we may have some video only content here. So if you're listening to the podcast on audio, we will put a link to the YouTube for any parts of episode that are video focused so that you can check those out. But Hey, how you doing Nick? Nick: Good. Happy to be here. Ben: So really excited to learn about what you're building. I know very little about audio engineering or digital signal processing or any of these things. So I'm extra excited to learn from you. Nick: Great. You are right in my target audience. So that's great. Ben: Could you give us a quick overview of what Elementary is? Nick: Sure. So Elementary is a new JavaScript run time for writing native audio applications and it's based on no JS. So what you write to work with Elementary looks a lot like what you would write to work with JS. So you can use NPM. Install the whole MPN package ecosystem, et cetera. But when you invoke your application, you would it with the Elementary command line tool instead of the, no JS, command line tool. And so the primary difference is that, when you work inside Elementary, there is basically a native module. Specific to Elementary that is able to interact with your JavaScript code on the front end and a real-time native audio lender intention on the backend. So you work in Java script, but you can use this application to ship native audio software. And the heart of the project is something you can embed in native audio applications and custom native stacks, if you have that type of application. Ben: And when you say, writing audio software, are we talking about editing audio samples or are we talking about digital music production or both? What are some of the common applications you see? Nick: Sure. So I think that this can cover all sorts of different applications, all of the types that you just mentioned, I think that you can do with Elementary. I'm personally coming at it from the digital music realm. Before I started this project, I started a company to make and sell audio appointments for computer musicians. And that was what prompted this project actually was, just the process for building those products and getting it out in the market is difficult. Takes a long time. Easy to get wrong. And before I got into that field, I was a frontend web developer for several years. And I really missed the speed and the tooling and the workflow that you have in that environment. So anyways, kind of a tangent. Right? Yes. We can handle all kinds of audio, software, real time digital music production. I have somebody who already who's interested in working on a digital guitar pedal. Picture of a guitar pedal with your quarter ins and outs, but whether DSP is actually running by Elementary, which is going to be cool. So it can do a lot. Ben: And typically this kind of software would be written in what, like C++ or is that where a lot of the library support is? Nick: Yeah, definitely. Ben: And so what are the advantages of being able to write audio software in JavaScript? Nick: I guess, I look at it, maybe this is the pessimistic way of looking at it. But in large part, I see the advantages as avoiding the disadvantages of working in C++. and to be honest, I think that the standard approach to building audio software in C++, in so far as my experience has shown me, leaves developers to solve the same types of problems over and over again. In ways that I don't think we really have to. For example, we spend a lot of time building native audio processors that want to filter a signal in some way, or generate a signal in some way or read from a file in some way. And as soon as you kick that can of worms open in C++ land, you have to deal with multithreading, lock free interaction with the real-time rendering thread. Nick: You really want deterministic operations to happen, strictly deterministic on the real-time rendering threads. If you're very careful about memory allocations of deallocations and this problem compounds pretty quickly. And I have not so far seen a lot of C++ solutions that generically solve this once and for all. So for my own products and for companies that I've worked with, I found myself rewriting the same code to solve the same set of problems. These are tricky problems also. Takes all the time. So sorry, I'm rambling. But in a lot of ways, I think JavaScript has advantages here because we can work in a way that feels much more like React. So you can work in a way that has this dynamic, low barrier to entry language, with this huge tooling ecosystem where you can work with a declarative expression of what you ultimately want the sound to be, read the rest of the Elementary engine. Ben: Yeah. So maybe on that note, could we jump into a demo and take a look at what it actually looks like to get write some audio in Elementary? Nick: Yeah, of course. So I'm going to start with a basically, a hello world of working in audio. And so, as I mentioned, it's going to be a lot like working in no dance, right? So to start, we open up a doctorate file. So to do that, we use the library function here, that cycle, which will generate a sign sound here at 440 Hertz. Then I use the EL dot train, that library method, which will generate a pulse signal, which is a signal that goes from zero to one and back to zero, and then repeats at whatever frequency you specify. So here I'm going to run that signal at two Hertz that signal will trigger an envelope generator. And if you don't know, if you move to audio, envelope generator will be a foreign word to you. Maybe many of what I've said already are foreign words to you. Nick: But this generator is just a way to smoothly get from zero amplitude, which is silent up to an altitude of one, which is as loud as the incoming signal is going to be by the time we're at one, because it's at the end of the day, we're going to multiply this envelope by the sine tone. And so at a gain value of one, it's just the identity function. So the envelope will go smoothly from zero to one and then back down to zero over time. And this will be triggered every time our gates signal goes from zero to one. So all you need to know really is that, we're approaching a, it's a pinging tone sound, very simply. So we've put that together. And then here is the primary render step, and this should feel a lot like React. So I'm describing what I want my signal to be. And then I'm asking Elementary to run it on signal. So just starting with that, if I then invoke this command line, just like you would with the no JS scripts, which up here are. Make sense so far? Ben: Yep. Nick: Let's go, let's take this one step further, but definitely stop me if I'm failing to explain any of these things adequately. What I want to show here is a series of examples that arrives at a way of synthesizing like a kick drum sound. Think like a techno song with that, like classic. Right? So synthesizing that is actually a pretty simple exercise. And I think it's fitting for a short podcast. So we're going to go step-by-step on that route. So to get there, we have this pinging sine tone, right? And now a lot of like a very simple way of synthesizing this type of kick drum sound is to get that tone every time it starts, to sweep down from a higher pitch. So what I'm doing here is we have the same gate that we started with last time. So this is going to trigger the same rate fire one. Nick: We have our same amplitude contour. So this is what gives it the, each instance of the kick drum. And then we have another envelope exactly the same like signal function. But this time we're going to use it to modulate the pitch for the sine tone is running out of. So then we're going to generate our tone and to do this, we take our same sine tone, but instead of just running at 440 Hertz like we were last time, we're going to add to that 440 Hertz. The result of multiplying our pitch envelope basically, so that it will go from five times, 440 Hertz, down to 440 Hertz itself, very quickly. And then we take the result there, we multiply the, that tone, via the volume envelope. And we asked Elementary to render it. So running this example, we're going to expect to hear a pinging sound tone. But each time it pings, it's going to have this chir at the beginning. Does that make sense? Ben: Yeah. So you can hear, I was going to say, if you can hear a bit more of the attack there, it sounds like. Nick: Yeah, exactly. And so we can like, for example, opening that same file back up, we have our pitch envelope, right? And the way this function works, the ADSR, this is a common abstraction for four envelope generators in DSP. So this is, the term ADSR is something you could Google can read all about. But in general, we have, this first argument would be the time it takes to get from zero to one, the attack phase of the sound. Then the next value is the decay. This will be the time that it takes to get back down from one to some sustained value. That's the asset ESR. And then we're going to hold it that sustained value until the gate deletes is basically, you can think of this as lifting your finger off of a piano key. And then the envelope will ring out slowly back to zero. Nick: So we're using this to modulate the pitch of the sine tone, which means that we can just come in here like tweaks and values. For example. We have a quick attack on the pitch envelope and if we change the decay time now, I'm going from 10 seconds here to a hundred milliseconds. It might not be it terribly drastic change, but let's see what that sounds like. So now the chirp is, takes a little bit longer time. Okay. So example number three, this is really the same thing that I just showed you. So we're going to take that whole process and just move it down to a lower pitch. So all of these, the idea of all of the envelope generators, we just worked through, that's all going to stay the same. The tone that we started with was prior, previously running at 440 Hertz with this sweep down from, five times, 440 Hertz. I'm going to pull that down to 34 Hertz because here we're getting into this sub base range. Nick: And so as we sweep down, we're going to get the chirp or the click of our kick drum. And then it should rain through with a brief sub sound. And then the last thing that I've changed here is, I'm taking that sound and I'm just saturating it slightly. This is like a subtle distortion. You can think like a guitar pedal type distortion. Just to give it a little bit more character. So with just those few changes, we should hear something more like a consistent techno kick drum. So that is the basic example of a really quick walkthrough from hello world to synthesizing sound that should hopefully, be somewhat familiar. And I have two more examples, if you want to go into it. We can skip them, if you don't want to. Leaping from hello world, and kick drums to a more complete drum kit synthesis. What do you think, go there or skip it? Ben: Yeah. Let's check it out. Nick: So I'm going to skip ahead, away from kick drum synthesis and I'm going to show you really briefly here, what it looks like to interact with midi. So I have a mini keyboard next to midi, and as I'm pressing notes on the keyboard, I want to trigger a drum samples. And these drum samples are all going to be synthesized in the same style that we just discussed. So I have a kick drum, a clap sound and a high hop. And I'm going to show you down here. This is where Elementary, I think gets really interesting. So after the load event, we install a media event hacker. And when we receive a media event, I have this function, your update voice state. And in a lot of ways you can think about this kind of being like, are you guys familiar with in React and Redux, you have this notion of like dispatching actions and then the stores have their reducers to update state. And then the components listen to the stores, right? So this is really similar. Nick: So when we receive a video events, I want to update some notion of the application state. And I don't want to use that state to render a new view of my application in a sense. Except here, the view is the description of the DSP. And once I have my new description with DSP, I can just pass that to the render function, just like you would do with React. And Elementary, under the hood, will just like React, the current state and the future state and make sure that we only make the necessary operations to get the underlying run time, to make those sounds. So really quickly update voice state. When we get a midi note. We're just going to look at this voices array. Nick: To see if the note we're receiving matches the midi note number in our math here. So I only want to interact with mini note number 60, 61 and 62. When I, when we get that many event, we'll find, for example, if I pushed mini note 60, that I want to go right into that voice as a right, a new gate value. This is going to be one and zero, just like the trigger for our previous example. And if you're letting go of the key, I'm going to go find that same note and I set the gate to zero. So this is your on and off. And once the state has finished updating, this is the, this right here, it's like the whole rendering pipeline, right? So we just look through each of the voices in our math and we invoke their function with their new state, whether or not they're going to be on or off. Pass that to core dot render. And if we run this, so we're not going to hear anything until I start pushing keys. Right? That's a big kick drum, you hearing that? Ben: Yep. Nick: So right there in, I don't know what that was 50 lines of JavaScript or something. We have an interactive example with totally synthesized elements, all in JavaScript. And each of those elements are synthesized just the same way that we walked through with the kick drum. And actually this whole library is available as an NPM package. If we want to NPM install and tweak your own application. So with that, I just wanted to show basically, mini handling and the render step. How the render step can take a new description of the tree and get from A to B. And that's it. And that's the example. Ben: Yeah.I mean, this is awesome and especially, cool to see how much you can do with so few lines of code. So I'm curious, like we talked about replicating the sound of a kick drum. If you were to think about replicating a different sound, I don't know, a violin, string or a trombone, or like any instrument, what would be your process to take a sound sample and figuring out how to replicate it in code? Nick: I mean, there are whole fields of research that go into answering that question. And I very particularly chose a kick drum because there is a well-known, it's accepted approach for synthesizing a kick drum. And that's what I showed you. And actually I even simplified it further. So if you were to go get an analog drum synthesize or at a guitar center or something like that, it probably doing something more complicated than what I just showed you to try to simulate a sound of the kick drum. And there are like totally whole fields of research going into like how we can try to recreate the sound of Ireland string accurately using DSP. And I'm definitely the wrong person to ask on that. Nick: So my process honestly, is if I'm interested in synthesizing something, I just start with Googling it. I go out and read about the existing research. And often the research is pretty intense, really interesting to read, but I don't know that some of it's even over my head, right. And so sometimes I like find research that I'm interested in and use that to, as inspiration for guiding my own synthesis approach and see if I can get something that I'm happy with, whether or not that is close to what I was going with, going for. Ben: Got it. So not an easy answer there. But it's good to see, I imagine the previous articles or codes to apples that have been written for kind the C++ tools. You could probably translate those fairly easily into what you're doing at JavaScript, right? Nick: Yeah, totally. And actually I'm finding it even more, I'm finding it more enjoyable actually to write it in JavaScript. So for example, one of the library features that you'll find in Elementary is an anti-aliasing sawtooth oscillator. And basically, that's just an oscillator that has this really sharp aggressive sound, but well, intentionally try to avoid aliasing in the digital audio spectrum, which is a whole conversation that we don't need to get into. But as I was, as I've written stuff in C++, I've written this approach in C++. and this particular approach that's in the Elementary library, it's called polyglot and I've written in C++, it's not so bad at C++, its pretty easy. But I rewrote it in JavaScript using Elementary to add it to the library. And it just felt far more intuitive to me to think, in that, because not only am I working with JavaScript, but the declarative way of expressing these things, to me feels much more intuitive. So yeah, it's been an interesting process. Ben: Well, one thing I was curious about, when I saw you writing the code and you were working at them. Any thoughts on whether in the future there might be an IDE for writing code in Elementary? It strikes me as the kind of thing where having an IDE that gives you ways to quickly listen to just certain effects or to drag a slider and hear how it changes the sound. Some of those things could be really cool and an IDE and make a program are much more productive. Is that something you've ever thought about? Nick: Yeah. Absolutely. Some of the work that I'm hoping to get into this fall is basically building, I don't know if an IDE is the right word for it. But either building out like a desktop application environment for working in this kind of way. It'll either be a desktop app or a plugin that loads inside of your doc, the environment that you might use to write music. That I think is a really interesting and exciting avenue. And actually for my own, as I'm hacking on this myself, I actually have a plugin already that I use locally that will spin up inside of my doc and read Elementary that I've written on my computer and process in real time inside the doc. And I've had a little sliders for real time manipulation and stuff, and that is such a cool workflow. So yes, absolutely mean, I think you're spot on and I definitely want to be able to offer something like that in the near future. Ben: And maybe this is a good segue into understanding, what is the current state of Elementary? What broadly things can you do? And then what can't you do that in the future you'd like to be able to, you would like to build? Nick: Yeah. That's a good question. So there are a few features that are missing from Elementary right now, that I think really need to be there for this to be a viable DSP platform. The first major one is feedback loops. So a lot of DSP involves feedback loops and a lot of creative uses of DSP involve feedback loops. And you can find that in, modular synthesis videos on YouTube stuff. Just creative ways to apply feedback as a whole, awesome field exploration. And right now Elementary does not have a generic notion of feedback. There is a delay element that you can use that can feed back into itself, which is helpful for, like a quick implementation of an echo, for example. But feedback in general, in the general case is something that I will plan to get to soon. I mean, designed, it's solved, but I haven't implemented it. That's number one. And then number two, I'm really looking forward to hosting VST plug-ins inside of Elementary itself. I think that'll be a really neat situation when it gets there. Ben: And from an architecture perspective, is Elementary built on top of, or explained to me like, how is the Elementary architected? What is under the hood in terms of the native code that powers the DSP? Nick: Yeah. So, I mean, at the end of the day, much of what Elementary is, is node.js. And so you can think of it as node.js with this custom native node add on. Right? And the add on is all C++, that I've written up from scratch that will open up the audio driver on your machine and start working with render and real time audio buffers. And then there's this layer that happens inside the native piece for, the render step has happened in JavaScript. We have this new desired state and you have to get to, and there's this middleman layer for getting there and making the real-time rendering threat, aware of these changes. In such a way that you don't have to worry about memory management at the end of the day. You don't have to worry about smooth transitions for the way that the audio is changing on the real time thread, stuff like that. So that's the architecture in 30 seconds. Ben: Yeah. And the reason I was curious is, I imagine Elementary couldn't run in a browser today, but is there a future where maybe it's possible, especially with like web assembly, could this be ported to run in or some, or all of the functionality run in a browser? Nick: Yes. In fact, it's funny that you say that because Elementary can run in a browser as of today. Ben: Okay. Then I'm wrong. And so what is the, how did you, has it worked in the browser from day one or did you have to port it over and I'm curious if you did use web assembly or what kind of APIs you use? Nick: Yep. So it did not work in the browser from day one, but it was always part of the design that it wouldn't be able to work in the browser. And so this is more, just a way an artifact of like the way that I've structured the code repository, ultimately. There's the brains that handles with DSP. The brains that handles interacting with node JS. And then that middle map brain for the go between. And so the DSP piece of this is really quite portable. And that piece have compiled to walk through, compiled into was and was unloads inside the web audio API. And then Elementary can speak to that. Basically, the whole runtime lives inside a single web audio node and Elementary can speak to that. So that's an exciting feature that will be publicly available pretty soon. But I want to get basically, a playground up on the website where you can go write Elementary, hit run in your browser. And you're right there for the web audio API. Ben: That would definitely be awesome. And it is web browser access, a midi device. Is there an API for that? Nick: Yeah. There's the web mini sec, which I don't have a ton of experience with. I remember finding it years ago when I was more into the web audio API. And I haven't looked at it the way that it will integrate with Elementary yet, but it's an orthogonal piece. I mean, as long as you're getting some kind of like event call back from the web API, you could just call elementary.corE.render again, after doing your [inaudible 00:29:20] stuff. Ben: And aside from running either at the command line or the browser, I think you mentioned earlier, guitar pedals or a doll plugin. Where is the, the other places that you see this running and being useful? Nick: Yeah, totally. So in a lot of ways the browser and web audio or the command line tool that you can use today for running your code are hosts or harnesses for the core run time. And the core runtime is totally designed to be lifted out of either of those harnesses and embedded wherever you want it to go. And so I also mentioned the plugin that I have locally for hacking with Elementary inside of my door. That is an application of embedding the Elementary run time in a target staff. And so I could see this going on all kinds of places I would, I'm personally really excited for when it's more broadly available inside of plug-ins. I would love to see this run natively inside of dot itself, which you could totally do or inside some native hardware. I mean, that guitar pedal is a great example. There's a lot of synthesizers these days, analog synthesizers, which do some of their synthesis or some of their effects digitally in a DSP chip that lives inside the board, things like that. I'd love to see it start to reach places like that. Ben: And Elementary is on GitHub. It's open source, right? Nick: Parts of it. Well, actually, this is a good question because I'm getting this question a lot. My goal is to be able to open source parts of Elementary. And there's a couple of pieces that I feel are, are pretty ready to be open source. Like the library, a lot of that code is, I feel like that should be out there and I want that to be out there because I think that's a good learning experience for people who are new to DSP. But currently most of Elementary is actually closed source. And that's an ongoing debate that I have in my head. I mean, ultimately my goal is to make this as open as I can, but also ideally to build a model around this that allows me to keep working on this and ideally work on it full-time. Nick: So I want to, I think basically, right now the core runtime piece that's embeddable will be behind a commercial licensing model, with the goal of generating enough revenue that I can keep working on this. And as I've developed, I hope that develops. As that develops, I want to make pieces as I can source and get that out there. Especially with the way that this interacts already with the no JS and MPM community, I mean, it's such an awesome community. There's so many active people in that community and I love to be a part of that. Ben: Awesome. Well, Nick, thank you so much for joining us today. I've loved learning about Elementary and getting a one-on-one education to audio coding, and DSP. So this has been great. Nick: Yeah. Thanks so much for having me. Brian: Thanks for listening to PodRocket. Find us at PodRocket pod on Twitter, or you could always email me, even though that's not a popular option. It's Brian at LogRocket.