The following is a rough transcript which has not been revised by Vanishing Gradients or Jacqueline Nolis. Please check with us before using any quotations from this transcript. Thank you. hugo bowne-anderson Hey there, Jacqueline. jacqueline nolis Hello. How's it going? hugo bowne-anderson Welcome to the show. I'm great. How are you? jacqueline nolis Good, exciting to be on your new exciting show. This is good. hugo bowne-anderson I'm so excited. And you and I, you know, we've been chatting for years, but we haven't released a podcast episode in several years together. And a lot has happened since we last spoke publicly. jacqueline nolis That's right. Or like, it feels like we're bringing back the band. I mean, you're bringing it back to the band. I'm just a guest in your band. But I'm very excited to be a part of your band. Yes. hugo bowne-anderson I appreciate that. But a lot has happened in your career as well. And I'm really so one of the things we've talked about last time is the importance of PowerPoint presentations in data science. Since we had that conversation, you've been doing a lot of machine learning work in production, I think that'll be a great conversation. The other thing that's I'd love to talk about at some point is that your transition mid career, and I think that'll be very interesting for our audience as well. jacqueline nolis well transitioned gender! hugo bowne-anderson also in your current job at Saturn, you're designing products for data science scientists, instead of being the customer. So you know, hitting all these points, I think will be very interesting. Well I'll find very interesting. And hopefully, our listeners will as well. But before we get to all of that, I think setting the scene with a bit of context around your career, how you started working in data, and how that led you to where you are today could be interesting. jacqueline nolis Okay. I will go through that. So hi, listeners, I'm Jacqueline Nolis, I started my career, 1015 years ago, I got an undergrad, a master's in math. And then I'm like, I'm gonna go do math for businesses. And now that's called data science. Back in the day, it didn't really have a name, maybe business analytics, or something. But I did that for a while, Oh, my God. I didn't want to do that, though. Because I'm like, I want to do something real. Yeah. hugo bowne-anderson When I finished grad, actually, just before I finished grad school, everyone who was going into industry was going into finance. And I was like, far out. I don't want to, I don't want to do that. And then so I went down the academic track, and then data science or something became a thing. I was like, Oh, I can use these skills elsewhere. jacqueline nolis Yes, I will say in like, 2010 ish, that time period. At one point, I was interviewing for a job and the interviewer straight up was like, why don't you go work on finance and make more money. And I'm like, I don't like, I don't want to work in finance. I want to work at a company that builds stuff instead. And like, I don't know, I got suspicious eyes like, so I don't think data science is really, I think people understood or knew about back. hugo bowne-anderson Yeah, and I did go to some recruitment drinks at several financial institutions, which were truly like, I definitely did not belong in those places. And that's not to disparage anyone who does belong there as well. But it wasn't for me jacqueline nolis And also in 2009, when I was interviewing, that was a common thing, where as part of the interview process, they would take you out for drinks and in 2021 that is mind boggling. Like that is such a deeply concerning thing that like the company wants to get you drunk as part of the recruiting process. And but I was straight out of school and didn't know any better and was like, Okay, I guess that's just how the business world works. But that is not how it works for me anymore. hugo bowne-anderson So mathematics and industry business decisions BI, data mining, whatever predictive and or whatever. So what happened then? jacqueline nolis I hated it. Well, cuz like, I was straight out of math, and you know, straight out of a math program, you want to prove theorems, and develop a new methodology and all that and like my jobs, like started with, like, using Excel, to take SAS data and make it into a PowerPoint we're like, you're just hitting run on the same report, someone else made once a month, like it just didn't feel technical felt very much like, just kind of like, rote work kind of thing. So then I went and got a PhD, which was a good stalling tactic. And it gave me a little more time to decide what I want and do the macro thing. realize I didn't want to go into academia and be a professor. In industrial engineering, but it was really like, operations research. So like, how do you? I did my PhD was in like, how do you decide where Tesla should put charging stations and like route cars to the right charging station? So it's very much like a math problem, but just an applied math problem? Yeah. But then, so I did that. And then during my PhD, I started doing consulting on the side for a tiny consulting firm. And then I like that. And so I finished my PhD. And I went into consulting, and I did consulting, became a director of a consulting firm, and then, and then I switched to working on my own as a consultant, then I was on your podcast, which is a big milestone in my career. But it was exciting. And actually, it's funny because I was consulting very much my career was working with companies taking ideas, doing data science to make PowerPoints and helping the companies understand like use data, understanding, make business decisions, and then it was on your podcast. And like a month later, my career at that point, then switched to actually just building machine learning models, which is like totally different. Well, not totally different. It's somewhat different, where the idea of what I was making went from power PowerPoints and ideas to help business people make decisions to actually models that would get deployed into production, which I have a lot of fun with. I liked it. My skills of making PowerPoints and convincing people things was very useful because I could help convince people that my machine learning methods were good, but I did that for a few years, as a freelance consultant and then COVID hit and I'm like, I don't want to consult anymore. This is really risky and stressful. And I went back to the industry. And you know, today I'm now a head of data science at Saturn cloud, which is a company that makes products for data scientists. And I help design the data science product and help, like use data science on the platform to help, you know, make it better and stuff to and lead the team that does that. It's a lot of fun. I've enjoyed my journey, I would say hugo bowne-anderson , Yeah, well, it's a fascinating journey. There are so many points in there, which I kind of want to zoom into. I mean, from developing insights, and helping people, decision makers make decisions under uncertainty using dashboards and slides is incredibly important to deploying products and machine learning models to production, all the way through to building products for data scientists to use. Before we get into those things I got to tell you, on and off, I used to be very bullish on data science and every now and then I get relatively disillusioned, so I'd love you to inspire me. And just tell me why you think what we're doing is even important at all. Why is data science important? jacqueline nolis I struggled with this a lot to and I think that's probably like anytime you're in a field for long enough, you start to get jaded about it up, like, hey, sometimes what we do isn't as useful as we think it is, and things like that. I think there is a net good of putting effort into understanding how things work. And I think at its core data science is using one way data to try and get just a better understanding of how the world works. And it's not going to be perfect. And it's not sometimes data is going to mislead us and things like that. But like at its core, it's still better to try and do that than to just be like, we will never investigate how the world works. We are just going to, you know, play it fast and loose like no, I think there is net good in, you know, trying to do things a little better. And I do think data science has a lot of ability to make things better, sometimes makes things worse, but also like, you know, there's I think it is an endeavor worth trying. hugo bowne-anderson I love that. And I love actually we need optimism as a strategy, particularly now getting given everything that's going on, I do want to play devil's advocate and push back on that slightly by saying I think all the insights work helps us understand the world and make better decisions in the world. Hopefully, I mean, we need methodologies around that as well. And tying data to to decisions in robust, principled ways, which I don't think we we yet have across across the board. When we're putting machine learning models in production or doing massive online experiments, I suppose there is an argument that, let's go back to Chris Anderson's piece in Wired in 2008, which is called something like big data, why the data deluge makes the scientific method obsolete. And part of the premise here is what we're seeing at the time is a lot of companies investing in big data. And the idea that you may not even need to understand things as much anymore, because you have enough data, right? And when we're putting things in production, we might not be understanding what we're doing and the impact we're having as well. So I think that's probably the contrarian aspect of this. But have you seen that played out and what business leaders actually want as well? jacqueline nolis I'm gonna rephrase what you just said, which is, hey, data science may not be good. Because when people use data in machine learning models and things like that, because they say, Hey, we don't even need to understand how it works. So long as we can see, like a percent improvement on our metric or whatever, it doesn't matter. Like, hey, actually, that's sometimes bad, because on the back end, that's actually secretly causing problems or people get addicted to your social platform in negative ways and develop eating just like they there's so many net negative effects. There's so many potential net negative effects that saying, hey, just using data to do stuff is bad. But it can be bad because you have missed that hugo bowne-anderson exactly. Because you missed the step of understanding is my point. That's why we're so interested in explainable and interpretable and causal ml these days, right? Because there's something missing from the deployment story in terms of understanding the causation behind what we're doing. jacqueline nolis I agree. I will say, I agree with that. But like, that is a real problem. And I think you can point to like, a bajillion articles throughout the news of like, situation where someone made an ML model didn't really think about how it might actually be implemented, like how it might actually be working, if you like, really dived into it. And then it turns out, there was like a really negative effect that came from it of like, oh, by the way, your coupon algorithm doesn't give coupons to this, like, subset of people in a, you know, like, really biased way or but but like, there's just a lot of those. I think that is true. I think there are lots of other ways that data science can also be bad. And yes, I agree. Yes, I think that what you said is true. And not only that other things are also bad, too. hugo bowne-anderson And I'm also reflecting you were the first person who came on my previous podcast, it was like, whoa, whoa, whoa, like, we shouldn't be using data all the time. There are other ways to make decisions and data can can make decisions kind of crappy in some places. jacqueline nolis Yes. And this is my soapbox that I think about standing on all the time when I lay up late at night thinking about data science. The by the way, perfect. people listening to this podcast it is I'm recording in Seattle is 930, which means I'm a little bit more passionate and loopy than I normally am on podcast. So enjoy. Anyway, hugo bowne-anderson Jacqueline is one of the great, great versions of Jacqueline. jacqueline nolis So when I'm sitting up late at night, staring at the wall sitting on a soapbox, I think to myself, boy, people use data in a lot of ways that it actually a lot of decision making ways that are bad. Like, for instance, I will talk about some things that I have seen that are bad, which is companies like say you wanted your company and you want to launch a loyalty program. So like, Hey, if you buy 10, sweaters, your 11th, once free, sort of like that. There are companies where when designing this product, this loyalty program, they say, We will not launch it, unless we have the data to prove that it's going to be a good positive return on investment. But there's no data that will tell you if a loyalty program that doesn't exist yet will work well or not, because that is not how data works. Data is historic, it doesn't tell you how we think that doesn't exist yet will go. But because the company says hey, we need to use data and make a model and decide how this works, people will just create a weird financial model that says whatever they want, and justify whatever. And that can be idea of using like by trying to force data to help you make that decision, you're doing worse than just saying, hey, you know, the executive who's designing this ran three loyalty programs before this, they probably have an idea of what works or not from, like, past experience. And that is just as value as valuable as an Excel file that has a you know, financial model on it. I've seen that happen a lot in my career. hugo bowne-anderson And speaking of Excel, I wasn't actually intending on going going here now. But it looks like we're jumping, jumping right in. So strap in for the ride everybody. Speaking of Excel, I want to talk about several examples of massive data science and statistical communication fails in the past several years. The first one or the one of the most prevalent I think, is the COVID excel cubic model. And I'll include a link in the show notes to that, but I presume most people will have seen the nightmare that that was whatever recently happened at Zillow. I honestly I don't I don't give a shit about getting into the details of what happened at Zillow. I think one thing that's clear is that, like an executive team, blaming an algorithm is ridiculous. And they clearly made poor decisions around incorporating machine learning into broader business and corporate risk assessment essentially, then going back to five years ago, five, six years ago, the inability of mainstream media to predict Trump's victory, or communicate the uncertainty around the prediction except perhaps for for Nate Silver. These are all big fails and or big communication files. And my question for you is, I honestly think we are entering some or have been entering some sort of credibility crisis in the space, I think these are harbingers of something bigger to come. I'm not I don't want to be too apocalyptic. I know that all industries and technologies have fails as you go on. But the amount of hype that's been generated around this, followed by these types of things I personally find quite concerning. jacqueline nolis Okay, so I'm gonna I agree, and then the maybe slightly contrarian or like, add some nuance maybe, which is I agree broadly, that there is a hey, wait, data isn't good for everything, or like give me like, like, like, Hey, your data told me that Hillary Clinton was gonna win, and then Trump win and see data can't always be true. Like there's, there's kind of like shock Can you believe it? hugo bowne-anderson Yeah. To be clear, also, we're losing public trust and executive trust and business leader trust in the field. jacqueline nolis Yes. But my contrarian point is perfect: we should lose some of that. Because I know I'm Josh, I really just 930 throughout the wild statements. No, here's, here's the reason why. Why I think it's good, which is I think what was happening is like, I'm going to pretend I'm like an analyst of a company, or I don't know, I'm analyzing that Trump election, whatever. And I'm going to be like, hey, you know, my model says that there's a 30% chance that Trump will win 70% chance that Hillary will win. And then my boss goes by boss puts out a press release, saying, Hillary will win! The data says it. And then it's like, well, as an analyst, I'm like, Ah, that's not exactly what I said. But okay, sure, whatever, you're my boss, I can't really fight that. And then Hillary loses. And then your boss is like, we can never trust data again, and you're an analyst, kind of like, um, and if you're a data scientist listening to this podcast, I think there's very likely you have a personal story of something like this, where you need a statement that was like, loose, you know, you tried to put caveats in there, and someone ran with it in a way that is incorrect. And then they got burned, and then everyone's mad. And I'm rather people get burned to get mad, and then we move on from that, then try and let the people just kind of run with data in ways that are odd, but just kind of back to my previous point of people use data to justify a lot of stuff. And I think it is good that we use data to like hone in on things and like try and make better assessments. But like, people who want to get what they want, will use any tool in front of them. And hey, data science is a cool attractive field, the 21st century blah, blah, blah, new data is the new oil, they will use that tool. And now we are learning that like hey, that tool can be abused. And so we have to add nuance to what we do. And I think that's a net good. hugo bowne-anderson and the oil industry of course has a lot of large negative externalities as well. So even if data were the new oil, that doesn't mean it's it's all that I do think the other thing around making decisions under uncertainty, the Trump example, I think, let me get this right. But um, some people said he had like a 20% chance of winning, which I think was misinterpreted in a lot of ways. And I'll link to a post that Allen Downey wrote at the time about this, or maybe just after, in which he made us do the thought experiment. If you flip a coin twice, getting two heads, there's a 25% chance of that, and you wouldn't be that surprised if that happened. Now, if you flipped it three times, and got three heads, that's slightly more surprising, but not super surprising. And there's a 12.5% chance of that happening, right, which in our heads seems really low. But when you do that thought experiment, you realize it isn't actually that low. So we, as humans don't even really we're not actually good at thinking probabilistically, statistically, under uncertainty, right? jacqueline nolis Yes, exactly. And I think when data science first came out, people didn't even necessarily consider that as much that just kind of like, cool, the data scientist told me XYZ. And now we're starting to realize as like a, I don't know, as a grandiose generalization of our field of like, the people around data scientists are starting to notice when data scientists say, hey, there's nuance here, like, you actually kind of have to listen to that. Or at least that's my hope my idol is I don't maybe, you know, your project manager won't listen to that stuff. But like, hopefully, people are moving more towards understanding the stuff that all has nuance, and you can't just hire three data scientists, and suddenly, everything in your company works better because the data scientists use data to do stuff like No, you really have to, like sit on it. Yeah, I think it really comes back to like data can't predict the future. It just tells you the past in different ways. And just like taking the time to reflect on that and understand the nuance hugo bowne-anderson tells you the pause based on whatever your way of collecting and interpreting the data was as well infiltrated by all types of biases in the data generating process, etc, etc. jacqueline nolis Yes. And in fact, here's my example that I think about is like my quintessential in my head example of this: suppose you're a data scientist, and you're building, you're working in a company that gives out coupons to restaurants, and or to your restaurant chain. And your company has an idea that like, hey, maybe we can try and understand what you do in your first purchase to predict what's going to happen next. And suppose you as a data scientist, notice in your data that a cheese pizza, if you buy a cheese pizza, and your first purchase the first time you come there, you will spend more money in the rest of your purchases. Suppose your data tells you that what do you infer from that? Should you A) give out coupons for cheese pizzas to everyone else? Because she's pizza is a if you spend the cheese pizza that will get people to spend more? Should it be, Hey, we should stop giving coupons to the cheese pizza people because we know they're going to spend more anyway. So we shouldn't do that? Or is it C) Like there's like there's maybe 10 different ways you could make a reasonable guess of what you do with that information. But like the data won't tell you what happens if you adjust what things you're doing to the customers, right? It only tells you what happened historically, with people who bought the cheese pizzas, not what happens once you start giving coupons and all of that, which is fine. We have A/B tests, we have lots of things that can try and do that. But like oftentimes, you don't have the time to run an A B test or do an experiment. You just have two resources. Yeah, and that's fine. But when you have business, you know business people or you know what, whoever who don't necessarily understand the nuance, just be like, Oh, hey, well, the data says more money after your first cheese pizza purchase. Therefore, I'm gonna do whatever I want with that information. Like, that's where the stuff gets scary. And I think we are starting to reckon with that more as I feel hugo bowne-anderson I love the cheese pizza example. And I'm, I mean, I love cheese pizza, but I may steal that with your permission. Yeah, go for it. It's not quite stealing if I know I'm okay. jacqueline nolis Yeah, sure. hugo bowne-anderson Okay, um, because the example I've given historically is, you know, churn prediction is something we seem fascinated with telling beginning data scientists about for whatever, for whatever reason, but if you predict someone's gonna churn that doesn't tell you what type of intervention to make, you want to understand why someone's churning, right? If they're churning because you haven't answered the phone, as opposed to a competitor undercuts you, your intervention is going to be totally different. jacqueline nolis Yeah. And if you've ever made a churn model, what you realize is your churn model doesn't tell you, this person see our percent likely to churn this person's 100%, like the churn, tells you that person's 2% likely to turn and that person 3% likely to turn, which is indeed more, but it's not like they still probably won't, you know, so it's like it's not cut and dry. It's just it's a very gray thing. And it's not obvious even if you have a reasonably good model, what you would do with it hugo bowne-anderson was something I really like here that I'm hearing is that all of these like, data science fails or AI fails, help us as a community with all the relevant stakeholders just be more realistic about what's possible and what isn't. jacqueline nolis Gosh, I hope so. I don't know if that's true, but I hope it's true. Yeah. Yeah, that's the optimistic. Yeah. hugo bowne-anderson Yeah. I'm wondering, are there any fails that you've experienced because people talk about successes in public or not, and most of the files we see publicly seem like potential PR stunts as well. So can you tell me about any files you use? have experienced or been part of? jacqueline nolis Hugo I have a talk that I give some times that is literally the talk is just five of my biggest fails in the field. So yes, I have I have many of these hugo bowne-anderson maybe, could you tell us a couple that you think could be instructive for, you know, the widest audience possible? jacqueline nolis Yeah. So sometimes some of my sales were like, I would have a data set. And it'd be something is like perfect for like a churn model. It's a churn model. And I, as a data scientist, would be like, Oh, heck, yeah, I'm gonna take that data, I'm gonna build your churn, model our business, we're gonna great from it. And the moment you actually get the data and try it, it doesn't work, right? That there's no actual signal in the data. So that model doesn't fit. Or like, you know, the model does fit. But like, because the churn predictions aren't strong enough, you can't actually act on it or whatever. And like just data science as a field, you don't know what's gonna work until you try it. But you kind of have to tell, like, sell people on it before you try it. And that conflict sometimes goes bad. So that's a type of failure I've seen. hugo bowne-anderson Then when, like, with the decision maker, or the person, if your consultant, whoever's paying your contract, or, jacqueline nolis yeah, well, if you're smart, what you do is, before you do it, you say, Hey, we're gonna take this data, and we're gonna try and build a model. And if it turns out, there's this a signal, we're gonna act on it, if there's not a signal that we're gonna abort the project, and not, you know, we're gonna abandon it, we're not gonna think too hard about it, we're just gonna move on, that's a smart thing to do. Now, that is not always what I have done, because sometimes it's just easy and fun to get excited. And then it's like, you really just have to be open and honest with your stakeholders of like, hey, we thought this would work. It did not work. Let's talk about like, how we pivot? Do we try using the data for a different thing? Do we whatever can we infer that hey, this didn't work, but also total, this eight other things won't work. So we know not to waste our time on them. Like you just kind of salvage whatever you can from the experience. Great. Yeah, that's one type of failure. What's another failure mode? Okay, here's my favorite failure I've had in my career. So let's go back to that, you know, like, suppose you're designing like loyalty programs, or, you know, like you're working for marketing, and you're designing coupon systems and things like that. And before you launch a big coupon marketing campaign, you do want to make a prediction on how much money it's going to bring in, what you would typically do, if you're in like consulting, or finance or whatever is you would build a financial model of how much you think you will make from this coupon. So you say something like we're giving out 100,000 coupons, we think 20,000 of them will be redeemed, the people who buy them will end up spending an extra $20 and blah, blah, blah, therefore, we will make a million dollars from this campaign. I as a data scientist said, Hey, doing things at a grandiose like a super aggregate level in Excel isn't that accurate? What if we built a agent based simulation tool, where we'd actually simulate each customer and be like, Well, if the customer got the coupon on this day, they would have made this purchase they would have ever bought, like I build a whole system for that. And I used all the programming language and models and stuff. And I and ended up kind of working like this, it actually would make predictions and the simulation run, and it would do all these things. But here's the problem, which is, I actually ran it for companies and customers and they got these numeric predictions. And it'd be like, Okay, well, it's cool to know that the model says we're gonna get $100 million in revenue, or whatever. But our financing team needs to sign off on it. And they don't understand like, they can't understand like your weird regression prediction thing, can you just give it to us as a couple simple formulas like in Excel. And so despite having a giant machine learning model, that all these things, I would then have to go redo it just at the last minute in Excel, which is the original way of doing things. Because what I predicted the thing that the company want is the most accurate model to give them the most accurate ROI prediction. But that was not the case, what they wanted was just proof that it won't bankrupt the company that they can hand hand to finance. And the Finance says, Okay, you're good. So that was an interesting failure. hugo bowne-anderson That's fantastic. Because it's actually dovetails really nicely with this idea of needing to understand what you're building as well. And this was, yeah, the decision makers coming and saying, Wait, we need in order to manage the risk, we need to understand even if the model is slightly less performant, or whatever it is. jacqueline nolis Yeah. And like, companies, when I was doing this work, like companies wanted the cool model, like they got interested, like they were sold, like they legitimately said they wanted that. But then just at the end of the day, it turns out, like, naaahh didn't really want it as much as they want. Just the ability to loosely say, Yeah, this will make us and finance will be okay with hugo bowne-anderson that. I mean, the comms team will probably still call it AI, right? jacqueline nolis Yes, and we got, you know, we would get customers and stuff. So like, it was technically good for the business, but it wasn't ultimately what people wanted. And also, the other problem, and this kind of goes back to I was talking about earlier, which is like, okay, when you're making this Excel model version, you'd be like, well, 20% of customers will buy the coupon, and they'll each spend an extra $20. And if you look at that, you'd be like, Well, how did you get 20%? How did you get $20 for those? And if you're a consultant, what you say is my experience from working in the market tells me these numbers, which mostly means I don't know, I guessed and it felt right. Yeah. And they like ah, but I don't want well, you just loosely guessed out of nowhere, it felt right. I wanted hard, concrete things. And so the machine learning model feels like it's a hard concrete thing. Look, we're doing all these regressions and blah, blah, blah. But like if you really dig into the model, your model and your agent based simulation still has lots of assumptions in it, right? Like, you're still like assuming a normal distribution and you're still Ba, ba ba. So it's actually still just as much made up pretend as the Excel version. But the made up pretend is less obvious. Because it's not like, oh, I guessed a 20%, churn rate or whatever, or coupon redemption rate instead, I guess a normal distribution or a blah, blah, blah. So it just becomes much fuzzier. So it's like the business people don't get to see it anymore. But it's still there, which is worse. So I didn't like that. I didn't like that part. hugo bowne-anderson Yeah, I understand that part of me wants to move on. But why don't we want to hear one more failure mode? If you have one? jacqueline nolis Let me think, Oh, my God, okay, here's my, here's another good one. This is my first job out of school. And I was like a month in and the company I was working for was an E commerce company, and they had a bug on their website. And sales were down by like a lot of money. And they only found it because the marketing people were looking at the Daily Sales Amount, and they're like, why is money down? Why didn't we not make money. And then they went to the engineering team, who eventually realized, oh, deep in the website was a bug. But they didn't notice until the marketing team noticed. So they came to the data science team I was on... their analytics is called at the time and they said, you know, an executives like, hey, this happened on a Tuesday, can you send me like the last eight data's a Tuesday because I want to look at how they compare. And I was like, a month out of school. And I had no idea not to, like just say these things, which is maybe probably good for me. But I'm like, hey, no, instead of doing that, well, I'll do that. But what I really want to do is actually build a whole statistical quality control model, like an anomaly detection model, where if we use all the data, you know, because the company is growing, so we need to account for that, like growth, and there's weekly seasonality and yearly seasonality. And we could build a huge model that tells us on any given day, how are our sales comparing to what we predicted and alerting that way. And that was like, people bought into that. And I went and built it. And it ended up being called the Adler alert, because that was my maiden name, which is not a good thing to have happen. Because anytime something goes wrong, they come to you. So that was a lesson for me. But what was worse was that the model didn't work. And it didn't work, like the actual prediction of like, what happens each day was reasonably good. But what didn't work was we didn't is very hard to set the bounds of like, well, how many standard deviations away from our prediction is enough that marketing should be worried. That was problem one. But then problem two is marketing didn't say, hey, we just want overall sales. They're like, well, we want this by region. We want this by not just predicting each daily sales. But also we need, we need predictions on average order value, and percent of customers who actually make a purchase, and blah, blah, blah. And so we end up having like 15 metrics, we were measuring each day, with uncertain bounds for whether it's is the problem. And I made this all in one giant Excel spreadsheet with a million shades of red and green, and it just became incredibly unusable. So the core idea, I think, was good, like you statistical quality control. But the actual execution wasn't good. Because things like what what is your UI look like? How do you avoid having an overflow of too many, like anomalies going off all at once, like all that stuff kind of tore it apart. And it was a lesson for me of like, it doesn't matter how good your model is, if all that other stuff isn't good to your product won't actually be useful. So that was kind of a failure. hugo bowne-anderson Yeah. And something that really speaks to, to reiterate, is the utility of what we're building, right? And what the user experience is like, and how people use it to inform decisions and or deploy things into production. jacqueline nolis Yeah. And you think, well, that's not really my problem. I'm the data scientist, not the UI designer, or not the product owner or Blablabla, but but it is always like, it's always your problem, you can't escape it. And that has been a ongoing lesson I learned over and over my career. hugo bowne-anderson So I'd love to shift gears now to talk about, as we discussed earlier, last time we spoke, you're very interested in helping people make decisions, use it like using insights in PowerPoints and dashboards and this type of stuff. And since then, you've done a lot of work in putting machine learning models in production. I think this will be interesting for a lot of people, particularly as correct me if I'm wrong. But the latter can be far more lucrative as a working data scientist these days, right? jacqueline nolis Yes, and no. So I think it is probably the case that if you are a like, quote, unquote, machine learning engineer, your salary is quote, unquote, more, I don't know why that one, the quote, your salary is more than a quote unquote, data scientist. But I also think there are far more data science jobs out there, and there's more data science work. So even if you have a machine learning engineer with like a higher salary, because there's not necessarily that much work at any given company, you might often find yourself doing regular software engineering a lot. So it's like, it's kind of not like, ah, this is the strictly superior career to be and it's like, they just kind of like, what trade offs Do you want to make, you know, for kind of like optimal happiness? hugo bowne-anderson Yeah, I think I'm also speaking to almost in the cultural consciousness or there is this kind of, I suppose gatekeeping of ml in production, or not quite well, there is the gatekeeping that isn't quite what I'm getting at. I'm also where my mind is going. There's a wonderful post that I'll share in the show notes by Mark Saroufim, who he's in developer relations at py torch, it's called. The post is called Working Class deep learner and it kind of takes you through a day. In the life of someone who's not necessarily a machine learning engineer, but kind of doing all just the building models and debugging and all of these, these types of things, so we almost, I suppose what I'm getting at is there's almost some, like, bizarre classes, like elitism happening here. Right? jacqueline nolis Yeah. And it's... okay. So I think you're hitting on a couple things. One in machine learning engineer, people like, idealize that machine learning engineer means you are like, doing complex gradient calculations, and like, oh, but the neural network architecture needs this many layers to blah, blah, like, like, you're like, like, you know, your GIF with like, 1000 equations flying around us know, like, you know, like, mathematical wizardry. And like, if you're a data scientist, you know, you're not doing that. You're just like, Okay, we're cool. You ran out of logistic regression. And that's like, and I think in practice, that's not, I think, most machine learning, machine learning. Job Work is still scikit-learn, you know, our logistic regression. Like, I think most of the work in machine learning is not that like, how many layers like like, I think that is still extremely rare within machine learning. And I also think there is a, and there's like, a weird, like, we put that kind of like galaxy, like like math equation, like that's the true fun, intelligent work. Like we put that on a weird pedestal. And also, and I've complained about this back on the podcast, me and Emily do, which is there's a weird sexism thing, where it's like, ah, the most like the work that like only the men can do is this, like, really mathy rawr. I'm so like, I don't want a job if I'm not using a, you know, a billion parameters in my model. And it's like, actually, running a billion parameter model may not require as much like dexterous thinking as trying to take messy churn data and figure out how to get a logistic regression to run consistently for market. You know, like, it's not, it's not like it's I want a strictly more thinking than the other. It's just like a weird like macho, like, or I need these parameters. And it's just it's, I don't know, I find it. hugo bowne-anderson Let's drill down into that a bit more the machoism. In the field, I find fascinate One thing I've always kind of thought is, like, throwing more compute at something like check out our, like, 1000, GPUs and TP whatever it is, right? It does almost seem like a quote unquote. And this is a term in the common lexicon. So I feel comfortable saying, it seems like a dick swinging contest a lot of the time in all honesty. Yeah. jacqueline nolis Yeah. Yeah. And I don't, I don't know it does. I don't know. Maybe it's because because of where I am, in my career, whatever, I have no interest in that. It's just so like, yeah, it's so like, what's the point, like, cool, your your model has an extra billion, like, I don't know, like, I've been a data scientist for 15 years, almost everything I can do I have done could have been solved by a well crafted logistic regression or linear regression. And I don't think that's true for just me, I think that's probably 80 to 90% of data scientists. And I think that work, we're 80-90% of work or data science work is doing is valuable and useful and good. And I think the work word is a billion parameters as well. But I think oftentimes, those models don't end up going into production into whatever, it's mostly just to get your like, company cred. And it's like, I don't know. It just rubs you the wrong way. Sorry. Yeah, shall we? Yeah, absolutely. I, hugo bowne-anderson so where I want to go now, I think, I'd like to know about kind of your career changing from developing insights to putting machine learning in production, maybe with a view to helping people understand what that looks like, because I assume there'll be a bunch of listeners who do a lot of data science, but maybe don't put machine learning in production. And they'd like to be able to do that so--- jacqueline nolis I've had a lot of people ask me about that, too. And the thing I say is like, I don't think that change is actually that hard. I don't think you have to learn that many skills. Like really, it's like you learn Docker, you learn some, you know, REST API, how do you make an, you know, hit a REST API, me and Heather Nolis have a couple blog posts about how to do this. And are there's a million about how to do them in Python, about like, there's a lot of material out there. The hardest part, I think, is finding the opportunities to try this. Because if you're a data scientist on a team that doesn't think at all about putting things in production, or making API's or stuff like that, it is hard to find an opportunity to start doing that. But if you're on a team that does do that kind of work and you haven't experienced that before. It's very, I think, in those cases, it's very quick to be able to pick up Docker, if you have three data scientists around you who know how to use Docker, they can probably help you. hugo bowne-anderson So what I'm hearing is it's more about exposure to the challenges and the tools. And the questions, then, like getting up and running with all the technical stuff. jacqueline nolis Yes, exactly. And I just have not seen that many people who are like, you know, reasonably happy app like PowerPoint, he ideas he kind of data science who haven't when put in the right situation, been able to transition to the Docker REST API, blah, blah, blah, kind of data scientists like I don't think it's that hard. But I don't think there are that many of those opportunities because of what we talked about. There's not as much machine learning work. And I also think, because so many people want those machine learning work kind of thing. It's just harder to break in. Because there's so many people you're like, please, this is basically busy. I gotta do that type of stuff. So I think if you're a data scientist who is interested in that work, and Do you want to do some of it? Think less about like, oh my god, what what boot camp do I need to do to learn whatever and much more about like, well, how can I create my own opportunities for the exposure to that? Or try and like find the situation at my current company or something like that, where I can pick it up organically. hugo bowne-anderson You also mentioned that you and Heather have written some posts about deploying machine learning in production using R, and I think the our deployment story is under knowing, I think a lot of feel like aka for whatever reason, there's something you know, and I think we're getting better over the past couple of years. But maybe you can tell us a bit like demystify or myth bust. jacqueline nolis Yeah, okay. Yeah, yeah. Yeah, myth bust. hugo bowne-anderson Let's bust the myth that our is no good for deploying models into production. jacqueline nolis Suppose you had a model that you did just want to predict every time a customer made a purchase? Do you think they're going to make another purchase? Or is this their last one, right? Like some sort of tiered model, right? That could be as simple as a logistic regression? With a couple of input parameters. It doesn't have to be a giant neural network of a blah, blah, blah. It's like, okay, cool. It's just a simple logistic regression. Well, how do you get that code to run? Oh, you do all these pipeline things? Or blah, blah, blah? It's like, no, really, all you need is a computer that runs that code, that somehow other computers can hit it or, you know, like, ask that computer what the answer is. And that's a REST API. And Python has REST APIs with libraries like flask with fast API, there are lots of them. And so I think people in the Python community are typically a little more software engineering, like and so they kind of pick those stuff up quickly, cool, R has libraries to do that to R has something called plumber, which is basically the exact same as flask or fast API or whatever, but it's in R. And because of that, you can do the exact same thing with R, where it's like, you take your R code, and you have it served to other computers. And that's what we did AT T Mobile, it got hit R code is getting hit these models are gonna hit a million times a day when this stuff we were building, and it was totally fine. And it worked. And it wasn't a big deal. But I think what happens is because most of the people who are doing this kind of machine learning work are kind of in Python. And most of people, you know, a lot of the people who are using are more on the data science, you make PowerPoints that I think people are just like, Well, since most people use Python, you can only use Python, and I'm not going to think very deeply about that. Beyond there, but like, there are companies that use R, it's not a big deal. There are lots of languages that have rest API's and things like that and model like, it's just not everything needs a million parameters, hyperfast blah, blah. And I think the gatekeeping kind of Python only, like, it just comes with this notion of like for something to be a production, it has to be big data, and it has to be massive, and it has to a bajillion retrain every hour on itself. And it's like, no, you have a model that, you know, that's in R, in Scala and whatever language you retrain it once a month, manually, all this stuff, and that's fine. That's still production that is still enoughrom a data science team. hugo bowne-anderson I also wonder how much of us feeling this way is an artifact of I suppose spending too much time on Twitter as well. jacqueline nolis I think no, I think the Twitter community is less bad. In terms of only Python in production. I think if you randomly pull a data scientist from any company, they say, Oh, well python's the production language. But I also think maybe it's like Twitter's more recalcitrant. Is that a word that feels like a word? I think maybe like people outside of Twitter would be more open to hey, let's try it. I don't know. Yeah, hugo bowne-anderson yeah, I think you can have better conversations or let's say, more well rounded conversations outside Twitter when the algorithm isn't just surfacing. The jacqueline nolis I don't think Twitter's well known for its nuanced takes. hugo bowne-anderson Yeah, definitely not. And that's actually one of one of the inspirations behind starting the podcast again. So the other thing that has happened that we mentioned at the start, is that you've transitioned genders? jacqueline nolis I sure have. Yes. And if you'll notice from our last podcast, my name has changed and stuff. Yes, yes. That is true. No, hugo bowne-anderson Youwill recall that we actually went on retroactively change and not in the recording, but Right, but jacqueline nolis yeah, it was chocolate. Yeah, yeah. But um, to that, yeah. hugo bowne-anderson I actually, I don't have a super well formed question. I'd really like to explore whatever, whatever issues you found most interesting in your transition, but I am interested what it's like to transition in the data science space at this point in time. jacqueline nolis Yeah, so I will say I'm very fortunate for many reasons. One is that I was quite senior by the time I transitioned, right I was already had been a director of data scientists and principal blah, blah, blah. So like, I was very fortunate my career was well established. I also was quite public on Twitter and podcasts like yours and things like that. So I couldn't like go away for a month and then suddenly come back at a new job with a different name and have people make the connection. I just had to bite the bullet. Yeah, I was really worried that especially because at the time I was in consulting, I was very worried that you know, if I transition then people won't take me seriously. I have to deal with transphobia like I won't be able to go in front of a boardroom of people and like you know anymore because people think I'm weird or whatever and like none of that happened to me and like I don't know I'm again fortunate I passed fairly well I have a lot of... careers are established, but like, that actually is good. on quite well, that said, I have had to deal with a lot of sexism. So that was a surprising I mean, that's shouldn't have been surprising, but that was kind of like a really hard thing to be like, I was surprised like now you're a woman congrats. Well, the first job I got after transitioning, I like oh, by the way, and also here's a ton of sexism you had to haven't had to deal with before. Enjoy. So that was a trouble. But the the transphobia, not really big thing, the sexism that was a problem. hugo bowne-anderson I do think you've hit on a key point there, which is being male, I can intellectually understand sexism, and I've seen it occur and all of these things, but I think transitioning makes you like, it becomes incredibly real for you in a way that, like, it isn't real fun for me. jacqueline nolis Yeah, I was kind of like, well, this sucks. Like, this is not good at all. And I knew it would suck. But it's just very tough to go into work every day when you're like, oh, and then I have to deal with a sexist boss and stuff like that. Which, you know, I'm sure women there lots of people dealing on this listening to this podcast might be like, Oh, yes, I've had to deal with a sexist situation. And yeah, gosh, I don't know. I am sorry for all of you. I'm new to this. And it is very awful for me. And, you know, I ended like that said, I, you know, I'm now at Saturn cloud. And Saturn is great. And I'm having a good time. But like, yeah, at points having transitioned, I have now had to deal with a new thing. And that that has been challenging. So I don't know, maybe that's kind of a downer answer. But overall, very happy on transitions of it's good. The only thing that I will say it's been a struggle is just surprised sexism, not surprised transphobia. hugo bowne-anderson What are the biggest challenges you've discovered in terms of sexism in the workplace? in data science in particular? jacqueline nolis Okay, that's a great question. So I was like a data scientist for like 12 or so a bunch of years before transitioning. And like, every, you know, like, quarterly review of me or whatever, I always get the review of like, Jacqueline's so Well, Jonathan at the time, Jonathan is so good at you know, always saying saying the truth and even in hard situations gonna, like get people to try and understand like, the core reality of it and like, like, thank goodness, Jonathan's always speaking out and things like that. I'm like, Oh, cool. And I got very used to that. And I think a lot of my career excelled, because I would say stuff. That wasn't always the best thing politically to say, but people like oh, good for Jonathan saying that, that helped my career. But boom, I let me tell you the moment I transitioned and got a new job, it was suddenly like, I literally for the first time in my life got a quarterly review. That is Jacqueline's difficult to work with. Jacqueline doesn't know how to speak to other people. Jacqueline needs to learn like when to not say stuff. And it's like, huh, like what has changed in the 15 years, just right now, or suddenly, I'm getting that feedback. And that was a real challenge. I did not handle that review, gracefully. I will say, hugo bowne-anderson I can't imagine anyone would handle it. I think it would take the utmost patience and to be able to handle something like that with any grace. jacqueline nolis Right? And but if you're being told you're, you're being told you're difficult, and then you don't handle that gracefully, that is a form of continuing like, Ah, see, you aren't, you're taking this typical to where you've proven our point. And it's like, that doesn't feel good. hugo bowne-anderson Is this something which we're seeing change currently? jacqueline nolis I don't know. I've only I've only been, you know, on this side of it for a couple years. So I honest to goodness couldn't tell you if there's a change. I will say for me personally, it is very much a situation what environment I've been, I've been in a number of situations where I haven't had to deal with sexism. So it's like, for me, it's not like, oh, we are getting better as a society. It's like I have learned how to quickly identify or I've started to learn, I have not learned I've started to learn how to quickly identify these situations and gracefully remove myself from from them when it is not in my best interest to be there. But that's not like a fix for the whole. All of it, you know? Yes. I'm not an expert on sexism in the workplace. And there are lots of people out here have much better thoughtful things to say than me. I will just say I personally am surprised. And I shouldn't an event, but it's just surprising how much it is not a great time once you're in it. hugo bowne-anderson Well, I think, as I say, maybe there's a slightly more sophisticated or the direct way of saying what I said before, I think the surprise comes with experiential knowledge, as opposed to propositional knowledge, particularly when you're on the receiving end of something. And I honestly I my intention wasn't to put you in a position to represent, you know, experts on sexism that that type of stuff I was genuinely interested in. I do I really appreciate you sharing, you know, difficult aspects about what it is to work. Yeah. As you jacqueline nolis Yeah. Well, and I will say, I mean, I have been on a lot of podcasts now. That's me. This is the humblebrag evidence. So many podcasts. Now. I've been on quite a few and like, lots of people want to ask me about like, Well, how do you become a data scientist? How do you get to be a manager, like ever, like I've not many people have been like, so tell me about sexism and transitioning and stuff. So I wish more people asked me about that, because I do think this is stuff we should talk about a lot on podcasts, but I don't know. Thanks. So I guess Thank you, Hugo for bringing up these difficult topics. hugo bowne-anderson I appreciate it. So this dovetails nicely into something I wanted to chat about the fact that you and our good friend and mutual colleague Emily Robinson had a podcast around building data science careers based on a book, or related correlated correlated with correlated to a book you wrote together, we'll include links in the show notes. But with data science careers, Jacqueline, what's the hardest part? jacqueline nolis Yeah, so that is, yeah, hugo bowne-anderson I apologize for asking a question that's easy to ask. And if someone sends you an email, like an email forwarding, like a document that's 12 pages, and it's like, hey, what do you think of this? And it's like, come on, you wrote four words. But yeah, what are some of the challenges in the space building a career. jacqueline nolis So here's the thing I have learned by writing a book about giving good wishes, some form of giving advice, right, like the book, build a career in data science, some form of, hey, here's how to get your first job, how to become a manager, how to blah, blah, blah. It's a lot of like, just practical tips for growing your data science career. And I think that book was a net good for the world. hugo bowne-anderson And you also interviewed a lot of interesting people in the space as well. So it was It wasn't necessarily just your and Emily's thoughts and potential biases there. But you know, getting a community. It takes a community to it takes a village to build a data science career. I'm sorry, I wouldn't say that again. jacqueline nolis yes, yeah, absolutely. Like and I don't think anything in there is like, well, controversial out of nowhere. But but the problem when you have these sorts of books is like when someone comes to you, and it says, like, Hey, I read your book, I listen, your podcast, whatever, but how do I like what what are the things for me specifically, that can help me get the data science career? What about like, what, like, oh, I have this particular hurdle, blah, blah. Like, the reality is, is that like data science is not a particularly easy field to get into these days, it's just there's a lot of people who want to get in, there aren't that many jobs, and the people who are giving up the jobs, like they're like, I don't feel like training people. I just want to hire someone who's a senior. not universally, but like it's just it's the odds are not particularly, you know, easy. And at some level -- hugo bowne-anderson Just hold on a second, though, does this also speak to their incentive system that's coming down from the executive, like, we've invested in ML and where blah, blah, blah, blah, blah, and we need to see this. And we need to see that, so that the hiring managers actually are in a bind of their own in terms of who they who they can bring on, in order to -- jacqueline nolis Yeah, I think it's like a risk mitigation thing. Like, which is to say, right, if someone hasn't been a data scientist before, and I don't know if I believe this, but like, I think you could be like, well, we don't know if once they get this job, they'll be, you know, quickly able to pick it up, or if they're gonna struggle, because they don't understand some things yet, or whatever. Like, you just don't know. And in theory, someone who has senior like, Oh, I've already been a data scientist, like, well, if they've been a data scientist at one job that I know they can handle this job. I don't think that's actually true. But I think that is an easy thing, when you're hiring to tell yourself to limit the field of people. So you don't have to interview literally everyone. So I think it is maybe just a heuristic people use to make hiring easier and faster. I don't know if I like I don't think I like it. And I don't think the logic is sound. But that would be my guess as to why it happens. hugo bowne-anderson So what I'm hearing is that getting your first data science job is one of the hardest parts. jacqueline nolis I think so. And we a lot of times, Emily, and I will tell people, it's like well, don't start by getting a data science job. Start by getting an analyst job or whatever. And like, slowly, like work towards that. But like, that's not easy, right? That's, like, I think this is a kind of the point is getting to, which is like, when people see a book, like build a career in data science, but they really hope is like, oh, what's the easy sweet trick, I can just, you know, or like, like, when they when they talk to us, like, oh, it's easy, sweet trick I can do just to like, really easily. Like, ah, that'll get me the job. If I if I put the word GitHub, if I put a GitHub link to my GitHub on my resume, that'll be the sweet quick thing that gets me the job. And I don't think the reality of that is the case. I think getting career in data science these days, it is a lot of hard things you have to do and it's like it's not easy. And I think we can't make that... like like we can help you and guide you and like the book and like give you direction, but I don't think we can just make that an easy thing. And that is the hard thing when people talk to us is like we have to like reckon with the fact that I can't just give out a here's a quick fast tip to get the job. It's like no, actually we're just gonna explain to you exactly what hard work you probably will want to do. hugo bowne-anderson I want to read to you a message someone sent me today it was a Twitter DM because I think it speaks to a challenge of finding this this type of work. Hi, Hugo. Hope all is well. I'm looking for career advice as I've been stuck on finding a new job or internship. I live in NYC doing my MS in financial engineering part time online. Bs in software engineering, which never practiced, started as a financial analyst, which I've been doing for a decade started doing my MS degree in hopes of finding a better job. Here comes the dilemma. Now, this is he hits the nail on the head here. This guy wrote to me, here comes the dilemma. None of the jobs I'm applying to will be full fledged developer roles, but interviews are literally designed for one. Yeah, appreciate we filter out a lot of you know, false negatives in order to reduce the number of false positives, but this is a real bind that a lot of Early career people find, right? jacqueline nolis Yes, I think there are not many data science jobs. And I would say like high percentage of people interviewing of interviewers are not well equipped to interview data scientists like they just, I think it's very easy to give a bad interview, which to your point here could be giving a coding test on the job isn't actually that much coding, or like for data science, asking you questions of like, well compute this, you know, compute the moment of this particular statistical function, when that has nothing to do with the job. Like, it's very easy to come up with questions that are both not actually accurate to what the day to day job is, and also harshly penalize people who are not necessarily good at working on the spot when people are looking at them, which often tend to be people like women or minorities. And so like, yeah, I just, it's a harsh, like, it's not easy to have all this stuff line up. So yes, I totally agree with your assessment. And I feel the plight of your poor, Twitter DM person. hugo bowne-anderson So for people wanting to break into the space, besides buy your book and listen to your podcast? What Yeah, what are a few pieces of advice you'd give them? jacqueline nolis Yeah. So this is where I get back to like, I don't think there's any advice that can make this easy, but I think you can do is recognize the fact that like, Hey, this is going to be tricky. It's doable, but it's tricky. So think about how can you put things on your resume that look like a data science position, but are not, right, so if you are getting the first data science position as hardest, what can you add on there that's similar, right? Is it a project you did during your, you know, grad school or whatever? Is it a boot camp? How can you help people who are reading your resume be like, Ah, I see, this person could be a good data scientist, without having the, I'd been a data scientist somewhere else. And I think there are lots of ways to do that. And I think there are a lot of ways to do that. And if you're a position to, like, I really don't have anything like that, well, then it's gonna, you know, it is gonna take work to get something on there that will be like that, which is like, like, then like, it's about, like making sure you can set that work up correctly, or decide, Hey, that's not like, that doesn't make sense. For me, I'm gonna do something slightly different, you know, hugo bowne-anderson yeah, something just came to mind for me. And I've seen this happen several times, not a lot. But you know, several times is, people in non data science positions. I used to work with someone on a marketing team who said to the head of marketing, hey, look, I've just built a dashboard to do this. And like, figure out any value, you can add in your current position, and then start talking to people in the company who are interested in data and that type of stuff and see what you can do there. jacqueline nolis Yep. And we talked about that in the book. And that is great, both for getting you experience. And you also may decide actually, I don't like this as much as I thought it would. Yeah, but you know, like, maybe I thought it'd be off, that's a cool TensorFlow, blah, blah, blah. And actually, it is still mostly data cleaning. And like, you know, like, it is both a good practice for you. And you learn a lot about if you like the work and what you don't like and stuff like that. So that is one great opportunity. There's also like doing stuff on your own on the side, there's, you know, enrolling in a boot camp, like there's, there's a lot of different approaches, but there's a lot of approaches, there's not five words you put on your resume to like secret trick code words that get your resume in. hugo bowne-anderson Yeah, exactly. So I think we've been a bit critical of data that well, I've definitely been a bit critical of, of data science, I want to get even more critical in a second, but I want to give us some room to breathe. beforehand. I'd love to hear a bit about what what currently interests you in the space or excites you? jacqueline nolis Yeah. Okay, so I'm gonna plug this maybe not a plug, maybe it's a plug, I don't know. I'm going to talk about why I like my current job. So I'm currently the head of data science at Saturn cloud. And we at Saturn cloud are building like a data science platform, we have built and continue to build a data science platform, basically like a place like, hey, instead of doing data science on your laptop, I want to do data science on the cloud. So I can start up a resource, I can attach a GPU to it, I can attach it to a task cluster, I can blah, blah, blah, whatever. I actually am really excited about this. Because this is the first time in my career, I am making a product for data scientists instead of consuming one as a data scientist. And I think this is actually a lot of fun, because it is fun to be like, I'm going to help the people who I know, and I'm gonna make a product that I think will make people's lives better. And like, I think that's really cool. And that really gets me excited. And I don't know, I've done a lot of like consulting for marketing, stuff like that, where like, I don't know, every day you kind of wake up and you're like, is marketing real, like I like are these email like, targeting like, this is actually a real thing. But like, No, I can look at the product we're building at saturn cloud. And that's really fun. So it gets, which is to say the thing that I really been jazzed up about, as well as Saturn cloud, which is fun. And also, I really have found I enjoy product development of hey, how can I make a product that people use? And like, when I think about data science and like careers and stuff, people are like, oh, I want to build machine learning models I'm about but like, there's like a whole thing of like, Hey, do you want to try building products, whatever that might be? And like that is like, I think tangental to data science and really fascinating and I got really excited about it. hugo bowne-anderson Can you tell us a bit about how data science can be used in the product development process? jacqueline nolis I will say the thinking I don't think I don't necessarily think like ah, you can use a linear regression to decide if this feature is part of your product or not. But I think a lot of the thinking is similar, right? Like with data science, you have to like take a data set, and you have to like, clean it up and you got to build a model. You got to say all the data science model is saying that, you know, like, oh, this thing is important to churn. But is it really important to churn? Or is it just an artifact of the particular type of regression? I'm doing, right? Like you want to do a lot of this like, reasoning about things you only have partial information on, which is exactly what product development is like, right? You're like, Hey, forget saturn cloud, I am building a USB coffee cup warmer or whatever, like, what is actually important to people wanting to use this product like, like, well, we know people tend to buy the color, the new color, should we invest more that way? It's, it's yeah, you're just again, you're trying to make decisions based on limited information. And sometimes that information is data. And sometimes it's not. Sometimes it's just like, colloquially, you're talking to other people in the field and all that night. I think there's a lot of overlap. And I find that fascinating. hugo bowne-anderson I think the importance of qualitative resources in product developments, incredibly important, just speaking with with users also, it can be slightly awkward at some point. But when I've worked in product development, watching users use the product for the first time, like getting them to share the screen and seeing what people do because you think you set up stuff and think they're going to do something, and I'll do all types of wacky stuff. jacqueline nolis Yeah. And that's the same thing with data science, where you look at data science, like you look at your data set, you're like, ah, do customers always buy a cheese pizza, if they buy a cheese pizza, like always spend more? And then it's like, okay, you actually go to a restaurant with one of your restaurant chains. And you look and it's like, oh, yeah, because the people who buy cheese pizza are family, and families buy more. So it doesn't have anything to do with the cheese pizza, it has to do with the fact that if a family is all there together, right? Like, there's such a overlap, and like, Hey, you don't want to just use data, you also want to think about things. And you also want to collect it like information other ways. And like, yeah, I love it's like, it's like, solve puzzles, you know, I don't know. hugo bowne-anderson Awesome. So yeah, now I want to talk we talked about well, we had a conversation being critical of the output of, of data science work, I'd like to now have a conversation to let's take a critical look at what data scientists do on on a daily basis. And I just want to make clear, this is with a view towards doing more fulfilling, more nourishing work that creates more value for the organization's we decide to spend, you know, nine to five, you know, 40 50 60 hours a week working with and working for in a lot of cases. To that end, I've actually I've recently read a book called bullshit jobs. And my intention is not to say that data science is a bullshit job. I clearly do not do not think that. What I do want to say is that I have conducted several surveys throughout data, science communities with respect to how many people think how much of their work is actually useful. And a lot of these surveys, I've seen that, you know, over 50% of people felt that less than 50% of their data science work had any impact on the organization measured in human hours. Okay. jacqueline nolis Yeah. Okay. Well, I will say first, if you are a data scientist, and 50% of your work is useful, I would say that's a great win. Like you're doing great, like 50% Again, but yes, I yeah, hugo bowne-anderson for many, it was a lot, a lot less. And so when I read this book by David Graeber, who's a wonderful anthropologist who sadly passed away in 2020, he wrote an essay around a decade ago about the increase in the number of bullshit jobs in the modern world. And he received hundreds of replies saying what you wrote resonates with me so much. I do this I do it, you know, telling him about all the bullshit they they had to do. Um, so he decided to write a book around it based on a lot of the conversations and and letters he had and and received, he defines I'm not going to give his precise definition. But essentially, the most important point isn't only that these jobs serve no purpose, create no value or create negative value. It's that the person who performs them feels they create no value or negative value. jacqueline nolis I've heard this referred to as job LARPing. Where you pretend to like Yeah, absolutely. hugo bowne-anderson There's I think both Kramer and George do it for most seasons of Seinfeld in a variety of ways. I think there's one season where Kramer just has a suitcase with crackers in it and he sits in his office and eats but um so graeber book, he creates a typology or taxonomy of bullshit jobs, which include what he calls flunkies, goons, duct tapers, box tickers, and taskmasters. So as I said, I don't think data science is a bullshit job per se, but I think it can have elements of these particularly duct tapers and box stickers so I'd love to discuss these with you and duct tape as of course jacqueline nolis I want to read this book so badly. It sounds fascinating. I haven't read it but yes, let's dive on. hugo bowne-anderson So duct tape is actually comes from software, of course. So he writes duck tables are employees whose jobs exist only because of a glitch or fault in the org, where they are to solve what sorry, who they are, sorry, who are there to solve a problem that ought not to exist. One example is taking a bunch of technologies and applying some duct tape to make them work together. Okay. He uses the term box tickers to refer to employees or functions who exist only or primarily to allow an organization to be able to claim it is doing something, as opposed to actually actually doing it. So with that said, as context, do these resonate in some way? jacqueline nolis So I have two data science scenarios of why data science jobs are bullshit that I think maybe align with these two, like somewhat. So it loosely resonates. Let me give you my example, you can tell me if these fall in. So the first one is, you're at a company, they don't have, or there's a company, they don't have data scientists, and maybe they they're maybe they're a company that I don't know, sends out, you know, their company, the sense that that says, You order flowers from them online, and they've never done anything with data science, as it was like, we should use our data on what flowers people order to like, like, like, automatically send emails to people based on you know, what flowers they should buy next in the season and stuff. But we should use data to better use for our marketing. And so you hire three data science team, and someone like some executives, like I'm going to be the data science executive of VA. And then you the data scientists get hired and quick, really quick, like quick, pretty quickly, they realize Oh, my God, this is not like a real like, like there's just not a use for data here. Like your you buying flowers has nothing to do with what machine learning models tell you what email sent and has to do with like, what holidays are happening or like events in your life. But you don't want to look at like go and your boss doesn't want to make it look bad that you have data scientists who aren't doing anything. So you quickly jump from model to model, okay, we're gonna make one that does email marketing, but they do churn we do bah, bah, bah, you're going through all these things. And each one, you're like, Ah, this is gonna be the machine. And just like, none of them can succeed, because like the data, like you're just not dealing with a problem set that really needs data scientists. And I think that's kind of the bullshit job. And I've been there where it's just like, I don't think what we're doing is real. And no one will listen to me when I say that, because like, it looks bad for everyone. So we're just going to keep playing pretend. And so I think that's one bummer scenario. hugo bowne-anderson Yeah. And I think that kind of aligns nicely with the box ticker, depending on the motivation for them doing it. But I mean, the definition was employees who exist to allow an organization to claim it's doing something that it isn't actually doing. So jacqueline nolis yes, yes. Okay. So that's a box ticker. Yeah, I think I would say maybe the majority of machine learning at companies falls into this, like, you're just in scenarios where like, actually, machine learning isn't really relevant here. But because we want to be like, on the cutting edge, and blah, blah, blah, data's new oil, we're trying machine learning, and it's not working out, which is fine. It's a risk and the bummer, and I don't think you can necessarily avoid it, like the people at that flower company may have not known not to hire data scientists. It's just a bummer scenario. hugo bowne-anderson And perhaps the competition was hiring data scientists as well. Or they thought that this was what their type of business needed to do at that point in time in order to gain competitive advantage. jacqueline nolis right. And if you're a flower company has been declining in sales for the last three years, you're gonna try anything, right? Like it's not a it's not a ridiculous thing to try it just whatever may not work, and then we're getting to our beginning. Right? All right. Makes sense? Yeah. And we're getting back into our beginning of the conversation thing we were saying of like, the second type, I think of data science, bullshit job is maybe tangentially related, but I think maybe closer to the box ticker, which is like you're a data science team to help produce ideas. And you are working for an executive who wants power in the company, and blah, blah, blah. And so the executive like knows that every time they go into a presentation, if they have data to back it, that makes them look better, or whatever, you know, so like, or maybe someone is like, give me the whole company, because it's like a, it's like a mutual, It's an Arms Race. Everyone who has data is better off than the people who don't have data, like every team is like arming each of themselves with data scientists and like, I need data scientists on my team to prove my points, right. And I need a data science, someone else like, well, I need data scientists to disprove their point. And so you have a whole group of data scientists who are making all these PowerPoints and insights and things like that. But like, all together, that's not really providing necessarily the company with more information. It's just burning a bunch of money so that people can argue with each other. I've been on those teams. hugo bowne-anderson So I was like, yeah, actually assured data mad. Right? Yeah. jacqueline nolis It's like Kathy O'Neill's book weapons of mass destruction. It's like, it's maybe like a different idea with the same pun. hugo bowne-anderson Absolutely. And who upon which, jacqueline nolis I think oh, yeah, yeah. I'll blame it on the 930 thing, or 1030. Now who hugo bowne-anderson know about the pun was mine? Oh, okay. Yeah, I do think I do. I am attracted to the idea, the analogy of an arms race to who has the most data and who can, you know, build the most sophisticated models and that type of stuff? Because we do, we do see that. And that is not once again, to denigrate the value that's created by data science and machine learning. I want to make that clear. jacqueline nolis Well, yeah. And where it's like we spent the last decade telling you that your data needs to be at your company needs to be a data driven company, like well, what does that mean, at its logical conclusion, that means what we just described, every team has a bunch of data scientists to argue with each other so the executives can do what they think is best. And that's like, you know, if we ease back on the data driven stuff, and really just, like use data, just where it was necessary, exposed to every possible scenario, like we might have actually been better. And I think that is the second type of video science bullshit job and that does not feel fun when you're like, all I'm doing is giving a PowerPoint to a person who's just gonna not really listen to it and just use it as a tool to do whatever they want. It's not great look hugo bowne-anderson was a fantastic examples of the type of bullshit I think we do encounter. And of course, this is not the only thing but I, but I think it's important to identify these in order to figure out what the non bullshit parts are. So we can do more of those. jacqueline nolis Yes, yes. And, and like, I don't know, if I'm going to make up a whole number 30% of data scientists have these sorts of bullshit of jobs. I don't know if the odds are that high, you know, if it's that high, but like, there are jobs that are not that if you're one of those people, like, there is hope, like, there are other jobs besides that. Also, it's not like your job is one of these bullshit things or not, it is often that you were like, half of your job, right? Like, half of the insights are for just weird political stuff, and half is super useful. And like you never really know is like, at, like, you can't really tell when you're making a PowerPoint, if it's gonna be used, because it's needed, or it's gonna be used as, like a political moment. So like, it's all gray and messy, and you can't, like you can't just like we can't, as a field, just like, remove the bad and now we're all good. Like, they're all tied in together, hugo bowne-anderson I'm gonna say something horrible, which is it reminds me of marketing. And in that sense, I mean, there's the old saying that, you know, 50% of marketing works. We just don't know which 50% or something like that, right? jacqueline nolis Yeah, no, I think that is I think that is 100% true with data science. Like we just don't know what, like you can't isolate which part is the useful part of data science, and we just kind of have to try our best to mitigate the-- hugo bowne-anderson and to learn what works and what doesn't, I want to take the liberty to read just one more section of bullshit jobs to our listeners and to you, because this gives you insight into how other people in organizations may view data scientists and the data function. Okay, so this is someone David Graeber correspond with, he changes their name and certain certain details, but he writes, Irene, for example, worked for several major investment banks in onboarding, monitoring whether the bank's clients in this case various hedge funds and private equity funds were in compliance with government regulations. In theory, every transaction the bank engaged in had to be assessed. The process was self evidently corrupt, since the real work was outsourced to Shady outfits in Bermuda, Mauritius, or the Cayman, and/or the Cayman Islands, and they invariably found everything to be an order. Nonetheless, since a 100% approval rate would hardly do an elaborate edifice had to be erected so as to make it look as if sometimes they did indeed find some problems. So Irene would report that the outsider reviews have okayed the transaction, and a quality control board would review Irene's paperwork and Julie locate typos and other minor errors, then the total number of quote unquote files in each department will be turned over to be tabulated by a metrics division. This allowing everyone involved to spend hours every week in meetings arguing over whether any particular fail quote unquote, was real. I rain wrote, there was an event and this is this is the point she wrote, there was an even higher caste of bullshit, propped atop the matrix bullshit, which were the data scientists, their job was to collect the fail metrics and apply complex software to make pretty pictures out of the data. The bosses would then take these pretty pictures to their bosses, which helped ease the awkwardness inherent in the fact that they had no idea what they were talking about, or what any of their teams actually did. At Big Bank, A, I had five bosses in two years, at Big Bank B, I had three, the vast majority were installed, cherry picked by higher ups and gifted these castles of shit. In many cases, sadly, it was how the companies met their minorities in management quota. So that diverge slightly from the critique of data science. But I think once again, this almost emergent class system of qualitative people and people above the API to other people in a workforce is something very much worth dissecting. jacqueline nolis Yeah, I would say I think that is maybe I would say that as maybe more extreme than I think I've ever seen, like that story was a lot like I like pay people listening. I would not worry that your next job is going to be that that like, I think that is unusually bad. hugo bowne-anderson Sorry, I should make Yes. Yeah. I think you said what I make. And this this book is, there is a bias to telling the horror stories of bullshit jobs as well, right? jacqueline nolis Yes. But I do think I don't want to talk about what like the loyalty machine learning, like where it's like, well, now we're not assuming a 20%, churn or whatever. But instead, we're assuming a normal distribution. And like, I do think data science very easily has the risk of becoming a method in which people hide uncertainty. So like the data science isn't providing value, except for Well, now we don't have to, we can pretend that all the stuff we don't know isn't real, because well, it's a machine learning model, which is quote, unquote, unbiased because it's using, quote, unquote, data, do a lot of quotes today. But I think data science does have the risk of really being used as one of these tools of like, actually, bad systems are happening and we aren't noticing because there's data science there, which I do think is a risk. I have been in places where that has happened. And that's kind of a bummer. hugo bowne-anderson Something I'd like to you've spent some time managing data science teams? and it'd be nice to hear so now, after some horror stories, Some of the some of the good stories of how how you manage a team of people and act as an interface between them and business decision makers who need to get value out of a data team. And the ways you've seen that work well? jacqueline nolis yeah, I do think that's kind of the data science manager's job is to, like, kind of deal with that stuff, right? Like, if you have a stakeholder who's like, I don't want to, you know, your AV test, I don't want to listen to it, I just want to do what I want. And like, like, you know, it's kind of your job to be like, Okay, well, you know, we really can't, because of blah, blah, you know, like, it's your kind of job to, like, smooth that over. And I don't, I think that sounds like a not fun job. But I do think in many ways, it can be quite fun, right? It can be quite fun to be the person who like, Hey, I'm going to talk to stakeholders to get them to understand why it is a good idea to use a model here. And it's not a good idea to use a model there, and all of that stuff. And I think like, I don't know, like, this is really like, management book that kind of is full of itself. But I do think like the idea that like a manager doesn't tell employees what to do, like the employees tell what the manager to do. And like, it's like more relation, I do think it's true. Like, as a manager, I'm kind of a team with, I'm like, teammates with the people who are my employees, like, you know, I'm working with them to make sure that they are getting the information they need, they are the stakeholders aren't bothering them, or, you know, and stakeholders are given the clarity, and they're working with me, because in exchange for me helping with that stuff, they will then, you know, create cool models and things like that, that I can then use. And it's like creating that kind of symbiotic relationship. So it's not like a pyramid. It's an inverted pyramid, I think they say, I don't know, I try to avoid management books. And I think that at its best, that can be really good. And I've had managers who I've really liked and who have really helped me with that sort of stuff. And I do think like, Yeah, I mean, the best jobs I have, I've always had, I've always had great managers, because it's just really valuable to like, have people who are who can get your back and things like that. And it can also tell you, you know what, we're not gonna fight that fight. Today, Jacqueline, we're just gonna do what the stakeholder wants and like, save our ammunition for later. Like, like, Yeah, it's nice to have a boss who can help me with that I try and be that person. But you know, it's, it's hard to assess yourself. hugo bowne-anderson Yeah, there's something in there, which involves being an interface between the data function and the decision function, which I think is one of the big unsolved problems of...not that it would ever be solved, per se. But something we can work towards a lot better. I think we discussed decision making under uncertainty and expressing uncertainty to stakeholders is absolutely key. I think the other challenge, of course, is taking business questions and translating them into data questions, and then getting data answers and translating them back into business answers being that interface, I suppose I'm speaking to the idea of like, type three errors, right, where you get the right answer, but to the wrong question. Yeah, making sure you get that right. jacqueline nolis Yeah. Or, right, kind of those things are like, sometimes someone makes a model and like, well, the model, okay, this sounds bad. But like, the model says thing A, and you'd be like, well, doesn't it kind of same thing? B, it's like, well, maybe it's like, Well, then let's have a conversation on stakeholder about thing B and say, like, well, the model only partially shows us we're gonna use this as an incentive to vote, you know, like, like, figuring out like, how do you diverge the original intent to kind of help the business, you know, like, like, things like that, like, hey, when should we stop using the model? For what we originally thought? And instead, use it prove a different point? Or when should we just drop this entirely or like, like, all this kind of meta stuff that gets really messy. And you have to think a lot about people and about the corporate strategy and things like that, that is like, hugely distracting from how do you build a good model, and like, often just very different skill set. And a good manager helps you by avoiding that stuff, but not in a way of like avoiding it until it all blows up in your face, like legitimately smoothing up these sorts of problems. hugo bowne-anderson Yeah, and you've just spoken to kind of, I suppose result, we call it result massaging or something like at the end of the pipeline, but this can creep in anywhere along the analytic pipeline, right? So if you kind of have an idea of what your manager wants to see, because you usually know what they want to see, you can actually make unconscious micro decisions during your data cleaning or analysis process that will lead you in these general directions. Right. So we need to be vigilant all the way throughout the process. jacqueline nolis Yeah, I think that is true. And also, when you are working on a model for two weeks, it's just becomes hard to step out of it and be like, Hey, maybe I should have never made this model, I should have made a similar but distinct in some ways model. Like it's hard to do that. And someone who has a manager who's kind of talking to you and talking to the stakeholder and blah, blah, blah, can kind of more easily see that stuff. They're just like kind of help guide you want these sorts of like, Hey, how can we twist this into a different way? And I think that stuff's really fun and helpful. And like, I like that part of doing the management. And it doesn't have to be a manager who does that a technical lead or a principal on a team can also do that kind of work, but it is kind of like a leadership kind of role to have now, hugo bowne-anderson how do you go about figuring out whether you want to stay managing teams or being an IC or principal data scientist, jacqueline nolis I'll tell you what I did, which is switch back and forth between them like four times. You know, I don't know if I would recommend that. Yeah, yeah, I went from being an IC to like a manager than a director. Going back to an IC and to a tech lead, and then back to a manager, and I've kind of bounced back and forth. And people like say, Hey, be careful, because once you start on the management track, your tech skills are gonna kind of get soft, and then you won't be able to bounce back. And I bet that is often true. But like there are lots of, especially in data science, there's lots of manager roles where you're still also doing coding and things like that. Whether it is literally a manager or like you're a principal, where you're doing a lot of architecture and mentoring that way. Like I think there are a lot of these jobs, where it's like you are leadership ish, but you're also still coding and mentoring and all that. And I think those sorts of roles are nice ways to dabble. That said, I do not think I super enjoyed being like purely a director or like, you know, an executive who, like all they do is go to meetings and talk strategy. Like I do not like that I really like coding and stuff. So it took me a while to learn that, hey, just because something is higher up on the corporate ladder means I should try and get that job and it will be more fun, like, and I think everyone like finding what trajectories for you is a hard thing. And I think we undervalue that kind of as, as a community as a society, broader than data science. We kind of undervalue this, Hey, not everyone has to try and want to be a vice president someday. hugo bowne-anderson Absolutely. And in all honesty, I think we could try to give people advice on which direction to go. But you kind of got to try it for yourself and follow your intuition, right, and talk with people about what you want. jacqueline nolis Yeah, yeah. Which is like a metaphor for what I'm talking about with data science, which is you can't just take historic data and use that to accurately predict what's going to happen next. Sometimes you got to run the experiment, sometimes you got to try being in a more leadership position, whether that's mentoring on your team, getting a position as manager, whatever, like you, like, I have certainly learned in my career that I don't learn until I try stuff on my own. hugo bowne-anderson So if data science would go the direction you'd like, what would it look like, in five years, jacqueline nolis I feel like given especially given what we saw this conversation, I feel like would lurk largely similar to what we are doing now. Except, I think we would have less of the kind of bullshit work right? Like we just get more nuanced in terms of like, where's data science useful? Where's it not useful? How can we tell, hey, this work we're doing now isn't really beneficial. So we're gonna pivot, or, Hey, these PowerPoints we're making are largely us just as political infighting, we should avoid doing this. Like, I think we're just gonna get more I hope we get over the next five year move more mature of this and like, better at knowing, Hey, should we have data scientists in this place at all? Or like, Hey, this is not a place for data science, you know, like, just, yeah, nuanced stuff, as opposed to, you know, like 10 billion parameter models? hugo bowne-anderson How do we get there? jacqueline nolis Gosh, I don't know, Hugo? I mean, it's like, I kind of just think it's gonna be a matter of luck. Have we as a field all collectively mature, and the stakeholders mature, and everyone kind of learns more? Or like, worst case scenario? Like we just trick ourselves into thinking that No, no, these are real problems. The only problem is our models aren't big enough and auto retraining and reinforcement learning like, yeah, it's just a, it's a collective community collective action community, sort of like, I don't know, does it happen or not? Like, how are we going to solve global warming? I don't know if we all can band together and, you know, change our society will do it. But how I don't know how to get how do you get a lot of people that change? hugo bowne-anderson One thing we hinted at earlier, is have more realistic conversations. I think, in the labor market data scientists like it is a hot market at the moment. So I think being more honest with potential employers and current employers around what is actually possible and what isn't, is super important. jacqueline nolis Yeah. Okay. So you're right. I agree with that. And I would say me personally, what I tried to do is give more conference talks, trying to add more conference talks to the world where I'm talking about ways in which I have failed ways in which data scientist is a risky business. I mean, they be careful ways. I've dealt with sexism, whatever, like talks about vulnerable things that are nuanced and complicated, more than, hey, look at this cool model I built in my company that's even bigger and better than what we did before. And I think the more the more of that kind of talk that becomes common, I do think that will help hugo bowne-anderson I feel like we should have like a file count or something like that. Gosh, jacqueline nolis that must exist. That must hugo bowne-anderson be dataspace. I don't know. jacqueline nolis Caitlin, who didn't? Did one, I think yeah, okay. Oh, lol, I'm googling do it and podcast materia? hugo bowne-anderson And we'll include that in the show notes as well. Almost jacqueline nolis positive. She ran a conference early on Monday talk about No, it's great. Great. I did not I did not attend to because I'd parenting. But hugo bowne-anderson But yeah, more stories of fail as well. Maybe someone should start a fail blog or something, something like that as well. So we can all I mean, it's a similar question in science, scientific research, right, that the incentive is to publish positive results, not not negative results. And how do you how do you learn when you do that? I wonder if Yeah, the question is, we've taken we've seen the real big proof of principles happen in tech and in Faang companies, and which I think we do live in the shadow of a lot of these techniques and results at this point and figuring out what data science at a smaller scale looks like. All that stuff is fantastic. for their purposes, and maybe for some of ours, but seeing what actually solves the problems for other types of businesses, and what you'd call reasonable scale or something like that. jacqueline nolis Yeah, I agree. And I think, I think it is a very common thing for people be like, well, how can we be like Facebook? How can we be like Google? How can we make our company more, but like, those are really different use cases. And like, I kinda like that's how like brick and Saturn cloud because our product is for smaller teams. So right, like, I think there's something, the more like, there's a lot of like, there, there's probably 100 companies that are five data scientists doing stuff for every one big tech company, you know, and like, they're really different problem spaces. And I think it's really like, there's so much positive good work we can do by improving methods, tooling, whatever for the kind of more realistic use cases as opposed to trying to make everyone face FAANG company. Yeah, hugo bowne-anderson for example, I think Kubernetes is fantastic for a lot of a lot of things. But it may be overused because of the clout that the Google has as well. jacqueline nolis Does that make it? Okay? We're just on like a light tangent. But this makes you mad as a data scientist where like, I'm reading a blog post or something or I reading about like some tax some data scientists, companies doing it like first. First, we built the model, then we made the model retrain itself every day. And then we added the reinforcement learning and then we put in the ML ops and blah, blah, blah, and you're like, man, like, like, sir, this is a Wendy's like, like, like, your model is like predicting churn once a month, like you don't need all that. And it's like, it really makes me mad when people over engineer this stuff to make it like cool and whatever. And it's like, gosh, your company would be way better if you just chilled out. And that I think that's part of this in five years, we'll have more nuance of like, when do you actually need the model to retrain itself? And when is it fine to once every three months? Put it in? Or, you know, like, like, just figuring that stuff out? All takes time? Absolutely. hugo bowne-anderson I love that you said so this is a way I'm gonna say that to people who don't even work at Wendy's. When this problem comes up, actually, yeah, when they're like, I'm complying with bla bla bla bla bla, I'll be like, so this is a Wendy's. jacqueline nolis I don't think this is a new to data science skills. people joke about like, you know, like, oh, it's time to write your own like, time to sell post a blog. Firstly, the Kubernetes to like host the blog and above, like, right, like sub fringing. Lots of fields have those problems. But yeah, we have it too. hugo bowne-anderson also, to be clear, like, a lot of the concerns and criticisms we're discussing today, aren't particular to data science, per se, we're discussing them in a data science context problem we're figuring out what an entirely new discipline and technology stack looks like, right? As an industry. So I do want to learn from from previous iterations of this such as software engineering, right? jacqueline nolis Yeah, exactly. And learning that like, hey, a light touch can be just as valuable as a heavy hammer of every mL tool we got. hugo bowne-anderson Exactly. Jacqueline, this has been an amazing conversation. Thank you. I had a lot of fun. I'm interested, if you have a final call to action for our listeners, something you'd encourage encourage people to do, jacqueline nolis I will say, see, when you word it like that, it should be like Go bring peace to the world, right? Like it's like, what should you do? If there's a lot of things you do? Well, I would say free. If you're interested in me and what I do and stuff like that, I would say check me out on Twitter, that's twitter.com/skyTetra sky e te tra or Twitter search for Jacqueline Nolis more likely get me Saturn cloud the product I'm working on it's really cool Saturncloud.io worth checking out if you're looking at Cloud Hosting data science stuff. And the book that you mentioned that me and Emily Robinson wrote build clear data science you can read up on that at best book dot cool that is best book cool. And if you use the offer code build book 40% You get 40% of that book which I said every podcast episode with me and Emily and it's like engrained in my brain now hugo bowne-anderson that's awesome. And I didn't even know dot cool. So congratulations on that URL as well. jacqueline nolis i Yeah, that the dividends that that URL is paying is incredible. Like you people remember it. hugo bowne-anderson Jacqueline, thanks once again. Thank you Transcribed by https://otter.ai