The following is a rough transcript which has not been revised by High Singal or the guest. Please check with us before using any quotations from this transcript. Thank you.

===

Daragh: [00:00:00] I don't think every problem needs to be solved with machine learning. Machine learning's a bazooka and you don't need to go after every problem with the bazookas. If and when You can solve a problem with analytics or cleverly deciding what exactly you need to count, you absolutely should with analytics or bi or analysis, but sometimes machine learning is the right tool for the job.

I think machine learning is really powerful when you're using it to generate a prediction and when you're then using that prediction to drive an action. At literati, one of our key decisions is what five books should we put in a box and mail to each child? Mm. I think that's a kind of problem that machine learning is really well suited to is helping us predict how much is each child gonna like each book?

And then for instance, on the inventory side, another really core question that we have is what book should we bring into inventory in the first place? And to do that well, you need to consider how good is this book, but also lots of things about the school. And so [00:01:00] when you have this kind of a problem where it's really high dimensional and lots of different people are gonna need different results and you need to make really good predictions to drive those actions, to me that's a place where machine learning is often the right tool.

One of the things which I've loved is going into legacy fields like fashion design and discovering how much inside of the fashion world you can again inform with machine learning. And we've definitely found the same thing again in the book space, there's loads of really core decisions when you go and you look at it 

Hugo: that 

Daragh: people 

Hugo: are making on a daily basis.

That was Dara Sibley reminding us that machine learning isn't just decoration. It can be the engine that turns raw data into real world action from matching every child with five perfect books, they're helping publishers spot the next bestseller. Dara's team uses models to decide, not just describe. In this episode of High Signal, I speak with Dara chief algorithms officer at literati and former Stitch Fix [00:02:00] data leader about what it takes to run data organizations that own the outcome.

We trace his journey from cognitive science labs to fashion algorithms, and now children's books exploring why so-called legacy domains are ripe for modern ml. We dig into designing workflows where humans and algorithms share accountability, balancing exploration with exploitation in inventory and building incentive structures that reward maintaining the models that move the p and l.

It's a conversation about placing machine learning at the center of business decisions, shaping what gets built, bought, and read while still keeping the human judgment that makes those predictions matter. If you enjoy the show, please leave a review, subscribe to the newsletter, and share the episode with a friend.

Links are in the show note. Let's now check in with Duncan Gilchrist from Delphina who produces the podcast. Before we jump into the interview. Hey there Duncan. Hey Hugo, how are you? I thought maybe you could tell us a bit about what you're up to at Delphina and why we make high 

Duncan: signal At [00:03:00] Delphia, we're building AI agents for data science.

Through the nature of our work. We speak with the very best in the field, and so with the podcast we're sharing that high signal. 

Hugo: So before we jump into the conversation with Dara, I'd love to know what resonated with you in it. 

Duncan: You know, over time so many data scientists have become pretty singularly focused on either decision science or ml and Derah has really refreshing clarity around using both together.

Think about his work at literati, using ML to predict which books will really light up a specific child. I love how he's taking that approach to legacy industries, fashion at Stitch Fix and now children's literature. These are really human problems. Let's get into it. Hey there, Dara, and welcome to 

Hugo: the show.

Oh, hi. Thank you so much for having me on it. Really excited to be here. Such a pleasure to have you here. And of course, we were introduced by your former colleague or current colleague as well, Eric Colson, who I've known for years and had on the podcast, [00:04:00] of course, who was at Netflix, and then Chief Algorithms Officer at Stitch Fix, which is part of your wonderful lineage as well.

Daragh: Yep, absolutely. Yeah. It was a real enormous pleasure to get to work with Eric before the fact that being able to talk him into working with us again is a gigantic pleasure 

Hugo: without a doubt. And of course, stitch Fix was a place where with your multi-threaded blog among many other things, was just so wonderful at not only building data and machine learning native solutions in a lot of ways, but also helped us all over the past decade really think about how to apply these techniques across a wide array of problems.

So I'm interested, you've looked at Stitch Fix and now. You lead both data science and inventory at literati. So I'm just wondering what drew you to these kinds of companies? 

Daragh: Yeah, absolutely. So I guess maybe at my, I'm gonna risk giving you my life story here at my core, I'm really recovering academic. I spent about a decade in academia building [00:05:00] neural network simulations of how kids learn to read and then testing their predictions with brain scans.

And I really loved that. But after a while, I really started to get frustrated. I felt like I wasn't having a load of impact on real people's lives, and I wasn't pushing my scientific field forward in as big of a way as I would've liked. So I thought about this problem for a little while and I decided that the big issue was that I had the same training and skillset and mental models as all of my colleagues.

And I thought this really decreased my chances of having a really novel and impactful contribution on the field. And so as I was leaving academia, I sought out companies and domains where people didn't have my kind of background. And this landed me after about a decade at Stitch Fix, which is, which was a startup of that phase.

And the whole idea there really was that we were gonna build really like a tech enabled fashion styling service. And I was really lucky with what I found there. At the time, there was only a couple of other data scientists and engineers [00:06:00] at the company, so I got to spend almost all of my days surrounded by stylists and buyers and fashion designers.

It was a load of fun working with them because my cross-functional partners, I had such a fundamentally different way of thinking about the problems we were working on, like how to design a great fitting pair of jeans. As a result, it felt like I got to have a novel idea on a daily basis, and many of these ideas could be really rapidly explored and quite a few of them proved impactful.

And for me, that was just an intoxicating environment. It felt like a playground for new ideas. So that's what drew me to Stitch Fix. I was there for about eight years until I was ready for something new, but I still was craving that same sort of experience. So I sought out another company in Domain where I felt like there was a lot of green fields, and I've been there now at Literati for about four years.

Literati is a a retail company ultimately for kids books. And so the cool things to find that [00:07:00] kids books and publishing is still pretty untouched by people with like my kind of background in science and statistics and machine learning. So I joined up and been really fortunate to discover that again, there's lots of green fields to go and explore and build things and try new ideas.

So a lot of fun. Amazing. 

Hugo: And I'm interested in why machine learning at both these organizations as opposed to analytics. Causal inference and these things may be involved as well. But why machine learning? And to give it a slightly different tinge, something you and I know, but for our audience, Eric was chief algorithms officer at Stitch Fix, which is a very different title to Head of Data or chief data Officer, these types of things.

And you at literati were VP of Data science, but now your Chief Algorithms officer. So I'm wondering if that title can give us a hint as to what value machine learning can create for these types of organizations. Plus. Plus. 

Daragh: Yeah, [00:08:00] absolutely. For what it's worth, I'm very proud to carry the title of chief Algorithms officer forward from like I need to interact with Eric who had it.

I think he might have been the first out there. And one of the reasons why I was really attracted to it compared to Chief Data Officer is Chief Data Officer really implies that your goal and your role is to provide data to the business. And that's not really the way that I've conceived of my role.

That is a part of my role is to provide data. But really it's to provide data and other things built upon data that have direct business impact, that are really trying to drive some outcome where we don't exist to give data. We exist to drive some outcome that we all really care about as a business or that we really care about for our clients.

And so that's really our focus. Now, for what it's worth, I don't think every problem needs to be solved with machine learning. Machine learning's kind of a bazooka, and you don't need to go after every problem with the bazooka. Some things are just don't need to be solved with machine learning and if and when you can solve a problem with analytics or cleverly [00:09:00] deciding what exactly you need to count, I think deciding what is the right thing to count is probably also an incredibly valuable exercise for many companies.

But when you can solve problems that way, you absolutely should with analytics or bi or analysis. But sometimes machine learning I think is the right tool for the job. In particular, I think machine learning is really powerful when you're using it to generate a prediction and when you're then using that prediction to drive an action.

So just to ground it, for instance, at Literati, one of our key decisions is what five books should we put in a box and mail to each child? And there's a lot of ways you might try to solve that problem, but a bi solution might not help you choose the perfect five books for each kid. I think that's a kind of problem that machine learning is really well suited to is helping us predict how much is each child gonna like each book and then send it to them.

And then for instance, on the inventory side, [00:10:00] another really core question that we have is what book should we bring into inventory in the first place? Like we run book fairs all around the country and we're in elementary schools helping them raise money for fundraisers. And again, we have to figure out what books to send to every school.

We don't wanna just peanut butter smear all the same books across every school. 'cause every school is different. If you talk to librarians, none of them are gonna tell you, I have a normal, average school. And so it's not possible to figure out what is a good average book and send the good average book to the school.

You need to figure out what does this kind of school need, and to do that well, you need to consider how good is this book. But also lots of things about the school and it's not enough to do that in a low dimensional space. You need to do that in a really high dimensional space. You need to be able to understand in a really deep way, what is this book using?

Not just structured data, but all the unstructured data from its cover art and from its text description. And you need to marry, marry that up with each school and understand that school. Again, [00:11:00] not just in terms of its structured data, like how many students are enrolled there, but also lots of unstructured data about it.

Like the name of the school actually ends up being a good predictor of how certain kinds of books are going to. Or what sort of demand you're gonna see for different kinds of books in each school. And so when you have this kind of a problem where it's really high dimensional and lots of different people are gonna need different results and you need to make really good predictions to drive those actions, to me that's a place where machine learning is often the right tool.

But when you don't need machine learning, when you can get away with intelligent analysis or statistics or all of the other tools in our toolkit, then yeah, we should definitely be using those tools 

Hugo: instead. Totally. So what I'm hearing is when there's a need to tie data to a decision function and you're in a pretty high dimensional space, machine learning can help you.

Yeah, absolutely. And those types of things occur everywhere. They may not be highest priority for any given organization, [00:12:00] but what I'm hearing is that machine learning is something that a lot of people can leverage. That's my belief. It doesn't 

Daragh: it, you don't have to use it for every problem, but one of the things which I've loved is going into legacy fields.

Like fashion design and discovering how much inside of the fashion world you can again, inform with machine learning or with all kinds of other sort of analytic methods. And we've definitely found the same thing again in the book space, there's loads of really core decisions when you go and you look at it that people are making on a daily basis.

For instance, in the publishing space, one of the biggest decisions is do we green light this book? Someone is shopping around the book, which is gonna be the next big seller. And the publishing house that figures out this book is gonna be the next big book and brings it on board is gonna be the one that does really well.

And the funny thing is, lots of publishers are gonna pass on the next big book before it finds its home. So if you were a publisher, how you're one of the, you're picking up the right books at the right times. Mm. [00:13:00] A really key problem. 

Hugo: Absolutely. And I'm wondering in to that extent, is ML enough or do we actually want.

Some form of causal ML or causal inference roped into our our models as well? 

Daragh: That's a great question. I think ML is sometimes enough. Sometimes it's the right tool. In the toolkit, there are definitely situations where you really need to know why the model is making some kind of a prediction. I'm fortunate that in this particular case, we have the full sort of breadth of machine learning models, including black box models that we can bring to bear on a lot of our problems because sometimes we don't need a really strong understanding of why a model thinks the book's going to be successful.

Hmm. Sometimes it's enough that it can just look at the cover art and using a embedding of that cover art. Just say, Hey, I think this book is gonna sell really well, or it's not going to sell well. And in those kinds of situations, it's not as necessary for us to be able to interpret why. But there are certainly lots of domains where you need to understand why a system is making its decisions and how it's going to [00:14:00] impact people downstream of it.

So I think a big concern if you're making really high value decisions, who gets a loan? You really wanna be able to introspect those kinds of models and understand why they're making their decisions and if you feel good about the way that they're making their decisions. So it's a little bit of a, the right tool for the right problem.

Hugo: Absolutely. And I like how you've pointed in, in, in several directions around when these tools might or may not be useful. I, something you, you said to me last time we spoke it, it was really stuck with me. So several things have, but with respect to your work at LITERATI in particular, but also everything you did at Stitch Fix, you said there's a big difference between advising on decisions and having data feed into the decision function and being accountable for them.

So maybe you can speak to that with respect to your current role at literati. 

Daragh: Sure. Maybe, I guess it, it would help to even go back a little bit to Stitch Fix. So the systems that I designed at Stitch Fix, they were really developed to, as you say, advise and recommend actions. [00:15:00] But the final decisions about what we were gonna really do, those lived with my cross-functional partners.

And at the time I thought that made a lot of sense because, and I still think it makes a lot of sense because ultimately I think the people who have accountability for some decision really need to have power for it. I think you have really bad things happen when power and accountability get separated.

You don't wanna have accountability without power and you really don't wanna work with someone else who has all the power, but none of the accountability. So you really want those two to be tightly coupled, but portray the kind of the main point. I think it's relatively common for data science to be structured up in this kind of structured, in this kind of advising or recommending role at a lot of different companies.

So for instance, I imagine probably a lot of people who are listening to the podcast, they are in a role where they make recommendations or they advise a product manager, for instance, who's gonna ultimately own that decision about what gets on the tech roadmap and what order do things can prioritize.

So I think this is a really common [00:16:00] paradigm. But I think anyone who has worked in this kind of environment has also got to experience situations where their recommendations are getting rejected and really wondering about what's going on. And over time, that really started to become something that I fixated on.

I really want to understand why some of my recommendations weren't getting accepted, both from an analytic perspective and an organizational perspective. And this was a major motivator for me then to, to join literati because at Literati I got to lead our data science and data engineering functions, but also, as you mentioned, our inventory curation and procurement function.

So I got to really experience that power and accountability that comes with really owning the decision. And I feel like this really changed my mindset and my approach. For instance, when I was building systems to recommend an action, I did tended to think in terms of a single objective function. Now this objective function, it might have been some complex algebra function of [00:17:00] other primitive metrics.

But in the end, it was really like a single numeric value that basically denoted goodness. But as I shifted to really owning the decision, I found my mindset shift in particular. For instance, I started to think more in terms of trade-offs and think more in terms of multiple objectives, whose relative importance is dynamically changing along with changing business conditions.

As an example, when we're making stocking decisions about inventory, you're always trying to figure out, what should I do here? I would choose some set of actions to optimize revenue, some set of options to maximize margin, maybe a different set for capital efficiency. And you have to figure out how to balance those against one another and how to do that in relation to the business' needs for the next quarter.

And there's lots of other kinds of just trade-offs that you end up thinking through. Like a big one for me is often how do you want to balance short verse long-term outcomes? For instance, managing inventory is a lot like a [00:18:00] big multi-arm bandit where at any moment in time you can choose to explore for new, better product or you can get more of the best product you've found so far, and so you end up making these trade-offs.

What do I prefer? Would I rather optimize for the short term? Do I want to do an exploit trial or do I wanna optimize for the long term? Do I wanna go do an explore trial? And that's a kind of decision that I find myself increasingly thinking about as increasingly present when I'm really owning the decision and the outcome fully end to end as opposed to making a recommendation.

Hugo: Super cool. Another thing about your work at literati, you own both the models, algorithms and the operational decisions. So I'm just wondering how that has changed your perspective on building data and ML and AI products. 

Daragh: Yeah, I can think of at least kind of two ways that it's really changed my perspective.

First, I think many of us data scientists and machine learning engineers. We're really comfortable operating in environments [00:19:00] where we can rigorously ab test a new capability on some restricted part of the business and see if it works, and then decide if you want to roll it out at scale. And we're used to situations then where we can ab test an idea if it doesn't work.

If some feature doesn't look good, we can just choose to not roll it out. We can essentially do a, like a null op, but many operational decisions just don't have that form. I think as you get closer and closer to owning, the decisions really end to end. For instance, a lot of decisions, they're just, they're rare and they have very immediate impact on the full business.

You don't get the opportunity to test some small scale, and there isn't an option to not make them. This decision has to get made next week, regardless of whether you feel like you have the data to support it or not, or regardless of whether the data supports a strong conclusion, you have to make this decision.

To give you an example, at Literati, for instance, we need to choose a budget about twice a year. For how many books are we gonna buy for the next year? Because there's about a six to nine month [00:20:00] lead time on our inventory buying. So we need to make these big, huge, consequential budget decisions. Mm-hmm. And there isn't the option to not make them.

You gotta make them. And there isn't a good option to ab test the last six months of work that you've put into the forecasting algorithms that are gonna help you make these inventory decisions. You can't really ab test 'em. And then even, and the hardest situation is you're gonna make this big decision and then you're gonna wait six, nine months to just start getting any data about whether it was even the right decision in the first place.

So there's gonna be huge lag. So you have to figure out how to operate in that kind of an environment where you have this different kind of set of constraints than you operate under when you're not owning as many of the operational decisions. And then second, looking back, I now sometimes ungenerously labeled myself as a bit of a, an armchair quarterback.

When I was in the role of more advising and recommending, and by that in armchair quarterback, [00:21:00] they don't typically have to own the execution that follows after a decision. And as a result, it can be hard for them sometimes to understand or really fully appreciate all of the operational constraints that are influencing the decisions that are getting made.

And as a result, it could be hard for them to understand why are or are not my recommendations getting accepted. Maybe just as an example, my team was working on a project at one point where we really developed a super elegant system to optimize a, a set of decisions. And it was optimizing all these decisions with respect to one another.

But then we would put this set of recommendations out in the world and we observed that people would only accept some portion of them. And there were other portions where they would go off the reservation and do something different. And I remember at the time feeling frustrat. Because it felt like the value of the algorithm was in jointly optimizing all of these decisions together.

But I only really started to understand [00:22:00] why this was occurring when I really spent a substantial amount of time with the team who was executing against those decisions. And what I discovered there, for instance, was that the relationships that we had with vendors was forcing us to lock subsets of those decisions sequentially.

And each time you would lock a subset decision you with a vendor, you would also be in a realtime negotiation with that vendor for exactly what form some of those decisions would take. And so this problem, which I had conceived algorithmically as a big one-shot optimization problem, simply wasn't, that just wasn't the form it took.

It was actually a set of smaller sequential optimizations. And it wasn't until I got really close to. The decision process and the execution process that I really understood that and began to appreciate how and why people were making the final decisions in the way they were. And so it's been really fun sort of getting closer to the decision and the execution so that you can really understand [00:23:00] how and why people interact the way they do.

Hugo: Absolutely. And one way to think about it is just speaking with your quote unquote consumers, right? Whoever is actually utilizing whatever you are building and being very close to the process where they turn that into value for themselves and stakeholders. Yeah, absolutely. That dovetails really nicely into the next topic I wanted to talk about, which is, so for example, you've worked on tools to help merchandisers choose what to buy, and we could use this as a case study or something else as a case study.

I am just interested in how you think about building tools and products that humans work with and how in the tool building you think about. For the use case, balancing human judgment with machine predictions. How do you get humans domain experts and humans in the loop in the right way with machine learning systems?

Daragh: Yeah. Yeah, that's a great question. It's really challenging [00:24:00] and I think it speaks to the, the mindset that you need to have sometimes when you're developing these systems where you're jointly designing some technical machine learning model, but you're also designing a workflow that's embedded within a organizational and social context.

So you have to think all about all those things at the same time. And I think it could be really challenging, but I think the framework that I bring to at least is, I like to think of it in terms or I, the kind of way that I ground myself is in thinking about how human domain experts and machine learning systems, they often have very complimentary strengths and weaknesses.

So I think the art is in designing some workflow and some algorithm. That enable you to take advantage of each system's relative strength and avoid each system's relative to weak weakness. Maybe it would help to, again, ground this in an example. I'll admit, I love Google Maps, and I think Google Maps is a beautiful example of how you can build one of these workflows and where things can go really [00:25:00] wi, right?

So for example, in in Google Maps, if I sit down, I can use in a huge variety of different ways, but I only only sit down and I, I have some goal and maybe it's, I want to go to the airport. And I think selecting that goal is this very human process where we need a human to really decide what they want.

May maybe with AI coming online, they're gonna get better at this, but for now, I think that's really a human task. The next thing which happens in Google Maps obviously, is the machine learning system. It'll say, if you wanna go there, here's three different routes. This is the fastest route. Here's the other route, which helps you avoid highways.

And here's this third option if you want to use public transit so you don't have to leave your car at the airport. And I think that's the kind of task that machine learning systems really thrive at is doing things to mature about prediction and consuming the highest quantity or the very latest data in real time and considering very high dimensional spaces and coming up with a couple different good options.

So I think that's a task which is great for a computer, but then it [00:26:00] goes back to the human. The human now can go, I have these three options. Which route is better? One's best if I don't want to pay for parking one's best, if I wanna go as fast as possible one's best. If I wanna avoid the highway, computers aren't gonna really help me make those trade-offs as well as a human will.

So let's push it back to the human to decide how do they want to make those trade offs and how are they contextualized? 'cause I think humans are great at contextualizing decisions. And then after that it goes, they go, they log into the computer and they press start route and it shows some path.

Computers are great at telling you exactly where am I, way easier than a person finding it on the map. But you let the human actually drive the execution and drive down the street, and you let the human exception handle, if you know there's a car crash directly in front of you, let the human decide to go on a side street.

That'll be a task that'll be really challenging by comparison for, uh, ML system to do. And so that's the way that I think about things is how do you build a system like Google Maps, which really blends humans and machine learning systems in a [00:27:00] way which takes advantage of both systems and their strengths, but which also is honestly just actually a lot easier for the people who are developing the technical system to build.

Because frankly, it's a whole lot easier probably to build Google Maps than it was to build the first self-driving car. And nonetheless, the Google Maps has generated an enormous amount of value for society. Thank goodness it exists. And as I think just this really beautiful melding of humans and machines.

And so that's a pattern that I think about often in my work. Is, how do I take advantage of humans and machines in that kind of way with staged decision making processes that are designed for each to really extract what's best about each of them. 

Hugo: I love it. The example of Google Maps is so fantastic and funnily, when, back in the day when people asked me what data products are and teaching, working a lot on, on, on data products, I'd always give Google Maps as an example of one of the best data products out there.

And I think I got that from Hillary Mason actually, who was also a guest on [00:28:00] the podcast. She used to speak about how Google Maps was one of the most, because it's wonderful at how the abstraction layer, it's a map, right? Mm-hmm. Where you do things on it. So it actually conforms to a previous paradigm and then adds new functionality on top of it.

The other reason I like this example, and I'm interested in your thoughts on this, how, how data and machine learning products change human behavior and how we might need to iterate on the product with respect to that. So in the case of Google Maps, if you send all the traffic one way. Perhaps you get congestion and need to start suggesting other routes.

So I'm wondering your thoughts on how machine learning can change human behavior and how we need to shift our algorithms with respect to that, and if you've had cases like that at Stitch Fix literati. 

Daragh: Yeah, I think it's a great question and I suspect it's gonna be a question that we struggle with increasingly as a society.

I think we've definitely seen the rise of recommendation engines created, or people like to speculate about online echo chambers as we got our own hyper curated reality. It [00:29:00] certainly, I think it's gonna be the case that there's gonna be very complex relationships merging between these AI systems and humans, and it's gonna, I think, be very challenging to anticipate exactly what direction they go.

It's a thing which I think a lot about, increasingly with you, knowis, l LM based AI coming online and how that's gonna affect kids' education. I've got a, a daughter who it's fascinating to see the way that she interacts with some of these models just in ways that it never occurred to me that you might like.

We've all heard kids are gonna try to use it to cheat, but I saw her taking a photo of a old test and saying, Hey, write me more practiced test questions like this test beautiful. Never heard. To me, that might be a way that you would use it to study. I think we're gonna see a lot more surprising things emerge, surprising things about ways that people are interacting with these tools, they're gonna have lots of impacts on society.

But I'll admit, I, I'm skeptical of futurists who have a really good idea of [00:30:00] what that impact is gonna be because I think we are just in this very complex dynamical system where there's gonna be just this huge feedback loop between us and these systems, and it's gonna be hard to tell which direction they go.

Hugo: I couldn't agree more, and I half joke that it's more of a chaotic system it feels like, than a complex system. Yeah. Currently I am interested in reasoning through more about like the human machine. Collaboration, which A, as we know a lot of the time, really outperforms either one or the other, a alone.

One thing I'm interested in is that you've, you've had to redesign workflows to make human machine collaboration actually work, work. And I'm wondering if you could give an example of something that didn't work until you had an organizational change. 

Daragh: Sure. There's a quote, I think it was from Ford, or it's at least Apocryphally given to Henry Ford, which is, if I ask people what they wanted, they would've said faster horses.

And that's a pattern that I see a lot [00:31:00] when people try to bring data science or machine learning based kind of solutions to existing problems. If you go and you ask your users or you go and you ask somebody who's running an existing process, what do you want? They're gonna ask you for faster horses. And of course the real grand irony here is that building faster horses really hard, right?

Building a faster horse was probably harder and less impactful than building an engine. But in order for someone to decide to build the engine, they had to be aware of the problem needing transportation. And they also needed to be highly aware of this emerging technology, the engine, to see the opportunity to really marry them together and find a reframing or a new kind of class of solutions to this problem.

And so that's something which I think a lot about is how do we avoid that trap of trying to build faster horses? And it's definitely a thing which I've seen on on several projects where we set out trying to automate something which was previously a human task. [00:32:00] Like for instance, forecasting how much clearance damage and shrink inventory there was going to be.

Mm-hmm. There was already a sort of existing human process to build that kind of a forecast. And if you tried emulating that workflow and that process, it actually was really hard. And unfortunately it really didn't take advantage of any of the. Sort of unique capabilities of the emerging technologies like machine learning that were really possible.

So it's only when I think we step back, for instance, as I shifted between, it was fun. I got to see two versions of a lot of these problems, the version at Stitch Fix. And then we solved a lot of sort of parallel problems at literati. And just getting to see, hey, if you're designing sometimes unencumbered by some notions of exactly what a inventory forecast needs to look like, you can build different kinds of solutions to that inventory forecasting problem that actually solve the problem in a way which is both simpler and a lot easier to [00:33:00] ultimately implement.

And so that's the thing which I've thought about is how do you embed your work inside of, again, these kind of organizational and societal constructs? And I think it's a thing that happens at micro scales as you're building a and trying to deploy a new recommendation engine if it's internally facing to a back office.

But it's a also a great big topic that we see everywhere else. There's a book I Love Innovator's Dilemma, which touches on this a lot. And how ultimately sometimes big companies that have reified ways of working in processes can struggle to get outta them. And it's fascinating. I at least I, I at least find these things fascinating.

But it's fun to see how often a company's org structure and a company's processes will end up mirroring one another. Or even how often you can look at a product that's being made by a company and make pretty good guesses about what their organizational structure looks like. So I think these are very bound up in together with one another.

And so [00:34:00] the question is, if you wanna change a product in some ways, sometimes I think you need to ask yourself, can I fundamentally change this product without changing the org structure or the workflow, or sometimes like literally the org chart. 

Hugo: Absolutely. And we even have a quote unquote. Proverbial law around it.

Conway's law, right, which is the structure of a system, reflects the structure of the organization that built it. Yeah. And we see that all over the place. For example, and people listening probably know a lot about this or even implicated in it as a lot of us, a lot of us are, but think about social media companies or fees.

Actually my old friend, Robert Chang, who he worked on the growth team at Twitter early in the day. Back in the day, they were figuring out how to notify people of certain things. And when the email growth team was different to the notification growth team, a user will get pinged about the same thing across different channels.

And that wasn't ideal. And that affected well that, that actually was a direct result of their org chart at at that point. So there are [00:35:00] examples. Absolutely everywhere. And I love that reared it, reared its head here. Something else I know you think about a lot is how incentive structures, how motivation, how performance review cycles, all of these things shape the success of data science projects.

So I'm wondering if you could give us just some insight in how you think about this. 

Daragh: Sure. Yeah. I think ultimately it has an enormous impact. 'cause again, we're all, we're all embedded inside of this big human and social context. We're all employees at company and we all, I think, respond to the incentives that we're given as well as the organizational culture and what's valued by that culture and what's stories that a company tells about itself I think is revealing often about what it really values and can have a lot of impact on the ways that people build and create.

I think as a result of this, a big part of the success or failure of a project is often really critically dependent on how all those incentives are set up. And how each team [00:36:00] member's performance is really measured against that incentive. And I think that often is the place where the rubber really meets the roles on roles and responsibilities.

It's always great to make a RACI chart and to have it in a Word document someplace, but it's when someone's incentives, like whether it's hiring, firing, promotion style decisions get really impacted that you start really seeing people changing their behavior often. And so it's interesting to think about that.

Maybe it would be fun to go back to the Google Maps metaphor if you have that, this kind of a complex human computer, symbiotic system. And then let's imagine a person uses it to decide to go to the airport and then they arrive at the airport at the wrong time, they missed their flight. Now you start to ask the question of whose fault was it that they missed their flight?

Was it the human fault? Was it the computer's fault? Was the machine learning component's fault? And of course, I think that's probably a broken framing, but it's. Trap that I see often, and you can get caught in if you aren't really careful and if [00:37:00] you haven't, it clearly stated sort of roles and responsibilities for each component of the system as well as understand whether you've aligned and ible to them.

So maybe unpacking that a little bit more with Google Maps, for instance, you might say that the, the machine learning model behind the scene is responsible for suggesting a few good routes and telling you how long it's going to take you to actually travel one of those routes. If you accept one of those routes and if you execute it, it's the human's responsibility to choose one of those routes and do their best to execute that route that was given to them.

Modular, outstanding circumstances like a car crash happening right in front of them. And so then I think you have to think about, okay, if we wanted to really measure which of those two people, which of those components in the system had done well, then you need to start really having solid metrics. You need to say, for instance, every time we have Google Maps, write out a.

Uh, prediction of how long it's gonna take us to drive some route. We should record that. And when the person arrives, [00:38:00] we should say, did you actually take that amount of time? And now there's an interesting question. What happens if the person diverges from the route? Do they, do you still hold the machine learning model responsible?

Do you penalize it and say that it was wrong? Or do you only penalize and say, what's wrong if people make mild detours from the root? So I think these are all the questions that we need to think about and struggle with to really decide how do we design the incentives? How do we set up the conditions for one of these complex human computer machine systems to produce good outputs and good results?

And when something goes wrong, how do we actually determine what went wrong in the system? 

Hugo: That makes a lot of sense. And I do all of these incentives within organizations I think are super important. But zooming out, I, I suppose there are incentives that are career level that you can't control within an organization and one that still.

And I don't know, I've got no idea how this works internally at literati and how you think about this, or even if it's a concern, but [00:39:00] it's one that irks me to be honest. Dara, which is the career level incentive to build and not maintain products. Mm-hmm. Which maybe a relic in some ways of traditional software engineering where maintenance wasn't.

Maintaining a data powered product is the product in so many ways because it is something that evolves continuously as it interacts with the messiness of the real world. So I'm wondering how you think it about this challenge with maintenance being so important, but the career ladder being around building.

Daragh: Yeah. Lots of fun angles on that one. First off, I think it's, I think it's unfortunate when our career ladder again, forces us to do things that we don't actually really want people to do, or that those people themselves don't really wanna do. So I think we should try to avoid designing ladders that encourage people to build, but not maintain if we want them to be building and maintaining.

[00:40:00] Mm. So I think first the onus is on leaders to properly design rubrics and to properly reward people for doing high quality maintenance work. Because systems don't just stay online on their own. As we all know, code needs to be maintained, it needs to be updated. And so if we don't have somebody doing that work, it's gonna lead to lots of bad outcomes.

It's gonna cost the company a lot of value. So I think we need to design systems well for that, and we need to properly reward maintenance. I, I can't help resist also riffing in terms of also how does this come to life when we start trying to deploy algorithmic systems and have them get closer and closer to decisions.

One thing which I see a lot is even outside of the one company's rubric. For promotion, people have some notion of what they want their career to look like, and they have some notion that I need to learn some technology in order to advance in machine learning or in some other domain, uh, creative domain.

For [00:41:00] instance, at Literati you might say, I wanna be a buyer. That means I need to get better and better at anticipating what books are going to perform well and predicting the performance of new books. Mm. And what then happens, I think, often is when you start trying to automate the tasks that, or bring machine learning to bear on tasks that someone either loves doing or that someone views as being important for their career progression, you end up often having a lot of challenges.

And those are places where I've seen a lot of frustration or a lot of projects stall is when you start trying to do that. When you start trying to solve these problems that people view as being important to my career ladder or that they love themselves or even that just gives them a sense of value at work.

And so I think that's a, an interesting problem for us to solve is what do we do in those situations? And I think that's with ais coming online increasingly and with the emergence of auto ml, I see this with data scientists. Lots of data scientists really want to spend their day model building. [00:42:00] That's what they love.

And more than doing maintenance. But as auto MLS come online, you can see that there's resistance from some data scientists. 'cause they're like, that's the part of the job that I loved as compared to the making sure that my ETL is unbroken tonight. But figuring out how do these tools really get used? How do we still preserve career progression ladders?

How do we preserve individual sense of identity in the presence of these tools? I think is gonna be really critical. And I think it is possible. Like it's a, I think that, I think this is a story as old as time where every time there's a new technology, people get really worried that this technology's gonna replace me or it's gonna make me no longer valuable.

But it's really no, this technology, I think. Uh, technology has historically, typically reshaped roles and it's changed the way in which people work and it's changed the things that people do or how they do it, but it hasn't necessarily disappeared. It. An example of what this, which I love is I've heard that when the radio first came out, lots of people who [00:43:00] performed live music said, ah, this is the death of the musician, and it's the death of music as a profession.

And of course that's not the case. We still have people who perform music live, but there's also been an incredible creation of new industries and new jobs around music. And now we have a DJ and we have sound technicians and sound engineers. And so there's loads of new kinds of jobs that emerged, but it certainly is going to challenge and change old jobs.

And I think that's the thing that we need to figure out as more of these technologies come online is what is that path and how do we find that and help people see that instead of just make people feel like, oh, this machine learning model's coming to replace you. 

Hugo: Absolutely. And I think my understanding is I need to fact check myself, but when the piano first came out, organists, some organists were like, how dare you?

Like you press a single button and get a note. You don't need to pull all these things in the bellows. And that's my horrible impersonation of what an organist in at at that time would've been like. I totally agree. And actually I had Tim O'Reilly on the podcast a while ago, and I'll link to this episode in the show notes.

[00:44:00] It's an episode called The End of Programming as we know it, and it's, it was based on we, I had him on the podcast 'cause he wrote an essay about, about this and in particular. That there's a huge amount of EV evidence that this will actually increase the surface area of what's possible with software and create a lot more different, different type types of software.

And Tim uses the analogy, which holds in some places fall downs in others of the industrial revolution. And sure, there was a huge period of chaos transformation, but it was a skills shift as well that a decade afterwards you actually had a whole bunch more people being employed with a different set of skills.

And there was still art, artisanal textiles and that type of stuff with loom, et cetera. But a huge, a far greater amount of people building with industrial machinery. 

Daragh: Yeah. And yeah, that's a, it's a funny thing I see a lot because I work with a lot of people who are in, have a tech background as well as who don't have a tech background in people who have a tech background.

I think we're more comfortable with the idea that [00:45:00] this is my job right now, but in three years time, who knows what I'll be doing? I'll be using conceivably some entirely different skill set and some entirely different set of methods. I think the challenge is often when you're trying to bring new technology into a domain, which hasn't historically looked like that, where somebody joined some career expecting that's what their career was gonna look like for the next 30 years.

They were just gonna do that job, but maybe with larger scope at different companies for the next 30 years. And so trying to help people see that path through it and actually create that path, I think is really important. Totally. 

Hugo: And I would wonder, even if it's technical people, or even more specifically what our selection bias is, and data ML people who, 'cause that's what we've been doing for a decade plus, right?

Is like on, okay, now psychic learn, PY taught what, whatever. And I'm not just mentioning tools here as opposed to methodologies and all types of things, but we've constantly been shifting, adopting new tools and processes, right. 

Daragh: Yeah. Yeah. [00:46:00] Which I think it's, it's simultaneously, I think the thing which many of us love about it, but I think it's scary if you're not used to that kind of a domain.

That's not a way that a lot of people are used to working. So, 

Hugo: and it's also not, and I don't wanna get too big picture here, but it is my want occasionally that's not what the 20th century created in terms of good production line for us. To your point earlier, tailor us workers who are nine to five and get the job done, the company man in, in quotation marks not my form of gendering there.

Whereas now, and I've worked in startup land for God knows how long now, and we always say in early days of a startup, you want explorers. And in Australia would say Bush bashes, people would run into the bush and find the river, boil the water, set up a camp overnight, move on. Right? And then at a certain point, you wanna have settlers who will build.

A build an actual robust fire pit with a little camp next to it, build a medic, a medics tent, that type of stuff. Maybe have some storage, these types of things. The reason I'm framing it like, like this is, I think both are still important across [00:47:00] all forms of organizations, but I do wonder whether with the way AI LLMs, but generative AI more generally is proceeding now if there's almost, we've gone through a phase transition where the more explorers we have who aren't these like rigid, in a good sense, this is my job, this is the process.

The more explorers we have to experiment and the more people we have who have one or two days a week that actually explore the better off and more nimble our organizations will be. 

Daragh: Yeah. I think it's gonna be fascinating to see what really emerges, honestly. Yeah. I think it's gonna be, it'll be an interesting time.

I think I wouldn't discount though the, as someone who, I dunno, I sometimes think in terms of, there's people who love starting things and there's people who love finishing things. I'll admit I'm, I love starting things. That's my, my disposition. And I think that's one of the reasons why I love startups. I love taking things from zero to one, really refining and cleaning up a system.

That's not a thing which I take joy in, in the same way that I take joy in the moment of discovery of cre, or the moment of creation of something brand [00:48:00] new. But it's really definitely the case that if you don't have people around you who are finishers, you suffer for it. So it's one thing which I think is really important is for both individuals, but also for organizations to understand where they currently are in this sort of dynamic and make sure that they have complimentary people around them to compliment their weaknesses.

Because you don't want someone like me who's just creating a new thing every week, but who isn't gonna love stepping back and saying, okay, let's actually systematize some things here. At the same time, if you have only people who wanna build systems, but we don't yet really know. How to systematize or what we need to systematize, yet you're prematurely optimizing everything.

That's gonna be a problem too. So I think there's this important balance and how that balance is going to shift in a world with gen ai. Honestly, I don't know that I, I don't know that I could even speculate, but I think it's gonna be really fun to see what happens. 

Hugo: Yeah, it's a really exciting time. I, I'm interested, we've talked about incentive [00:49:00] systems and making sure people are accountable for and excited about the work they do.

I am wondering in terms of, and we've touched on organizational structure as well. I'm just wondering what the main models you've seen work for structuring data science teams within an org and how these, how this may change depending on whether you're doing BI analytics forecasting ML or LLM powered product work.

Daragh: Yeah, I've seen a few different models. I think they often begin with what is the. I'm gonna call them data person here. What is the data scientists sort of relationship to decision making and where is the genesis of the project? 

Hugo: Mm. 

Daragh: For instance, the classic scientist I mentioned, I'm recovering academic there you're relatively decoupled from the decision making process, but you get to generate your own ideas and that's really fun.

I think it's great if you are trying to push some field of knowledge and understanding forward, but it makes it [00:50:00] comparatively hard if you're trying to drive decision making. Like for instance, you see that with the risk of getting political with the global warming, where you have a lot of scientists who are screaming like, this is a real problem, we need to do something about it.

And they're free to do a lot of research on it, but it's really challenging for them then to couple a lot of their research to action in. I think the way that they would like, and this contrasts interestingly, I think with more of it, an analyst model. Where I think analysts have, a lot of companies are in a comparatively similar role, but the genesis for a question often is a business partner marketing comes to you and asks you, what is our client acquisition cost?

Split out by client segments? Can you figure out what client segments we have? And a question like that. And then you go off and you do a analysis and you present again your findings to some other decision maker. So I think that's the other kind of model analyst, which I see. And then you have a third model, which I think is getting a little closer to like machine learning now, which I like to think of is the [00:51:00] recommender model.

And the reprimand model is someone who is trying to understand the actual action. They're not providing insights, they're trying to align to an action and trying to say, here are the specific actions we should take. And the ones that I recommend and the ones that I don't recommend, and this is often ultimately based on a, some kind of a predicted algorithm under the hood.

Not always, but often it is. In this situation, I think it can be either the sort of business or the data scientist sees the opportunity to create this recommendation algorithm, but it's closer coupled to that decision in action, which I think is the key part to it. I think one other model, which you know we've talked about a lot is the sort of the Google Maps model where you have data science trying to understand and create a consideration set of options that can be then handed to a decision maker for their selection and their execution.

I think that's an interesting model because the data scientists are, again, they're getting closer to the decision process, [00:52:00] but they're leaving. I think it's an interesting space for humans to really do contextualization and trade-offs in a way, which I think models aren't always as great at. And then the final kind of project, which you see a lot of, I think of it as the self-driving car, is when you try to go in and fully automate a process, often you try to automate a existing process at the business.

And replace it now with a machine learning based process. 

Hugo: Hmm. 

Daragh: Um, where there, the nice thing is you're really tightly coupled to not just the decision but also, or not just the recommendation, but the final decision with the one caveat that there can be often challenging if you fall into that trap of trying to automate the system in the way that the humans would've performed the task if you try to build faster horses.

I find there's a lot of challenges there. I think each of these models, I've seen pros and cons to benefits and weaknesses of them, but I think the important thing probably is not for people to just say, Hey, here's [00:53:00] the better model, but instead to figure out which model is right for the next project that they're working on.

And or probably what portfolio of projects are right. Because I think for instance, it would be foolhardy to embark on a process of automating a task and trying to build a self-driving car, so to speak. If you haven't first spent a meaningful amount of time. Doing more like analysis style work. Make sure you really understand the domain and you can generate insights that are actually sensible from it.

So I think it's probably a portfolio of projects that really makes sense for any organization. 

Hugo: I love it. And funnily, this has come to mind several times in our conversation. I just haven't mentioned it when we're talking about counting earlier. It's Monica Tis Data Science or AI Hierarchy of Needs, which essentially, and I'll share it in the show notes, but it's essentially a new version.

A, a data AI centric version of Maslow's Hierarchy of Needs where you have collect at the bottom and then move store and explore, transform, and then aggregate label. And only then do you start doing learning and optimizing. Like [00:54:00] counting is what we start learning, start to learn. Doing. Start to learn to do early on, but it takes a long time.

There's the old joke, it's not even a joke. I was talking with someone about this earlier today that if you are joining a startup as their first data scientist, almost guaranteed you'll be the data engineer for the first 12 to 18 months. 

Daragh: Absolutely agreed. And I think just the handling all of the actual mechanical process of counting is really challenging.

Also, figuring out what to count. I think there's actually a lot of really deep thinking that can go into figuring out what you count and or how to do good metric design, ultimately incredibly valuable because if you can derive the right metrics and if you can get an organization to align its incentives to those metrics, I think you can drive a lot of really great outcomes.

Hugo: Absolutely. I'm hearing some people I've spoken with recently in my head say, can't you just get an agent to do that? And it's, it's shocking. But that that belies a lack of curiosity as as well, I think what a fascinating thing to get in the weeds and. Figure out yourself. We are gonna have to wrap up in, in [00:55:00] a minute sadly.

'cause I feel like we could chat for so long about, about these things. We recently had, um, du Shari on the podcast and he is a VP of AI Data Science and Foundation Engineering at Alto Pharmacy. And he, he was a Groupon before that and he gave a really interesting framing of. The progression of data in, in, in our space where a decade ago he framed data as for a lot of places, maybe not fang, but a lot of places being the bottleneck in, in, in a lot of organizations.

Whereas now it's in places where it's really working, it's really becoming the backbone of the decision function, which dovetails really nicely in, in, into our conversation. And he said something he's seeing and wants to see more and more is data becoming even more foundational and actually being the backbone for making bigger, irreversible decisions very well supported.

And the reason I, I frame it like this is because last time we spoke, you mentioned the shift from high frequency, low [00:56:00] stakes decisions to low frequency, high stakes decisions. So I'm just wondering how you think about this and how you build systems and processes to support low frequency, but seriously impactful decisions.

Daragh: Yeah, that's a great question. Ultimately, we talked a lot about machine learning as being really helpful in these situations where you're generating a lot of predictions, you're in a very rich space, or you've got very high degrees of freedom, but that's not every important decision and companies need to make these big calls.

It's, I think, really different the approaches and the mindsets that you bring to solving these kinds of problems. Just as one example, I find in the normal, like high frequency, low stake decision making environment, you tend to build one machine, one really strong machine learning model, back test it, ab test it, and then decide if you wanna roll it out.

But when you're instead making low frequency, very high stakes decisions, instead of building one really strong system, a approach, which I find myself increasingly [00:57:00] using, is more of like a converging operations approach. I want to come at a problem from multiple different directions using multiple different methodologies, which have baked into them fundamentally different kinds of assumptions.

And see if I can arrive at, uh, a similar decision using a variety of these different methods. And if and when converging operations point you in different directions and lead you or would point you towards different decisions, that's when I think you realize, okay, something's not quite right here. I need to dig in and understand what's more deeply, what's going on.

And so that's for instance, a strategy which I find myself using, which is a little different from the normal kind of approach that you'd use for developing and deploying a model. And there's other important kind of dimensions, which I think they differ on. Another is with these very large stakes decisions, I think often you want be able to understand what tradeoffs are really implicit to that decision process.

Because often you're making a decision which is going to, it's gonna trade off the [00:58:00] company's revenue versus margin, for instance. And you wanna really understand all of those tradeoffs and be able to model out and hopefully quantitatively and rigorously understand some of those tradeoffs. If you're making.

And be able to play with that slider and choose where you wanna be. And I guess maybe even coming back to the question that you raised on earlier on about, um, explainability in systems or interpretability of our models, I think here it's actually often really helpful to have models that are more interpretable.

And it's because often you'll find yourself in these kinds of situations having to negotiate decisions with other executives at the company. So for instance, uh, I mentioned earlier on having to choose a budget for the half year for what, how much we're gonna spend on inventory for books. And that's the kind of question which is invariably gonna lead to a meeting with myself and the CFO at literati where we're gonna be actually talking about, is this the right number?

Is this the right budget for us to have? And what are all of the different trade-offs and [00:59:00] implications for the company based on it? And to be able to explain how you arrived at the numbers that you're showing, I think is increasingly really. Helpful when you start having to justify or describe your rationale for some big ask.

Hugo: Makes a lot of sense. So as a final question, I think you have, you have built at literati and at Stitch Fix and been part of the building a culture where data leaders are considered really first class citizens who are accountable for the impact they have. This doesn't occur everywhere. So I'm wondering if there's any advice or heuristics or ways of thinking you have for data leaders who want to have more impact and be accountable for more of the impact that they have in their organizations.

Daragh: Yeah, I think it's a great question. First, I think it begins with a, a mindset and it's a, I think, which I like to tell to people who join my team. You have the permission and the responsibility to drive impact. I think a lot of people [01:00:00] either don't realize they have the permission or they don't realize that they have the responsibility to drive impact.

So that's the first, which I do, is make sure everyone knows they have permission and responsibility to drive it. And then to your question though. Okay, so now I wanna do this. How do I do it? I think there's a couple steps that are really helpful. First, figure out what's really important at the business.

Figure out what you really need to drive, what are the outcomes that are undeniably important? And it sounds almost goosh sometimes, but frankly, the easiest unit to denote impact in often at a company is dollars. So figure out how can I describe the impact that I'm having in dollars? And next figure out who is that important to and who that's important to.

Might depend on what kind of dollars your impact's going to be in. If it's dollars of revenue versus margin, different people might care. If it's dollars of LTV or dollars of client acquisition cost, then you know they're gonna wanna go talk to marketing leaders. So figuring out who cares about this [01:01:00] metric really moving, who is it who loses sleep when this dollar based metric changes?

That's the person who you wanna go and talk to. Next. You need to get confident that you know how to actually drive that outcome. You need to make sure you actually know how to move that metric. Fortunately this, I think the part of the job or the part of the stack that like most people in our profession are most comfortable with, is going and analyzing the business, trying to understand where are the big sensitivities in the business and how do I actually nudge them in different directions?

Hugo: Mm. 

Daragh: And then finally, now that you have all that information, you really need to go to the people who care the most about that metric and offer to drive that metric for them. And there's a book about this, which I love called Leading Executive Communication. And it gives a great framing for this kind of conversation.

And it sounds transactional, but I think it still works. And it's, if you give me this, I will give you that. And so if you go to someone, you say, Hey, if you give me the ability to AB test my idea, [01:02:00] I'll give you X million dollars of revenue. And if that person they care all day long, they lose sleep about revenue, then it's gonna sound awfully tempting to them to say, okay, I'm gonna, I'm gonna green light your AV test.

So that's really, I think, the way then to start breaking it. So tho those are the big steps in my head are, first I know that you have responsibility and freedom to drive impact. Second, figure out what impact you really want to drive. What does that really mean? Figure out who it's important to figure out how to drive it and then go and ask for help to actually drive it.

Hugo: Fantastic. Well, Dara, I just wanna thank you so much for such a wonderful conversation, and not only your time and expertise, but all the wonderful work you've done at Stitch Fix, and you're doing it literati and can't wait to see what happens next. Thank you so much. Really appreciate it, and it was a gigantic pleasure getting to chat today.

Thanks so much for listening to High Signal, brought to you by Delphina. If you enjoyed this episode, don't forget to sign up for our newsletter, follow us on YouTube and [01:03:00] share the podcast with your friends and colleagues like and subscribe on YouTube and give us five stars and a review on iTunes and Spotify.

This will help us bring you more of the conversations you love. All the links are in the show notes. We'll catch you next time.