douglas-hubbard-2020-07-01.mp3

Douglas Hubbard: [00:00:00] I guess that's the thing is everyone just went around assuming that certain things aren't measurable. And I said, I don't think that's true. I think I can show otherwise. I think I can show that everything's measurable. And I think I can show that the belief that something's immeasurable is based on a short list of misconceptions that we can correct. And so, yeah, I guess that's the story I would have people take away from this. As you know, if you've got an idea that maybe a lot of people would initially reject, but you've done your research and you know they're rejecting it for the wrong reasons, then maybe that's something you could build on.

Harpreet Sahota: [00:00:45] What's up everyone, welcome to another episode of The Artists and Data Science. Be sure to follow the show on Instagram @theartistsofdatascience and on Twitter at @ArtistsOfData. I'll be sharing awesome tips and wisdom on Data science as well as clips from the show. join the free open mastermind selection by going to bitly.com/artistsofdatascience. I'll keep you updated on biweekly OpenOffice hours I'll be hosting for the community. I'm your host Harpreet Sahota. Let's ride this beat out into another awesome episode. And don't forget to subscribe, rate, and review the show. 

Harpreet Sahota: [00:01:36] Our guest today is a Management Consulting, speaker, and author in decision sciences  He's the inventor of the Applied Information Economics method and he's an internationally recognized expert in the field of measuring intangibles, risks and value, especially in I.T. and he's a popular speaker at numerous conferences. His methods have been adopted in a number of industries, including insurance, pharmaceuticals, aerospace, R&D, cyber security and manufacturing, just to name a few. He's known for taking a critical view of several popular methods and standards in risk management and decision making in organizations. He also makes compelling arguments that extensive research methods such as risk matrices, the use of weighted scores and decision making and even expert intuition are inferior to certain quantitative methods. He's applied information economics method has been applied to dozens of large Fortune 500 I.T. investments, military logistics, venture capital, aerospace and environmental issues. In addition to the many article he's written for magazines like Information Week, CIO enterprise, and DBMS magazines, he is the author of books such as: How to Measure Anything, Finding the Value of Intangibles in Business, Failure of Risk Management, Why It's Broken, How to Fix It, Pulse the new science of Harnessing Internet buzz to track threats and opportunities, and his latest book, How to Measure Anything in Cyber security Risk. So please help me in welcoming our guest today, the founder of Hubbard Decision Research, Douglas W. Hubbard. Doug, thank you so much for taking time at your schedule to be here today. I really appreciate you coming on to the show.

Douglas Hubbard: [00:02:58] Yeah, thanks for having me.

Harpreet Sahota: [00:02:59] So talk to us how you first got interested in measuring the intangibles. What drew you to the field, a quantitative methodology?

Douglas Hubbard: [00:03:07] Yeah, I guess it really goes all the way back to when I first started in Management Consulting in the 1980s. I was with a firm called Coopers and Lybrand at the time. Now we know it partly as Price Waterhouse Coopers because Price Waterhouse bought Coopers and Lybrand, at one point after I left. And I was still kind of a quantitative analyst at that time, and I still am. But that's what my quantitative analysis, you know, charts really start to get, you know, exercised. So I was a young analyst and we kept running into situations when we were applying quantitative methods where a client would say that they couldn't measure something that we needed for a model. And, you know, at first I just took their word for it. I mean, how would I know? I mean, they tell me that you can't measure collaboration or you can't measure flexibility or something like this. I initially would take their word for it. But after a while, I saw that they would say that about things that I knew we had already measured in other cases. And then I started to doubt it was ever true. And in fact, I started keeping a list of these things. It was a list of all the kinds of objections I ran into and it just fell into a few categories of objections that were all based on misconceptions. And that was how I got interested. I saw that there was a lot of need for it in management consulting. There was a very soft side of management consulting. And back in the 80s, a lot of people even then were getting very interested in measuring things and the challenge was measuring stuff. I guess I had enough of a math and physics slash science background, you know, the few courses I took in my undergraduate, that I was a relatively quantitative guy among my team. I just had a few more classes than my colleagues, and that was how I got into that. But I really like my operations research courses, like quantitative analytics courses. So back then, it was more called management science or quantitative analysis, they didn't really talk about predictive analytics or something like that so much then. But, you know, those are similar concepts, they're almost different words for the same thing. So that's how I got started in all of this. I kept on running into situations where people were trying to measure something that they couldn't measure. And I saw solution for it. Or sometimes they would say that's something we needed in a decision model wasn't measurable at all and we would find a solution for it.

Harpreet Sahota: [00:05:14] So what were some notable projects that you worked on during the early part of your career that helped you shape your philosophy of being able to measure anything?

Douglas Hubbard: [00:05:22] Yeah, I suppose I can talk about this one now because it was so long ago with a different firm that no longer exists that I'm sure I'm not violating any NDA. So that when I was at Coopers and Lybrand, the first big project I was on was Mutual of Omaha in Omaha, Nebraska, and they were trying to figure out should they do their own printing or not. So big insurance company back then would have a huge printing operation as bigger, bigger than some dedicated printing operations. The whole floor of a building at Mutual Omaha was the printing operations floor. And these are proper presses like you would see in a big newspaper printer. Right. So these are big operations going on all the time. So all the policies are all paper, you know, all their sales brochures, all this kind of stuff, they just had people that did that all the time. And there was local print operators that say, why do you do your own printing? We don't sell insurance, so we're printing. We'll do printing, you can sell insurance. And apparently they had some connections at the board of directors level and so forth. So they brought in Coopers and Lybrand and which meant my director of Management Consulting and me to go in and analyze whether or not they should do that any differently. Among other things, they were trying to measure what I guess you would say community awareness or community collaboration, because they want to use local businesses instead of themselves. And I guess I kind of framed it differently for them. I said, well, there's what we can do, we'll work out how much it's costing you to do business externally. If you move more business externally versus internally, if there is a difference in cost, and then you can tell us whether or not your reputation and your community collaboration is worth that much money. And it was several million bucks a year that it would cost them to do more printing externally rather than internally. And so whatever it was worth to them, it wasn't worth that much. So they didn't outsource more of their printing because of that. They saw the numbers and they said, well, you know, it's nice to have these relationships with other local businesses and so forth, but their relationships aren't worth that much to us. And so is re-framing the kind of a question to them. And I kept on running into situations like that, I would run into situations where we were doing business for work for a municipality or a bank or other organizations that we're doing work for. And lots of things turned out to be very difficult to measure from the point of view of the client, like risk. I haven't heard that one, by the way, at an insurance company in the I.T. department, at the insurance company, the director, one of the directors there with direct report to CIO said, Doug, the problem with I.T. is it's risky and there's no way to measure risk. I said you work for an insurance company, what are you talking about? Of course, is a way to measure risk. I think I saw I'm having an epiphany at that moment. Like he only then realized it was kind of absurd for him to say that there's no way to measure risk when you worked for insurance company, of course, the risks. And do insurance companies measure risk for things where they don't have a lot of Data where there's new phenomenon? Do they ensure things like launches of satellites on Elon Musk rockets with only a few data points or no data points at the first launch, measuring the risk of new pandemics when the amount of travel in the world is changed or the concentration of people in urban areas has changed? How do they do that? How do risk analysts who work for nuclear power plants work out the probability of events that have never occurred anywhere but are possible? They know they're possibly, they know the risk is not zero, so what is the risk? Well, there are methods actually for each those, they're not terribly obvious, but they make sense when you walk through them. Right? So those are some things that got me involved in measuring difficult things early on.

Harpreet Sahota: [00:08:51] What's up, artists? Be sure to join the free, open, Mastermind slack community by going to bitly.com/artistsofdatascience. It's a great environment for us to talk all things Data science, to learn together, to grow together. And I'll also keep you updated on the open biweekly weekly office hours that I'll be hosting for our community. Check out the show on Instagram @theartistsofdatascience. Follow us on Twitter at @ArtistsOfData. Look forward to seeing you all there.

Harpreet Sahota: [00:09:20] And tell me about the genesis of the applied information economics type of methodology. How did you come up with that and what are its principles?

Douglas Hubbard: [00:09:29] Well, the whole idea was, as I was doing this research and, you know, adapting different methods to client problems. I would see that there was research in different fields about some techniques that people were using, including consultants in my own firm and not so much others. And in fact, some of the methods that people were using were very popular, actually didn't really have any evidence of improving estimates of decisions. And some of the most effective methods that had, you know, decades of and dozens of studies behind them were relatively obscure. And so I thought, well, if I took all the pieces that had the strongest evidence of improving estimates and decisions, you could make a complete methodology. So I had come across these different components that have been research by many other people for many years before I started adopting. And that a lot of the more recent popular reads along these lines just reinforce what we discovered earlier. There's more of the same. There's a lot more decision psychology, popular reading material out there now. And a lot of that stuff goes back many decades, goes back to the 70s and 50s and 40s even. So I was looking for what if we could put together a methodology where every component in the method showed a measurable improvement on estimates of decisions and the measurable improvement would be published in peer reviewed journals, so not just testimonials and anecdotes. Because as a consultant, I was coming across a lot of methodologies where somebody said this is a best practice or they would have a couple of clients say we really like this methodology. Well, that's obviously not that doesn't demonstrate that estimates are better than anybody can find a couple of positive testimonials for virtually anything. Right? So I started to be very suspicious of a lot of methodologies that consultants are kind of pushing. And I just ask, why do we think this stuff works? And I'd find out that some very popular methods, as I said, don't work at all and very obscure methods actually had a lot of evidence and I need to put them together into a method. Now, I call it Applied Information Economics, because central to the whole methodology is some work from Decision science where you compute the economic value measurements. But I was doing it more systematically on large decision models where a lot of the examples you'd see in decision science and decision analysis or game theory problems are pretty isolated simple techniques. So I came up with a few different methods for estimating the value of information on each uncertain variable in a large decision model. And when we started doing that, we started finding early on that the things we really needed to measure were not what the client would have measured otherwise. That's why I was first identifying something I later called the measurement inversion and that's why I came up with the name Applied Information Economics, because it focuses on measuring things that have a high value to improving decisions and that we can demonstrate that because there's no other research. 

Harpreet Sahota: [00:12:14] That is really interesting. It sounds like you're looking at research from a wide variety of different domains, wide variety, different fields, and kind of taking that problem statement and applying it to whatever new one you had in front of you and kind of viewing it from a different angle. Do I understand that correctly?

Douglas Hubbard: [00:12:28] There were experts in different parts of the problem, but nobody was pulling them altogether. I would find experts in decision psychology and people that were focusing on that component, just the psychology of decisions. There were people who were experts in optimization problems, operations research and simulations, but wouldn't apply the methods from the first thing. Even though a lot of the estimates that went subject matter experts. So they weren't leveraging the research on how you measure the performance and how you use subjective estimates from experts. And then there would be people who would be experts in empirical measurement methods. You know, like how can I make an inference about a larger, mostly invisible population from a random sample or controlled experiment? And that would be different than the people doing operations research or optimization problems or simulations and the people doing decision psychology. In fact, when it came to decision psychology or optimization methods, people tend to specialize further within those. It's kind of like you've heard the expression, if your only tool is a hammer, every problem is a nail. Well, that's what they were doing. And I felt like we needed a complete decision analysis solution that borrowed techniques from each of these fields. Only the methods that showed a measurable improvement and not even everything in operations research had evidence showing a measurable improvement, even though is quantitative. Certainly there's areas of finance that don't show a measurable improvement and say portfolio selection, even though they're very quantitative. And also there's some very soft methods that people become sort of expert in. And there's lots of methods that show no measurable improvement at all but they're very popular, they're widely adopted. And so that's why I decided to come up with that methodology. I called it Applied Information Economics because I built on one particular feature, things I was finding, which is that the high value information in a decision is not typically what people would have measure. So that was a more unique finding that I've come across that wasn't part of previous literature. And so that was the part I kind of focused on for the name Applied Information Economics. But everything else about it is all thoroughly research and shows a measurable improvement.

Harpreet Sahota: [00:14:32] So now that we're in a time where Data science and machine learning is ubiquitous in every aspect of technology and the work that people are doing, how do you see Data scientists benefiting from using the methodologies of applied information economics?

Douglas Hubbard: [00:14:47] Yeah. Well, first off, it's great when you have a bunch of data. You should definitely leverage stuff that's as free as a query on an existing database. Why not? In fact, it's so inexpensive to query those things. You can do it in a purely exploratory fashion where you don't necessarily have to know what you're looking for. Right? Because it's so easy to do that. But sometimes it becomes a crutch. Because when they don't have all the data, they say, I don't have enough data. Well, the scientific revolution wouldn't have started if they needed all the data to start doing science. Right? You make inferences from limited observations. And in data science, By the way I would like to ask the Data scientists what exactly is the other kind of science you're talking about besides data science? Isn't all science data science really? Well, granted, the term now has come to mean a collection of database oriented, maybe machine learning and stuff, as well as the empirical methods that go around. So all science is data science, it's all empirical. So I guess what happens a lot is that they fall back on that as their only method. It's that no, if your only tool is a hammer, problem again. So if that's your only method, then you feel stumped. If you feel like you don't already have the data in a nicely organized database of some sort or at least something that you could scrape together into a database, it might not occur to a data scientist to conduct original experiments or new random samples or other kinds of indirect inferences or for that matter, even secondary research. Now, somebody is probably measuring something like that before. So the first thing I do is I look at similar research. Right? So if you think that knowledge that you have to make inputs strong is the stuff in your database that's very limited no matter how big your database, that's very limited. In fact, when we ran information values, when we would compute the value of information for each uncertain variable and indecision model, we would tend to find that the high information value variables are things you had never measured before. Well, that makes sense. They're more uncertain because you've never measured them before, because they're more uncertain, depend on where they're using the model. They might have a big impact on the outcome of the models. So that information value stop is the stuff that you don't have in databases, typically because everything you already had a database that you could query that you'd end up with a very narrow range, if not an exact number for all the other variables. So the variables that had lots of uncertainty are the things you don't have in the database.

Harpreet Sahota: [00:17:04] So where do you see the field of quantitative methodology headed in the next two to five years? And how do you see applied information economics being used in that vision of the future?

Douglas Hubbard: [00:17:15] Yeah, I think in the next two to five years, first off, I would like to see I don't know if this is a forecast as much as a wish list, but I would like to see them start tracking measure performance of methods. So that should be the clincher, it's which methods measurably outperform others. Now, there's some parts of it better way ahead on this game, like the Kaggle competition where people compete on algorithms. I think that's all about picking the winners. And a lot of these cases you can show that the winners are just winners by luck. Because they're better by enough that their performance difference can't be explained as a random fluke. So that's important on the Kaggle competition. It's not always true but that's the case. Some people are just the champion coin flipper. Right? So you have 10 people in a room and start flipping coins while somebody is going to get the most heads in a row. That doesn't mean they're really good at flipping, but so sometimes you do that. But there's statistical tests for that. You can still work out separately from that. It's the best person better by enough that it would be extremely unlikely for random chance to explain their performance difference. So I think that's one thing. Now, I think I do see a trend more toward Bayesian inference. So that's kind of the art of the Data science and predictive analytics and things like this for a while. So that's good. I think that's the move in the right direction. But I think the other big thing is they should really be about decision models because data science what it for I mean, there's only three reasons why information has any value at all. I've described this in the first book. Now, these are the three I could think of. I've never heard of anything at all outside of it. But I focused the entire book on information that reduces uncertainty about variables used in decisions. So you're trying to improve the chance that you're going to choose a strategically superior alternative. Then something else you could have done because you have less uncertainty on some variables. So that's one thing where information has value. That's the one that the book tends to focus on. There's two other areas that I mentioned as tangentially. Maybe you gather information because it has value itself as marketable product, you know, so you sell it to other people. Then it's more like a commodity pricing or, you know, your pricing product is what you are. So there's still a value you can compute for that. The other one is just pure entertainment, intellectual curiosity, what would you pay for that? Well, people pay for documentaries now, they might be interested, they pay for scientific research often as a country or a community, as taxpayers, we pay for it. And the value that might be just purely intellectual curiosity, there's nothing wrong with that, that's the same reason you buy art that hangs on a wall or listen to and buy music or something like that. So we derive direct pleasure from that. That's sort of a willingness to pay kind of argument, it's worth what you would pay for it in that case. But most of the reasons that people want to measure things in businesses or governments are for supporting decisions. And so it's all about uncertainty reduction on decisions and that's where a Data scientist should focus. They're really dealing with just part of the problem, just by the name of the method. But Data science is one side of it. You're trying to make - and even saying big data. All right. So you have a lot of data. You want the right data, you want the data that's high value for particular decisions. But it really needs somebody. Now, it's OK that they specialize in that part of that. But there should be somebody in the organization that takes that input to a decision model, because often what I'll see is, I'll see people come up with a dashboard. We've seen companies start with dashboards. Right? And they're fed by some big data and the hope is, it's just a hope that the executives looking at all these charts on a big dashboard will somehow have the epiphany that will guide them to the right decision because they'll just strike a chord somehow. Well, you know, hope is not a strategy. Those charts on a dashboard, presumably they're there because they would inform decisions. So I tell people; you should work out in advance. What decisions do you think would be informed by this? Are those charts and dials there because you computed an information value and you have the high information value variables. What you should have, instead of those direct reads on various Data are trigger points, thresholds where when you reach some combination of particular bits of information that you're showing on a dashboard, some combination of those means that you should execute Plan X, don't leave that to your intuition. You've heard the expression measured with micrometer cut with an axe. So they're getting all this great data. And then it's supposed to be filtered through my human brain. And I'm supposed to optimize decision models by looking at my whole dashboard. That's not how we work at all. I think the dashboard makes us feel better about our decisions. Does it actually improve performance? Well, it seems unlikely, given the haphazard way that people choose things to go on the dashboard in a haphazard way that they use them. So what you need are decision models. Be ready to use decision models as part of a tool part of your brain. We're kind of Cyborgs, right? We should be. There's things that - our algorithms are good at, we still control the algorithms. We can directly use those as tools. I don't try to win a foot race against a car. The car is a tool. I don't feel threatened by the car. Right? I don't feel threatened by a power drill because it can do things I can't do. It's a tool I use together with the rest of my organic self. So let's be more cyborg on the problem solving side. We don't have to feel threatened by that stuff, but yeah. Don't throw your Data up against a wall on a dashboard and hope you're going to have the cognitive spark that's gonna cause you to - You know, I suppose it's possible. I mean, it'll happen once a while. But I would rather if I were betting millions or billions of dollars like or human lives or like a lot of these decisions are, I think you should do the math.

Harpreet Sahota: [00:22:30] So can you mark the difference between a decision model and, let's say, a predictive model or machine learning model for us?

Douglas Hubbard: [00:22:37] Yeah, well, predictive model is useful to forecast sales. Right? But what's the decision, the spokesman form, if you could forecast sales better, does that mean you build up more or less inventory in advance for some products, not others? Good. Now you're talking about something where there's a cost of being wrong and a chance of being wrong. So I'm highly uncertain about my sales. I might have too much inventory or too little or my factory might have too high capacity or too little or might have too much raw material or too little raw material. Those are all possibilities. Right? I was talking to someone about measuring things like collaboration and teamwork, I said, why do you want to measure collaborate, they sound like great things to measure. So why do you want to measure them. They will say well it's important. I go, yes, it's important. So why do you want to measure it? You can have collaboration, teamwork and not measure it. Yeah. We want to know who's collaborating well. Great. Why? You kind of have to keep asking the same question. Until they just out of fatigue, maybe, give in. And finally define the decision. You're modeling team work because you're trying to assign the right team leaders or construct teams that can get R&D projects done faster. And if you construct the wrong teams R&D doesn't get done this fast or they fail, is that what you're saying? Right? Now, we have a choice, a dilemma where there's a cost of being wrong. And chance of being wrong. Now we've identified a decision. So the decision part means there's multiple things you could do. You're uncertain about them and there's a cost of being wrong. If you choose a sub-optimal one. So, yes, it's great to predict sales and predict teamwork and predict collaboration and customer satisfaction. But why? That's where you connected to a decision. So that's where you say if the sales are this much, I should actually expand the factory. And why expand the factory if it turns out to be a mistake, right? And I don't want to fail to expand the factory if I ended up giving up a lot of sales because I couldn't meet demand. So now I've got a decision and there's a cost of being wrong about it. Either way, whether you reject the idea or accept the idea, there's still cost to be bought. So that's the connection we need to make. If not the Data scientists themselves, it's OK that they specialize. Right? That's OK. But somebody in your organization has to take it from there to the decisions. It's important. And that's another quantitative analytics problem. It might be the same person. I think it's convenient in many cases to have the same person built on the same skills, so one Data skills. But maybe there's areas of specialty there that people can go into. What you don't want to do is have Data scientists and no decision models, that's a waste. You can have lots of Data, lots of dashboards and what do you do with it? You don't know.

Harpreet Sahota: [00:25:04] So I'd like to start jumping into book, How to measure anything. I really, really enjoyed that book. And there's so much good stuff you have in there. And I like how you open the book. Well fairly close to the beginning. Talking about Fermi problems and Fermi decomposition. Could you talk to us a bit about those Fermi problems, the Fermi decomposition, and how that line of thinking can help us get a better grasp on how to measure and quantify intangible things?

Douglas Hubbard: [00:25:28] Well, it was a way for me to illustrate a particular approach to solving problems. That was demonstrated very well by the physicist Enrico Fermi, a Nobel Prize winning physicist, back in the early part of the 20th century. And he would try to stump his classroom. And he taught at the University of Chicago to try to stump his classroom with questions like estimate for me how many piano tuners in the city of Chicago. And these are all engineering and science students. I suppose they're pretty smart people. They're going to class by Enrico Fermi in University of Chicago and they're engineering science major. So, yeah, they're pretty smart, but they feel stumped by that question. Their first response is, how could I possibly know that? I don't know how many piano tuners in the city of Chicago. Well, Enrico Fermi said, let's figure out what we do know and see what we can infer from that. How big is the city? What's the population? Well, they estimate the population. How many people are in the average household? So let's estimate the number of households in the population. What percentage of households have pianos now? I don't know what the room is, representative or not, but maybe he would sample people in the room who grew up in a household with a piano. Right? And they would say, OK, about this percentage of them, great. So there's about this many range of pianos in the city. How many pianos can a piano tuner tune in a day? Given travel time between pianos and how frequently do you have to tune pianos or you look at it from what do you think piano tuning cost and how many would they have to tune to make a living at it? Right. So they would look at it from a couple of different angles. They'll end up with a range like thirty to ninety five. And then they'd look up the actual number. Back then they had something called Yellow Pages. And before that they had something called a guild list, which maybe they still do, but it's online line or whatever. They would find out the answer was you know Forty five. Well, actually the number was in the range they'd come up with and the students would start to figure out, OK, I knew something actually, it's really easy to get lost in all the stuff you don't know, it's really easy to get lost in complexity and exceptions and all this other kind of stuff. And I run to this routinely. People where we talk, we talk about measuring something and people start thinking of all the exceptions that might come up. But let's figure out what we do know and what we can infer from that. So these are called Fermi problems. When Enrico Fermi would come up with these fanciful estimates for his class, they would try to work on that. And the Fermi solution is that approach. Now, that's just another name for maybe something that accountants had done for a long time or engineers. When you estimate how much material you need for a bridge, you think of components. Right? And you had them up. Right? Or you trying to think about project cost estimate for I.T. or software development. You think of various tasks and different people are assigned for very much time per task. And, you know, there's a labor cost per hour. Well, your estimates, this has been research might be terribly obvious, but somebody did confirm that your estimates are better. When you do that, you actually get better at estimating, especially when things are highly uncertain. Like when these engineering science majors, Enrico Fermi students, when they were given that particular problem, the number of piano tutors in the city of Chicago, they had lots of uncertainty, they felt like they had no idea whatsoever. And Enrico Fermi brought them through this process to think about what you can infer from what you do know. So really, any time you just a cost benefit analysis that you might see on a spreadsheet, technically that's a type of Fermi decomposition, even though people can do it that way before Fermi was doing that. But that was just a thought process. Right? So if somebody says, what's your net present value on this new project or initiative, they might their first response like, I have no idea, there's no way I could know that. Well, you know something about the cost of it. You have some anticipated benefits. Are there ways we can estimate those benefits. So just like with any complicated problem in science or engineering or anything else, you break it down in pieces, you solve the pieces, so that's all there. 

Harpreet Sahota: [00:28:56] Instead of being stunned into inaction, right?

Douglas Hubbard: [00:28:59] Yeah, Yeah. What do you do next? I don't know which way to look like. I mean, people had really hard measurement. You know, how many baby shoes are sold the United States every year. Well, we have some idea about how many babies are born. Right? Or what would be a harder one. How much time do people spend sitting on average United States? Could probably start with a few things. Right. Or how much time do teams spend arguing?

Harpreet Sahota: [00:29:20] There's an excellent book I picked up a couple years ago called Guesstimation. I'm not if you're familiar with that, but it's just filled with, like, problems like that. And it's such a fun book, recommend it for...

Douglas Hubbard: [00:29:30] You know, I thought I might have referenced that one somewhere in one of my books, actually, because I know the book. I thought I might have it somewhere, but I don't think I mentioned the first one. But yes, just exactly like this. That's what you're doing with Guesstimation. Stop dwelling on all the stuff you don't know. Of course, you don't know all sorts of things. So what do you know and what can you infer from it? Right? I think it's fascinating examples. And that's why I tried to do in the book quite a lot in the first book - How to Measure Anything. And it talked about what could be inferred from all this. The Data you could get you have more Data than you think. And you probably need less than you think actually if you think about. So I call those measurement maxims.

Harpreet Sahota: [00:30:12] Are you an aspiring Data scientist struggling to break into the field or then check out dsdj.co/artists to reserve your spot for a free informational webinar on how you can break into the field. It's going to be filled with amazing tips that are specifically designed to help you land your first job. Check it out; dsdj.co/artists.

Harpreet Sahota: [00:30:37] You also talk about in your book. You have three reasons why people think something can't be measured. And that's the concept, the object and the method. Can you explain these to our audience and how you think understanding these are particularly important in this age of big data and the construction of machine learning systems?

Douglas Hubbard: [00:30:54] Yeah. These are the three categories I identify a long time ago of the three reasons why anybody ever thought something was immeasurable. And they're all three based on misconceptions. And I suppose it's possible that there's something that falls outside of these three. But I haven't found any yet in over 20 years. Every time that somebody thought something was a measurable, it was a problem of one of these three categories. So concept has to do with what the definition of measurement itself means. What's the concept? The measurement. Sometimes people think things aren't measurable because they might believe that measurement means that exact point. It actually hasn't meant that since the early 20th century at least. Measurement means that de facto use of the term in the empirical sciences is observations that reduce uncertainty expressed as a quantity. Right? So it's a quantitatively expressed uncertainty reduction based on observation. So you had some prior state of uncertainty, you made some observations, do some math often very trivial math and it's less uncertainty than you had before. So that's the way that it's used in the empirical sciences, whether they say explicitly that or not. But it's also the most relevant use of the term for decision makers. It's about uncertainty reductions. About improving your bets. Making better bets on uncertain decisions. You rarely get to be certain in a business or government decision, you have to reduce uncertainty up to a point where it's economically infeasible that it reduces any further. So that's what the concept of measurement is. And that means you have a prior state of uncertainty and almost anything that you do to reduce your uncertainty could be valuable by itself. Right? So even marginal reductions in uncertainty can be worth a lot of money. You understand the concept of card counting at the blackjack table. It's a very marginal uncertainty reduction, but it's enough that the casino doesn't like you doing it. So think of this, suppose a large bank could reduce its error of loan decisions by one percent. So one percent fewer bad loans are given out and one percent fewer good loans rejected. People that would have been a good risk or a good bet you rejected them. So what if you reduce those by one percent? What's that worth to the bank? Well, that's not really high resolution model by itself and certainly not a great predictor of outcomes. You've only slightly changed during uncertainty on each individual borrower, but could be easily worth millions of dollars a year, much more than that actually for a large bank. So, yeah, even marginal uncertainty reduction can be worth a lot of money. So that's the thing - concept to measurement. The second one, the object of measurement. Also, I think that one might be the bigger reason why a lot of things seem immeasurable is the object of measurement. The thing that you're trying to measure is not well defined or understood. So when people say, "I want to measure innovation" or "I want a measure of flexibility" or "strategic alignment" or "customer satisfaction", what do they mean? That they figured out what they see when they see more of it. So whenever I ask, I think of a really difficult measurement, a hard one. I mean, an impossible one. What do you think? Can you think of one that you can give me? That would be or one that I already listed. You think sounds really difficult to measure or what do you think?

Harpreet Sahota: [00:33:50] Oh, man, how about the number of diapers that are being used in my city every single day?

Douglas Hubbard: [00:33:56] Oh yeah. Actually, that at least started out. That's not so much of an object, a measurement problem, because you have a well-defined unit already. So diapers used per day in City X. That's well defined. But when someone says I want to measure collaboration, they don't even have a unit of measure in mind. Right? They don't know what strategic flexibility means. Right? So I asked them, what do you see when you see more of it? Have you seen some people who are better leaders than other people?

Harpreet Sahota: [00:34:24] Yes. Yeah, most definitely.

Douglas Hubbard: [00:34:27] What did you see when you saw more leadership?

Harpreet Sahota: [00:34:30] I saw better communication, I saw better engagement from the team and just overall kind of commitment from the employee to that leader.

Douglas Hubbard: [00:34:39] Ok. So even communication and engagement, we might have to decompose those little bit further. So engagement. What did you see when you saw more engagement? What did that mean?

Harpreet Sahota: [00:34:48] People were showing up to work a little bit earlier and you stay a bit later, taking less water breaks, overall looked like people were focusing more to get their tasks done.

Douglas Hubbard: [00:34:59] And we can keep going on that one for a while. But often what you'll find out is that when something sounds really difficult to measure, it's actually an ambiguous umbrella term that. Means a bunch of different kinds of observations. You see what I mean? Yeah. So leadership sounds difficult to measure because it's actually a whole bunch of individual observations. You just haven't put them together in your head and identified them explicitly so you could talk about leadership. You kind of have this vague idea of leadership. But when you think about what you actually saw when you saw leadership, it was a list of specific observations. Right? Otherwise, you had no reason to believe it ever varied at all. In fact, if that were the case, you might not even have thought of leadership as something to measure. When somebody asks about measuring anything, there's generally an implication that they've seen it, more of it in some cases than others. That always underline any measurement problem, right? They see more of it in some cases than others. Or they can imagine more of it in some cases than others. Well, what are you imagining, if you even haven't seen it yourself, what would you imagine? And then I ask, why do you care? So why would you measure leadership?

Harpreet Sahota: [00:36:03] Well, we'd like to see better employee retention, we'd like to see employees obviously stick around longer and be more productive, produce more. Keep saying the same thing. But reasons like that. Yeah, definitely. You know, employee retention, higher employee morale.

Douglas Hubbard: [00:36:17] Right. Okay. So, and morale, we can dive into that further too what you see when you see more morale. I think you already kind of listed some of them. Right? So we always want to go back to these observable things. We define things in terms of their observable consequences. And when you start thinking of it that way, I started imagine, OK. So the decision you might want to inform there with these observations is who should be the leader? Like who would you promote into a particular project lead position or something? Right. So what we're doing is you're saying I have these observations about this person and if I put person A there instead of person B, I forecast outcomes more, preferably for A than I would for B, does that make sense?

Harpreet Sahota: [00:36:56] Yeah.

Douglas Hubbard: [00:36:56] So you're basically saying I've got a list of 12 things that previously you kind of loosely put under the umbrella of leadership. But when you thought about it, there's really these 12 things. You know, how much time the individual spent communicating with the team? How early they show up for work. Right? And you're saying if I put those into a model, they predict something about the future success of projects. So then what happens is; you almost end up replacing the original term. It was too vague. Right? You can still use the original term as a sort of an umbrella label for all of the things you collected under it. But then it became these specific observable things that you relate to, a forecast of something that you're using to inform a decision. You have these observations about an individual, those observations reduce your uncertainty about forecasting outcomes of projects. And based on that, you can decide who to put in charge of that project. That would be one way, a lot of people come up with different lists, and that's fine. But that's part of the problem too, is people say leadership may mean different things because they know no one ever sat down and defined it. But as soon as you figure out what you see, when you see more of it and why you care, you're pretty close to measuring already. The rest is some trivial math.

Harpreet Sahota: [00:38:04] Just like decomposing. Fermi decomposition in a sense. Breaking it down into smaller, observable...

Douglas Hubbard: [00:38:08] And I mean love, art, anything you want to say. It only seems immeasurable because you haven't quite figured out what you mean when you say it.

Harpreet Sahota: [00:38:16] So what would you say are some of the most common misconception is that you've seen people hold about statistics?

Douglas Hubbard: [00:38:23] Yeah. Well, that kind of gets to the third thing that we talked about. I mentioned concept, object and the last one is method. So the methods of measurement are the third category of misconceptions that people have. So, for example, I often hear people say we don't have enough data to measure that. If you ever heard that. 

Harpreet Sahota: [00:38:39] Quite often.

Douglas Hubbard: [00:38:40] Yeah, sure. Well, that's actually a very specific mathematical claim when you think about it. Are they saying that for a given amount of data that they have, that they already computed the uncertainty reduction they could get from it and it didn't actually reduce uncertainty when they did the math? Or are they saying that the additional effort to gather some data, or collect some data and do this analysis wouldn't be justified by its information value? They computed the information, or are they saying that they already did an audit of all of the data that might even indirectly inform this measurement and none of it, through any kind of Bayes indirect Bayesian analysis at all, could possibly form this decision? No, they're waiting every time you ever heard somebody say that. We don't have enough data there. I guarantee you they're winging it. It would be very rare that someone actually did the math to support that claim. Have you ever heard someone say that's not a statistically significant sample size?

Harpreet Sahota: [00:39:33] Yes.

Douglas Hubbard: [00:39:33] That's wrong. That's it, there's no such thing as a "statistically significant sample size". What I mean by that is there is a thing called statistical significance. That's something. And of course, sample size is something. But there is no such thing as a "statistically significant sample size". That's a misunderstanding of the concept. There's no universal number where if you're one short of that, you can't make an inference. That's how the math works at all. The way the math actually works is every observation slightly adjust your probability distribution of possible outcomes you're trying to estimate. Like maybe you're trying to estimate the average of some population or what percentage of your customers would buy X if you did this advertisement versus that one, etc. Right? So every single observation you make, whether it's a survey or a test market or something like this, it could be a very small sample would reduce your uncertainty about this thing you're trying to estimate, like the proportion of customers who would buy X, you maybe had a wide range initially and each observation slightly adjust that range a little bit on the narrower side each time, okay, on average, it gets narrower, narrower, and it gets narrow much more quickly than you think, actually. So there is no universal threshold where you only have twenty nine, you can't make an inference, not that it would work that way. In fact, there's a lot more that goes into competing statistical significance than just sample size. You know, we have to take the values that you sample, not just the number of them, but the actual values in the sample and compute something called the P value, which is the probability that you would have observed those results or something more extreme if the null hypothesis were true. And you didn't even start at all hypothesis. So you can't compute the P value certain just from knowing there's 28 samples. And you'd have to compare it to a state stated significance level. And none of that was given. So you couldn't even know whether or not something is statistically significant based on sample size. You can do a null hypothesis significance test on samples of like three or four. That's entirely feasible. You could have a sample of 10000 and not get a statistically significant result. It's a function of all those other things I just mentioned, not just sample size. You said I mean. Yeah. So we talk about the power of a test, etc.. Those are all things that matter when you're talking about how much information you're getting out of something. But once you figure out what statistical significance means, you figure out that you don't even really care about it. It's not for informing decisions. If something statistically significant, that does not mean probably true.

Harpreet Sahota: [00:41:52] So why is it so challenging for people to understand that concept of statistical significance and what it actually represents?

Douglas Hubbard: [00:41:59] They kind of remember that they heard the term sometime, maybe in some undergraduate stats course or something, and they might have heard a few times in business. No one ever challenged. They heard somebody say that's not statistically significant sample size. That person didn't do any math and was never challenged on it, so they think it's OK. In fact, the resistance to the adoption of quantitative methods, a big part, it comes down to a statistical illiteracy. So I've actually discussed some of that, in my other books. like, especially the fourth book on how to measure anything in cyber security risk. We found that a very strong correlation between in a survey between statistical literacy and how accepting people were quantitative methods. It's probably not too surprising to learn that the higher people scored in statistical literacy, the more accepting they were, you know, statistical methods in something like cyber security risk analysis. But what was interesting, actually, is that the people who resisted weren't just the people who chose. I don't know a lot on each of the stats literacy questions. There would be like four or five choices and multiple choice, multiple choice tests. And one of the choices was always, "I don't know", in each each case you had that as an option. The people who chose "I don't know" a lot weren't necessarily more resistant to quantitative methods. It's the people who thought they did know and were wrong. It's not just literacy. You have to improve. It's not just that we have to learn new things about statistics. We have to unlearn misconceptions. That's just as important. People have all kinds of misconceptions about statistics. They think they can't learn anything from the sample size if it's only twelve. They didn't do anything after that. When they do the math, they're routinely surprised what they could infer. In fact, maybe even run across people thinking this comes up a lot in Data science. You've got something where there's a lot of uncertainty. Therefore, you need a lot of data to measure it. That might be an implication, a lot of people act as if that's the case. But mathematically speaking, just the opposite is true. The more uncertainty you have, the bigger uncertainty reduction you get from the first few observations. If you know almost nothing, almost anything will tell you something. That's the way to think of it. That's what the math actually says. Everything else is people winging it. There's a Mark Twain quote I was just going to mention. He said it. It's not what you don't know that will hurt you, it's what you know that ain't so.

Harpreet Sahota: [00:44:06] Yes. So what do you think those people who say that, who make that claim that sample size is not statistically significant? What do you think they're actually trying to say?

Douglas Hubbard: [00:44:14] Well, they're saying more data is better. Which is true yeah, sure more data is better. Well does that mean you can't make any inference at all from the data given? No, it does not mean that. When you do the math, every single sample slightly modifies your uncertainty. And as we showed, as I talked about with the information value calculations, even marginal reductions uncertainty can sometimes be worth a lot of money. Depends on what it is. Right? So could sampling three more customers be worth a hundred thousand dollars? Yeah. You could certainly come up with situations where that would be the case. Sampling three more customers would reduce your risk on some big risky decision by enough that it be worth a hundred thousand dollars to avoid that sure absolutely. When they test cars for safety and they do crashes with cars, they sample a thousand cars or a hundred or even 30 cars and crash them all. Now they get very few samples of cars to do this with and they have to apply a lot of the math. This is more about you have more data than you think kind of problem. So you have more data than you think and you need less than you think. The other way that people make an error here is they think the Data that's relevant is only the data right in front of them. So, for example, you might say, suppose you work for a firm and you say, well, we've never gone into this kind of product before because maybe you're chemical engineers should come up with a new kind of battery, let's say. Well, you've never done a battery before. So you think you've got no data on the success of batteries, of new batteries. Wait a second, you're just in one company. Why aren't you looking at all the other companies? Right. Why don't we look at a big history of battery development or something? So I hear this quite a lot where somebody says this is a new technology. This has never been done before. But wait, there's a long history of adopting new technologies. What is the history of the adoption of new technologies tells us? It's sort of like this, I've never died before. I've never died once. Take my word for it. I have not yet died. So how does my life insurance company compute a life insurance premium for me? Well, they say they have zero Data on my rate of death. Other than I haven't died yet. Well, now that's a wider net. They look at a bunch of people not exactly like me, and they interpolate statistically. But if everybody thought that they were so unique that making inferences from a larger population would be uninformative. Well, if that were true, then, you know, insurance companies wouldn't make nearly as much money as they do. So they are pretty profitable. Insurance companies in good times and bad, right? They're pretty profitable. Real insurance companies and people like this. So, yeah, they know what makes money and they know that if you want to forecast the risk of somebody dying, one way to do it is to gather a bunch of data on people dying and try to interpolate it statistically for individuals.

Harpreet Sahota: [00:46:42] So you get some really cool and interesting, purely philosophical interludes in how to measure anything. Can you talk to us a bit about the inspiration for these interludes and what you're hoping to communicate with them.

Douglas Hubbard: [00:46:54] Yeah. So I thought there were some philosophical, conceptual issues that needed to be addressed at some point, but I didn't want them to get in the way of a practical business discussion about, you know, here's what you do to reduce uncertainty on this kind of problem. Right? But sometimes people are reading explanations about how to measure something and they resist because they have a disposition that's related to one of those philosophical interludes. So I have to address that kind of objection at some point. So my first edition of that book did not have purely philosophical interludes. And I added those later on because even though I'm trying to keep it very pragmatic, I would keep running into people that would resist parts of it because they didn't realize that they were actually adopting a particular philosophical method or philosophical position, which was not, you know, resolved or universally accepted among statisticians or mathematicians or scientists in general. So I had to point out that, hey, what you think is true is necessarily true. And let's talk about the issues with it. So one of the philosophical interludes I had was the difference between Bayesian and frequentists. And so when somebody says, "I don't know the true probability", you may have heard someone say that. "I don't know the true probability". That statement by itself presumes a particular position on this issue. That statement implies that probability is an objective feature of the universe, not a feature of you. To a Bayesian, you're the one that has the probability, you have uncertainty about your environment, your universe, the economy, COVID 19, etc. And a probability is an expression of your uncertainty. That's the way a Bayesian looks at it. And Bayesian's can do a lot of things that frequentists can't because of that. Like, for example, I don't know if you know this, you're not. But a frequentist can't tell you the probability that a new drug works. Even though all clinical trials are based on frequentist methods. Well, almost all of them, there's more Bayesian methods now. So why is it that some study that comes out and says that this diet had a statistically significant result for obesity reduction? How come they can't say there's a 90 percent chance claim is true? They can't by the way, they can say it's statistically significant. But that's not the same as saying the probably the claim is true. The only way you can state the probability the claim is true is you had to have a prior probability. In other words, you had to have a probability that the claim is true even before you did the study. If you had that, then you can work out what the probability is after some set of observations. Because the observations, in a Bayesian point of view, update your prior probabilities. Probability is an uncertainty of the observer. It's a characteristic of the observer, not of the thing that you observe. And we the observer makes a series of observations, they update their uncertainty. So a Bayesian can tell you the probability that a claim is true, [Inaudible] the frequentist approach can't. Even though that tends to be the... That it was actually kind of historical accident, that a lot of science ended up being more frequentist. There are actually dates on this early on. There are a couple of guys went around and gave a kind of a show about this, well can you imagine this? A frequentist and a Bayesian debate and they go around the country giving talks, debating each other and people would flock to these things. That's what happened. This guy, RA Fisher and Harold Jeffreys. So Harold Jeffreys was the early Bayesian, for some reason I drew a blank. Like I say his name all the time and I just drew a blank. I'm 58. I hope I'm not going into early dementia. So Harold Jeffreys, he was one of the earlier Bayesians. He and RA Fisher will go around the country debating, basing versus frequentist methods in science. And RA Fisher was the frequentist and RA Fisher was a pretty obstinate guy and everybody knew him as being as noxious, obstinate guy. And they were both scientists. I thought was telling what kind of science they were in at what time, what year they were in that. But RA Fisher was a botanist. Right? So he in fact, for most of his career, he was a Pre-DNA botanist. So this was botany without the knowledge of DNA. So in a lot of ways, other than heredity and so forth, that there weren't a lot of the fundamental theories in Botany they You know, Darwin's evolution. You know, that was pretty widely understood. And we understood something about heredity. But there wasn't really DNA, so there wasn't deeper philosophical understanding of what was going on. There were still a lot of randomness that could explain most observations. They all they had was just the observations and trying to make inferences about these observations. Well well, RA Fisher was the frequentist, he said probability is an objective thing. That's a feature of the universe and you can't do things like put probabilities on one-off events, like you can't put a probability that volcano is going to erupt in the next hundred years. And it's never been observed erupting previously. Or you can't put a probability on that nuclear power plant is gonna have this previously unseen event occur that causes a release of radioactive material. According to him, you could never put a probability on those things. And the way he was describing it. Yes, I'm sure you could. But Harold Jeffreys, in fact, many mathematicians earlier than Jeffrey said, no, you can. In fact, pragmatic actuaries do that routinely. And this goes all the way back to Bayes and LaPlace. LaPlace had something called the rule of succession, where you could make inferences about really rare things occurring. Even things that have never occurred before. So it's sort of like this kind of the problems close I've got to earn with a bunch of marbles in it. Right. And the marbles are each red or green. And what you don't know is the proportion of red marbles to green marbles. Right? It could be 10 percent red, it could be 92 percent red, 51 whatever. Anything between zero and a high percent red. And each possible proportion is equally likely as far as you're concerned. Right? So you start drawing models. You draw eight in a row that are all green. What's the probability of the next one you draw will be red? Well, some people would say, RA Fisher would say that you have no way of computing that, you have no knowledge of what the true probability is and what class would say, no, it's 10%. In fact, you can run that simulation over over again and you'll find that that would be true if you randomly generated urns that had uniform distribution of proportions of red marbles. And then you do this experiment over and over again where you'd selected a number of marbles and look at just the situations where you select eight in a row that are green, of all of those you would find the next one was red 10 percent of the time. You'd have to run a lot of simulations to see that. But again, it's not that hard anymore to do those sorts of things. I think anyway that's sort of intuitive, but it's true. That's easily testable. So, yeah, that was one of the philosophical analysis was Bayesian versus frequentists, because someone who is assuming that probability is ultimately a frequentist point of view would say a lot things I was talking about were impossible. But they're just presuming the frequentist point of view. You might think the probability is well defined term among statisticians, no it's not. It's still fundamentally debated what probability that means. But to a Bayesian probability is a state of mind of an observer, which is the most useful way of thinking about poor decision makers, (Inaudible)

Harpreet Sahota: [00:53:31] Absolutely agree with that. Well, I used to be an actuary. I used to be a bio-statistician having done a study statistics in grad school and from the very frequentist point of view. Once I started adopting the Bayesian outlook on life and even "Bayesian psychology", things just became so much more clear to me. I almost felt like I became a happier person. By adopting a Bayesian point of view.

Douglas Hubbard: [00:53:55] You know, actually what I've always said, though, no matter what somebody says, they are Bayesian or frequentist. Everyone is Bayesian when they bet their own money.

Harpreet Sahota: [00:54:02] Yeah.

Douglas Hubbard: [00:54:02] Because a frequentist would be stunned into inactivity if they were die-hard frequentist and that never violated their principles. Every frequentist on a daily basis betrays their philosophy. They routinely make decisions under a state of uncertainty, and they make an optimal choice anyway. Now, I guess what they would say is, well, they do that without making an explicit probability. [Inaudible]. But they're still thinking that somethings are more likely than others. That's why they take the actions they have, because they expect different outcomes under some actions than others. So they're saying that this probability of this occurring is more here than there. You probability of dying is higher without a seat belt than with even though you're in a car that you've never drove before and down a road you've never driven before. All you're saying is one state is higher than the other. Well, I think you're saying something there already. That's fundamentally Bayesian. So, yeah. And by the way, the article I wrote in The American Statistician was all about using Bayesian methods to infer the probability that hypothesis is true in a field that had only previously done null hypothesis significance tests. What you do, this is the trick, you actually use their replication data in this calculation. I have the whole proof for it. So if you're in a field of research like clinical psychology, where some of these have very low replication rates, that they have a replication rate of, say, 40%, like somebody finds us that this less significant result. And the next person tries the same experiment, you know, doesn't find us that this statistically significant result. They weren't able to replicate the experiment or they might say reproduce the experiment, they use Pedantic statisticians will make a difference between those two terms. But if they're not able to get the same result, again, that results in a low reproducibility rate for their field. And if I know the reproducibility rate for the field, I can actually make an inference about how likely the claim is true once they found a statistically significant result. Well, I can computed if there's a higher reproducibility in the field as well. But if you have during field where the reproducibility of studies is, say, 40%, the chance that the claim is true, given that you found a statistically significant result. It's really more like 50%. It's when it's replicated that it's started a little over 90 percent chance being sure.

Harpreet Sahota: [00:56:08] I would have to check that paper out. Not definitely. 

Douglas Hubbard: [00:56:10] The fault control in a scientific research.

Harpreet Sahota: [00:56:12] Definitely I'll leave a link to that in the show notes as well. So last formal question here before I jump into real quick lightning round, and that is what's the one thing you want people to learn from your story and from your work?

Douglas Hubbard: [00:56:24] One thing I want people to learn from my story and my work , if you have an idea that there's a lot of evidence, objective evidence, we all have confirmation bias. Right? But if you keep finding evidence that something is true and you feel a lot of resistance to it, stick to that idea because there's a market there. Being an outlier, being the contrarian can be very successful. If you have a good reason for being the contrarian. Right? So if you happen to stumble across that kind of an idea where you believe something different than most other people, but you can identify that most other people disagree because they misunderstand something or they are not aware of some data that you're aware of. And you've got lots of things to back up your position. I'm not just saying, you know, you just happen to you're a flat and most people aren't. You know that. But if that's the case, now you want to focus on things where you can actually support your claims very strongly. You cross all the T's and dotting the I's. You can see that some things work better than others. Some statements are more true than are true and others are false. And there's a large part of the population disagrees with that. There could be a market for that. You could be the town crier on that issue and it could be almost anything. Right? So if you if you come up with a what's the list of six things that cause small businesses to fail and they're not what people expect, it's based on a bunch of research. You might think about how to turn that into something of value for people. Right? I guess that's the thing is everyone just went around assuming that certain things aren't measurable. And I said, I don't think that's true. I think I can show otherwise. I think that to show that everything's measurable. And I think I can show that the belief that something is immeasurable is based on a short list of misconceptions that we can correct. And so, yeah, I guess that's the the story I would have people take away from this is, you know, if you've got an idea that maybe a lot of people would initially reject, but you've done your research and, you know, they rejected it for the wrong reasons, then maybe that's something you could build on.

Harpreet Sahota: [00:58:19] I didn't jump into a quick lightning round here. First question is, if you could meet any historical figure, who would it be?

Douglas Hubbard: [00:58:27] Enrico Fermi I would guess. Just because he's all really smart. But, yeah, I mean, a lot of guys I say are smart people that Enrico Fermi was one of those hundred maybe IQ people support that. So sure should be a great to meet him.

Harpreet Sahota: [00:58:38] So I got to ask what's the one thing you would (Inaudible) Oh, yes. That's who I would want to meet as well. Richard Feynman. So what's the one thing you would say we truly cannot measure? Is there anything?

Douglas Hubbard: [00:58:52] The thing you utterly can't define at all. In other words, it don't know how to measure "a bleem". I don't even know what I mean when I say it. Right? That's what people ask me that how would you measure faith? And I start asking the same kinds of questions you know that I asked you. And I say, well, basically, if it seems like something you can't measure, it means that you don't really know. You thought you did. But there's a famous quote from Lord Kelvin about, you know, if you can't measure something, you don't really understand it. Right? I just paraphrasing there, but I've got the quote at one point in my book, but that's the idea. It might feel like you understand what something is. But if you can even name the consequences of it, why did you think you knew what it meant? Right? Are we fooling ourselves? The only other thing you can't measure, by the way, are things you already know perfectly. So those things you can't reduce uncertainty because apparently you've already measured them to the maximum logical limit. So I actually talk about that in a book because I talk about Heisenberg's uncertainty principle. But sometimes people will say, aha, I can stump you. Heisenberg's uncertainty principle. You can all be measured. Let's say the position of a particle at some point. OK. And I would say that's not a stamper. If your limit to measuring the velocity and position of a particle is Heisenberg's uncertainty principle. Well, then you've already measured it quite a lot. You basically eliminate all uncertainty could it possibly be eliminated. Yes, it's the same as knowing something exactly. For all practical purposes, there's no theoretical state of less uncertainty about that. It's like you've got zero remaining uncertainty that could be reduced. So if somebody says, I know exactly the change in my pocket. Well, I guess I technically can't measure that any further because it's all about uncertainty reduction. You didn't measure it once because that's the only way that you know it exactly, but you can't measure further. So then that really changes the question to whether or not you mention it further or indefinitely. Once you get the zero uncertainty, you can't go negative uncertainty. So, yeah, those are the only things you can't do. Either you have no idea what something means. It's a word from a foreign language or it's a made up random word and you just can't even think about what it is. You can't give a single example of it about when you've ever observed anything like it. It's meaningless, that can't be measured, although the meaninglessness can be measured, actually. And the things that you've already measured so perfectly that you can't possibly, logically, empirically reduce uncertainty any further because it's impossible. That's it.

Harpreet Sahota: [01:01:19] If you could have a billboards placed anywhere, what would you put on it?

Douglas Hubbard: [01:01:23] Measure what matters, make better decisions.

Harpreet Sahota: [01:01:25] So what's the number one book, either fiction or nonfiction or even one of each that you would recommend our audience read, and what was your most impactful takeaway from it?

Douglas Hubbard: [01:01:33] Isaac Asimov's Foundation series Right? There was this guy, this character in that series, Henry Seldon and he was an expert in psycho history. It was about forecasting things, about broad patterns. So he called psycho-history and I don't know why he called it psycho-history doesn't seem like an obvious thing. But it's sort of like, you know, really big data sort of analytics about very broad patterns in society, which you kind of could do now, actually, because we leave so many data points in our social media in a publicly available way that you can get these general maps about the movement of society that were never possible before. It's kind of like seeing the first weather map when they put together the first one in the London Times by people at different telegraph stations around the country reporting the weather. And then the following day in the London Times, they would show where a weather front was and people never seen that before. It's kind of like seeing a satellite map, right? A satellite image of cloud formation. But it was actually in the 19th century when the first one came up. It didn't have satellites that obviously, but they had a bunch of telegraph stations and people sending in their data. And so people were looking like wow, weather has a shape on a macro scale, they never saw that before. Well, we kind of have something like that now for social sciences. So, hey, we psycho-history that Henry Seldon psycho-history in a foundation series, Isaac Asimov. For nonfiction. Robert Zubrin, The case for Space, kind of got me interested in certain areas of engineering again, because he's a big plan of Mars mission, big fan Mars missions and so forth. But also, what's a good one, I think he plans thinking slow well it's a lot more recent. I was well, I was already well into my field. I already had a well-developed business by the time I read that Daniel comments book there. Yeah. You know, in a way, I can almost say I had more influential journal articles. So now, but for fiction that I think I have to think about that one further because that the most important thing I had for fiction. Yeah, I think from the point of view of what I'm doing right now, that's a foundation series from Isaac Asimov with Henry Seldon.

Harpreet Sahota: [01:03:33] Yeah, definitely. I'll add that to the show notes and check that out myself. Yeah. So what is the best advice you have ever received?

Douglas Hubbard: [01:03:40] Best advice are I've ever received. You don't have to be fearless but you can control your fear, I think so. I was actually in the Army National Guard for many years and at one point in time I went to airborne school in Port Maine, Georgia. And they push out of a plane, most guys had to be pushed out, but some did. I shouldn't have been pushed out. But you get a let your heart beat. So it's going up. And you jump at night too, that's something civilian jumpers don't do. When you jump at night and you come out low altitude, that's the other thing that you get this different and you jump up and jump with a bunch of other equipment, that's the other thing that's different in those situations. So, yeah, you know, your heart gets beating, that kind of stuff. But it's not about being fearless. It's about controlling fear. You know, I mean, you don't have to be fearless. You can that's not what people military people tell you. That's not what bravery is. Bravery isn't a lack of fear. It's the overcoming of it. Right? Fearlessness is just crazy. I was truly fearless. Not is truly fearless is what it is. But no, you have fear because you're a rational person and then you can still follow through what needs to be done. So that's good. Yeah. OK, good. Yeah.

Harpreet Sahota: [01:04:42] Hey, so where can people find your books?

Douglas Hubbard: [01:04:44] Oh well of course I've been on Amazon for many years. So just Google how to measure anything. If you want a signed copy, you can go to our Web site, Hubbardresearch.com. We actually now have we're watching the AIE Academy. So we've been teaching webinars on all these topics for many years. And now we're moving more toward computer based training in combination with wide workshops with instructors. So you can take part of it self-paced Computer based training and part of life instructors and many of the classes have other useful spreadsheet tools for solving a wide variety of problems. You don't have to memorize the statistics, we just give you this control experiments and regression models, random sampling, new friends and all sorts of problems. Even Monte Carlo simulations in a simple Excel spreadsheet. So we just give people the tools for that.

Harpreet Sahota: [01:05:30] And people could find that right there on the Hubbard research Web site as well?

Douglas Hubbard: [01:05:33] Yeah, yeah. That'll be in the AIE Academy section. And we've got all these scanned on courses, which you can take one at a time where you can get the whole curriculum so that you can sit for suffocation and be certified in Applied Information Economics.

Harpreet Sahota: [01:05:46] So you mentioned your website. How else can people connect with you? Where else can they find you online?

Douglas Hubbard: [01:05:51] Well, my business is one Web site. Hubbardresearch.com. But the books have their Web site called Howtomeasureanything.com. And there's different pages within that site. One for each book because they're sort of books out now. And each of those pages has downloads unique to that book. So for each of the books, we made spreadsheets that show specific calculations that are relevant for different kinds of problems that you can just download. Those are back, those are free, the ones that we have for the books. So it'd be helpful to read the book. So, you know, the spreadsheet does that button. And then, of course, you know, the webinars make you skilled at it and we give you a whole bunch of additional tools that you can download from the site. So if you really want to be a wizard at measuring things that seem utterly impossible to measure and how to improve even those complicated, riskiest decisions. Well, then AIE academy is waiting.

Harpreet Sahota: [01:06:43] I'm definitely going to take a look into that myself. Doug, Thank you so much for taking time out of your schedule to be on the show today. Really appreciate you coming on and sharing your insight and wisdom with us. Thank you.

Douglas Hubbard: [01:06:53] Yeah. Thanks.