David Spiegelhalter_mixdown.mp3

David: [00:00:00] Essentially what probability theory allows us to do is to make assumptions about how the world works, how the data is generated, and turn it and flip it around after we observe some data into statements about our uncertainty about underlying features of the world. We can do that, which of course is very explicit based on work indeed, where after processing data or uncertainty and it turns into uncertainty about the underlying quantities.

Harpreet: [00:00:38] What's up, everybody? Welcome to the Artists of Data Science Podcast, the only self development podcast for data scientists you're going to learn from and be inspired by the people, ideas and conversations that will encourage creativity and innovation in yourself so that you can do the same for others. I also host Open Office Hours. You can register to attend by going to Bitly.com/adsoh forward slash a. Ds0h. I look forward to seeing you all there. Let's ride this beat out into another awesome episode and don't forget to subscribe to the show and leave a five star review. Our guest today is a mathematical statistician dedicated to helping the public understand, risk and make better decisions under conditions of uncertainty. He owes a PhD in mathematical statistics from the University of London, was the president of the Royal Statistical Society. From 2017 to 2018 and since 2016 has served as the chair of the Winton Center for Risk and Evidence Communication at Cambridge University. He's a fantastic [00:02:00] scientific communicator who has been featured numerous times on Radio four and in many documentaries such as Morgan Freeman's Through the Wormhole and the Joy of Stats. In addition to hosting a podcast called Risky Talk, he's written articles with entertaining titles such as How Dangerous Is Burnt Toast, Choose the Yum and Rise of the Yuck and A nine point guide to spotting a dodgy statistic. However, you may recognize him as the author of several books on the topic of statistics, including The Art of Statistics and his latest book, COVID by the Numbers. So please help me. Welcoming our guests today, a knight in shining armor. Professor Risk himself. Sir David Spiegelhalter. Professor, thank you so much for taking time out of your schedule to be on the show today. I appreciate having you here.

David: [00:02:55] Oh, it. Great. Thank you. Wonderful introduction. I did do this other article which was very successful on the BBC website. Will I Live Longer Than My Cat? And that attracted a lot of viewers. It's still there.

Harpreet: [00:03:08] Yeah, I'll definitely have to link to that one in the show notes for sure. Where I live longer than my cat. That is not a very interesting title. I mean, you've definitely written some really interesting articles on statistics, a number of amazing books on statistics. This is definitely one of my favorite books on statistics right here. But talk to us about how you first got interested in statistics and what was it that drew you to this field?

David: [00:03:31] Oh, yeah. Well, I started off doing math and I did math at school is what I was quite good at. So I did math at university. I went to Oxford and it's really very pure math and I like the pure math. But to be honest, by about halfway through the second year, it got too difficult, you know, just the level of abstraction so high. And I was getting a bit discouraged, but fortunately I had an inspiring like so many people, I can go back to an inspiring teacher. And he was a young man. [00:04:00] He's now Professor Sir Adrian Smith, president of the Royal Society, top scientist in the country, essentially one of them. And then he was then 25 or something like that, and he was our tutor in to teach us math. And he was he was not only interested in statistics, he was interested in Bayesian statistics. And so he was translating definitely his book on the theory of probability. So we used to sit in the pub and have a long discussion about what is probability, what does it mean? Whereas and those, those, those arguments over beer have stayed with me and inspired me and kept me going. And I can just as I'll have an argument about what is probability unchanged from 50 years ago and I'll just launch in and we'll start arguing about does it exist or not, the Bayesian versus the frequentist paradigm. And I just let just rewind me up and off I'll go just like 50 years ago.

Harpreet: [00:04:55] And this is one of the main reasons that I'm super excited to have you on the show because, you know, I want to ask you some questions about this, about what will probability is in Bayesian statistics and stuff. A couple of weeks ago, maybe about a month ago already, I had a friend of yours, a professor, Marcus du Sautoy, on the show, and he's the one that introduced us and put us in touch. Professor du Sautoy, thank you very much for that. Never in my life would have thought that a kid destined for failure, such as myself, be sitting here talking about probability and mathematics, which esteemed professors like yourself. But let's let's get into this. First thing I want to ask you is, is why is it that it seems like mathematicians tend to dislike teaching statistics?

David: [00:05:32] Oh, yeah, yeah. No, because statistics is not part of mathematics. And there's there's mathematical statistics, which I used to teach in Cambridge to the math students. And the only way to make it acceptable, I think, to the students was to make it very mathematical and is all in terms of proving theorems and proving rules and laws and things like that. So there is this part of statistics that essentially is very mathematical, and that's what I learned and that's what I've taught. But that's not [00:06:00] statistics. That's not what I do. I've been a professional statistician for 45 years or so and I've done some math and it's been incredibly valuable to know some math. But that's not what I do, and I think I'm a proper statistician. It's not a part of mathematics. The crucial difference is that in statistics there are not yes no black or white answers. That's the big difference. I've finally decided that's what makes it. Parts of mathematics. There's always judgment involved in my Art of Statistics book. I got a lovely quote from Sigel in The Noise. Nate Silver's book. Great book. And he really hammers this on. He says, The numbers do not speak for themselves. We speak for them. We imbue them with meaning. So that's what I really love about statistics, is that it is uses math.

David: [00:06:50] It's a mathematical science, I suppose, but it is to do with interactions with the real world and people. It is embedded in real problems that people face. And so that's why I love it. But it's not part of math. And no wonder mathematicians find it. They don't want to teach it. I don't blame them at all. It's very unsatisfying, partly because you don't even know what is probability anyway. You don't even know what these things mean. And so it's a deeply contested subject. That's what makes it such fun. I love it. It's it's yeah. I'm not at all surprised that mathematicians don't want to teach it. And that's why they shouldn't teach it in universities. Mathematicians should not teach statistics in universities. And the real problem happens in schools, because in schools you've got math teachers and some of them quite like teaching stats. A lot don't like teaching the stats because there's no yes, no black white answer. And so that's a real problem because there is not a dedicated training group of people who are trained to teach statistics and data science in schools. And I think that is a real, real problem. We might get to this for the future of education in the [00:08:00] world.

Harpreet: [00:08:01] It's interesting. I was interviewed, Professor Andrew Gelman, maybe a year or so ago. Can't remember how long it's been now. He's awesome. And he made this a statement that I just it really stuck with me. He said Statistics is the least important part of data science. And I just found that to be such an interesting, interesting idea.

David: [00:08:18] But it's just it's just part of it. It's just parts of it. So I've been really inspired by it. I've been really inspired by Andrew as well, I find. Yeah, yeah.

Harpreet: [00:08:27] Yeah. Definitely. Very interesting. Interesting. Professor. I enjoyed speaking with him. But as you mentioned in your book, rigorous definitions are important in statistics. So I guess what is statistical science and what is it all about?

David: [00:08:40] Yeah, that's a good question. I kind of think of it as the art of learning from data, and that's what I call my book, sort of use those terms because although it's a statistical science, I call it the art of statistics, my book, because there is elements of strong judgment in there. It's not it's not like some algorithm you apply to data and it gives up its answers. No, no. And that's what Nate Silver said. It's it's it necessarily involves judgment. And that's what makes it so delightful. But it is but it is based on data, on numerical information, on counting things. And then and it involves the whole business of deciding what to count and and going out there and getting it and cleaning it up and all that kind of stuff. But crucially, it then involves that step of deciding, well, what does it mean? You know, what can we learn from it? What can we conclude with all the uncertainties and all the limitations? How does it how does it answer our questions? We actually started with and that's what's so interesting about it. And I think it makes it into a beautiful, valuable, fascinating and infuriating subject.

Harpreet: [00:09:46] And as you talk about in your book, The Art of Statistics, which I highly recommend everyone check out definitely one of my favorite books on statistics and probably one of the only books on statistics that I've read that has given a clear framework for how to handle problems and approach problems in statistics. [00:10:00] And you call the P, p, b, a C cycle. Tell us about that framework.

David: [00:10:05] Yeah. I mean, I think I if someone from New Zealand told me they refer to it as P, I think so. That's stolen from New Zealand. But originally Wayne Alford I think developed that in Canada, in Waterloo University. And the whole point is I start the book is about problem solving. So the pdac you've got it starts with P, the first B is a problem. It's something you actually want to know about the world. And it may be a prediction maybe to understand causation. It may be just to know how many of something there are out there. So some problem that you want. And then the second P is the plan. You know, actually, can the data answer maybe you just can't answer it, you know, b then just give up, you know, just know that you have it. You just cannot answer this question a bit like saying, okay, I want to know how effective face masks are. Well, there isn't any data out there. There's something I've got to tell you this this is unanswerable, essentially from the data that's available. So then you want to go out and collect what you can, see what there is, and maybe do an experiment or whatever. So then you have to plan it and think about it in advance. I think that's a really crucial statistics is not just responding to data, it's innovative and experimental and and imaginative. So then then you get the data and that's just you just get the data. Well, of course, that's a massive issue. Collecting it and cleaning it and checking it and doing what? Missing data and all that stuff. So but basically it's turning it into a nice form.

David: [00:11:37] It's that wrangling with the collection and wrangling to turn into something you can do something with and then you do the analysis. That's the A But even that's complicated because you've got exploratory analysis. Just looking at the data, drawing graphs, that may be quite enough. And then you've got a confirmatory analysis where as the one little bit and this is what Andrew I think is referring to this, that's the one little tiny bit where [00:12:00] all this stuff that we learn in statistics like probability theory and sampling, distribution of the sample mean and confidence intervals, blah, blah, blah, there's one little bit where that comes in. Then you get into the see, which is conclusions and communication, and that's unbelievably important because it's working out what can I actually say and how am I going to say it? And then the whole thing starts again. The crucial thing is then you just go straight. All that does always is generate another problem, another round around you go. So when they really develop this in the New Zealand education system and they really get kids to do this cycle very quickly, the whole thing in an hour can just, just really do it again and again and again, very, very brilliant training. And so I this is just learning about this different way of teaching it and has revolutionized my, my ideas about teaching stats and led to the book being having a very different structure, which probability doesn't come in until two thirds of the way through and so on. But there's lots of people now who've flipped around. That.

Harpreet: [00:13:02] Yeah. Yeah. That's something I want to touch on a little bit later is why is that probability? Because, I mean, I studied stats in in grad school a little bit in undergrad as well. And I took a ton of statistics before taking the first probability theory course. And at that point it was probably like third year undergraduate before I took like first year probability course. But yeah, I guess why is it why is it that like we put these statistics cart in front of the probability course?

David: [00:13:26] Well yeah. I mean the traditional way of teaching is the probability comes first. And when I was teaching it in Cambridge to the math students, yeah, they'd done all probability because traditional statistics start straight away. You might do mean median and mode and a few summary stuff, and then you get straight onto sampling distributions of individual data points and then that leads to sampling distributions of statistics, sample means central limit theorem and all that stuff that that builds this structure for the mathematical results, for constructing confidence intervals and so on. But you don't have to [00:14:00] do it that way. That assumes probability comes first. And in the book I was amazed in writing book took ages to structure it, and it was a revelation to me that I could write nearly the whole book with only the idea of picking something at random, which a three year old child can understand that you stick your hand in and pull something out. That's all you need for nearly all the statistics, especially if you introduce uncertainty through bootstrapping. So it's quite extraordinary how you can get with those ideas before bringing in probability theory. Then it is useful to bring it together because then it means that you can do some stuff using mathematical results rather than simulation. And you can. And then if you start going to start doing P values and hypothesis tests, you can do a lot of that without probability theory, but actually it makes it hugely easier if you've got some probability theory behind you. And of course the Bayesian stuff absolutely beat probability because Bayesian statistics is just a branch of probability, so it's absolutely essential for that.

Harpreet: [00:15:03] Before we move into some of those philosophical discussions, that last question. You mentioned in the book that statistics is kind of to blame for the reproducibility and replication crises in science. Why? Why is that?

David: [00:15:19] Well, I wouldn't say the statistics is the statistics is to blame for it. It's just that there is a you know, there is a real problem. And and statistical, you know, in a way, misuse of stats is part of that, I think. And that's very recognized that I know that I can get a set of data and I can prove almost anything from it. If you give me long enough, talk to the data to it, gives up the answer you want. And so that's why the important pre specification analysis and so on is though if you really want to do a confirmatory analysis, you should specify it beforehand and and so on. So I think that it's a part of that, [00:16:00] but there's lots of other aspects to do with lack of reproducibility and replication. I mean, just but since so much of the claim, the claims in science are based on finding a statistical result, then it's the lack of reproducibility of those statistical results does become a very, very important aspect here.

Harpreet: [00:16:21] So speaking of of learning from from data, going from essentially a sample out to general population, you talk about inductive versus deductive inference. So, yes, how can we use this process of inductive inference when we want to use data to learn about something? I guess I might have just leak the answer there a little bit, but.

David: [00:16:41] Yeah, no, I, I, you know, there's supposed to be a problem with induction. I don't see much of a problem at all, provided you realize that everything you do when you're doing induction is assumption based. There's no true way of doing it in a correct way of doing it. It's all based on assumptions because you're generalizing from a particular to a general, whether it's the future or whether it's a general population. And the only way to do that is make assumptions. Maybe the sun won't rise tomorrow. You have to make assumptions. And so once you've got the idea of a statistical model, which is a map of the world, it's not reality. All models are wrong. But once you've got that idea of a model, in other words, your assumptions about how the world works, then you can do induction and but it's model dependent. You know, whatever your understanding is, you can't make any conclusions if you've got some understanding about the underlying processes. Otherwise you can't say anything because anything could happen, everything, you know. So you'd be like a baby where everything is a total surprise. Everything just will pop up. But my dog asking, my dog is more intelligent than that. My dog realizes that one thing tends to follow another and she gets very upset. I don't feed her at the right. If I picked up a bowl and didn't feed it, she'd be staggered. She just couldn't. So the dog can do it. So but that's because the dog has got a model, a little internal model of the world that I'm [00:18:00] somehow a reliable person. Probably not, right? But never mind. But so I don't see a problem with it, really. But as long as you realize that everything is dependent on assumptions and those assumptions will be wrong, you're always inadequate. Every model is wrong, but you can go a long way with it. But you have to have some humility about acknowledging the limitations of what you're doing.

Harpreet: [00:18:23] So I guess just kind of personal question a little bit more here. So when we talk about induction and inductive inference, like should the should the philosopher in us get worried at all about the problem of induction in statistics? Not at all.

David: [00:18:37] I don't think, no. I mean, I can read a little bit of HUME and see the sort of complaints about that. But there's no you know, there's obvious paradox. There's no reason, no logical reason why this should be possible. And I completely agree. But that doesn't matter because you've got to do it and everyone does it. And so yeah, I mean, I guess that's why people don't like, you know, people don't like mathematicians, don't like statistics. You have to make this sort of leap of faith almost because it's not like from axioms you can show something with deduction in the very problem in statistics is that, you know, the axioms are untrue. You know, your assumptions are untrue. There's no such thing as a normal distribution. There's no such thing as independent observations. There's no all these things we assume are untrue. We know they're not true. So everything we say is conditional on assumptions that we know are untrue. Now, the crucial thing is how much it matters, and that's a matter of judgment. And so I think no wonder mathematicians don't like it. It's totally opposite subject, totally completely in the wrong direction.

Harpreet: [00:19:40] So just the statement that we know, the normal distribution, it's untrue. Talk a little bit about that. I know that some of the audience members listening are going to be scratching their heads like, what do you mean, normal distribution?

David: [00:19:50] Oh, for example, you know, the normal distribution has got a domain from minus infinity to infinity. When we analyze people's heights, it's actually they're [00:20:00] pretty normally distributed in the population, but they don't go off to minus infinity. Now, they may not master the fact that they're truncated at zero. It doesn't matter very much. And they don't go off to plus infinity. They don't get 50 foot high people even even in 10 billion. But you should do if it was really a normal distribution. So maybe not 50, I don't know. You'd have to get you'd get staggered. So anyway, so but it's in the area that matters, it's a really good assumption and it doesn't matter that much anyway. So we know all these things run, but they but they work.

Harpreet: [00:20:34] So do you have any examples of when inductive inference has failed in statistics that you could share with us?

David: [00:20:41] Well, by a million every time. Yeah. Yeah. The financial crash in 2008 was a classic example where people's models were wrong. And so they make all these judgments about what's going to happen. They completely just completely deluded themselves because of the wrong models. So there's endless examples where people just made the wrong assumption. They're making inductive inference by what will happen now in other situations, given the ones they've observed and it didn't.

Harpreet: [00:21:12] Yeah, that's something I've been running into a lot of lately is that that talk about the 2008 financial crisis, I've been reading a lot of Nassim Taleb and then some Mandelbrot misbehavior markets and it's just been changing. Kind of like I feel like everything I knew about the law of large numbers, the central limit theorem was incomplete or wrong.

David: [00:21:33] Yeah, no, no. I think and you know, in fact, as Taleb says, you just have to assume, you know, heavy tail power tails. And if the power's is wider than a Koshy, there's no central limit theorem at all. You just know things don't converge. You don't get this nice behavior. Things happen suddenly, far bigger, more extreme than anything you've ever observed before. But there's you know, there's a statistical theory for all this extreme [00:22:00] value theory. There's all these things. It's not like this is magic we know about. We actually had to deal with these situations. And it's not like this is some oh, that means statistics is wrong. It doesn't work. No, we know how to deal with these things. It's just that it hasn't been done very well.

Harpreet: [00:22:15] We're getting into it to probability here. And we touched on this a little bit earlier, but just to really solidify the concept here, why do we need probability theory when we're doing statistics?

David: [00:22:24] Yeah, this good problem. As I said, I don't think you do until you get on quite advanced stuff you don't actually know. You could say you do almost all of stats without probability theory. If you use the simulation method, which is called bootstrapping and you simulate new data sets similar to the one you've already got, look at the variation in those data sets in your analysis, and it gives you an idea of what the uncertainty is about what you're doing. But if you can use probability theory, it makes a hell of a lot easier because you can do quite a lot in closed form or through approximations and so on. It just really helps to act as if the world works probabilistically, whether it does or not, whether it's all it might all be preordained by some great intelligence where it might all be sitting on the backs of a pile of turtles. But it doesn't matter. We we act as if things work in a stochastic way. We act as if murders happen according to some stochastic process. Well, it doesn't it, but it's as if they do. You know, it's amazing how well, it's what I show in the book. Things like Poisson distributions fit the number of murders each day and things like that. So it is extraordinary how the world, through enormous complexity and vast numbers of possibilities of things happening.

David: [00:23:36] Actually what does happen follow then starts following laws, reasonable laws as if it was a random process. So. So this is brilliant. It's wonderful. So we can use probability theory to do extraordinary things prior to we don't actually believe it's all right. And essentially what probability theory allows us to do is to take assumptions about how the world [00:24:00] works, how the data is generated, and turn it and flip it around. After we observe some data into statements about our uncertainty about underlying features of the world, where we can do that flip, which of course is very explicit in Bayesian work indeed, where after seeing some data, ah, uncertainty turns into uncertainty about the underlying quantities that are, that are doing the generator. So we learn about the underlying processes and our uncertainty about those are come from naturally assumptions about the uncertainty, the variability in the data. Yeah, so statistical inference is a way of turning variability into uncertainty, which I think I was told 50 years ago by some wise statisticians and I thought, what on earth are they talking about? No idea what that is. So after 50 years, I finally come to that conclusion. They were right.

Harpreet: [00:24:52] Run that back one more time that the quote that statistics turns variability.

David: [00:24:56] Into variability into uncertainty because variability, the probability distributions in the world, the normal distribution. The point is all to do with variability. It's all to do with just how things how things vary from time to time, place to place things. And by making assumptions about that and going through the statistical machine, we end up making statements about the personal uncertainty, about what? About these underlying mechanisms, about the parameters. We could go into things with Greek letters. The things that actually produce this variability underlie this variability. The process is underneath. So it's an extraordinary thing. So because in the end, we want to use to we, we want to learn from data. And that means taking the data and afterwards coming up with some uncertain statements about how the world works and by making assumptions about how the data was, how the data are, the the mechanisms by which the data was presented. But we the clever thing is that we can do it by just making certain assumptions about how the data [00:26:00] was generated, for example, normal distributions. But we don't make assumptions about where that normal distribution is now. We have to learn from the data. So that's a simplification because we can learn about the shape of the distribution from the data as well. And also within a Bayesian framework. We start off also with some judgments about where the center of that distribution. So that's not a hard and fast rule at all, but that's the general idea.

Harpreet: [00:26:25] I think pouring into the Bayesian stuff is kind of taking a step back here, maybe first principles. I don't know if that's the right word to use in this, but but what what is probability? How do we measure it? I mean, it seems like such a strange epistemological concept.

David: [00:26:38] Exactly. You tell me what it is. I haven't got a clue. Well, I do have a belief. I mean, there probably is I would consider is a virtual quantity. It doesn't exist. I mean, if you've got time and mass and length and all that sort of stuff, the scales for it. We could temperature, we can measure it. You can't measure probability. There's no probability ometer. You can't measure it doesn't exist. It's not out there in the world at all. So I genuinely believe it's I think it's the only way only for me, it's the only way to think about it. So it's socially constructed, it's a construction. So that's why I actually I don't believe it objectively. Except possibly the subatomic level, because although there's apparently still an argument about whether there are hidden causes behind, for example, you know, breaking up or whatever it is, I think Hawking uses the term determined probabilities for the probability that a uranium atom will will fall apart in the next hour, minute, second or something like that. So these are actual the only unconditional probabilities. These are the only ones I would consider are properties of the world. Everything else is conditional, and probably any number we put on anything is conditional on assumptions and knowledge, and they will vary from person to person and their constructions. These numbers do not exist out there in the world. And definitely in [00:28:00] his book, Theory of Probability, which Agent Smith was translating when I first met him. Right on page one, in big, bold letters, probability does not exist. So I learned that when I was 19, I suppose never shifted my not actually. So that means I'm a Bayesian subjectivist and I don't believe probabilities are constructed by argument and discussion. They're not estimated, they're assessed and they'll vary from person to person and place to place.

Harpreet: [00:28:27] Can we say there's a at least some type of difference between maybe epistemic probability and some physical or I believe you say a allestree tree?

David: [00:28:37] Yeah, yeah, that can be useful, but it's like the way I usually do it. I've got a coin, I'm trying to find a coin there and I say, okay, what's the probability of this coin coming up? Heads can't find a coin and we can imagine a coin. Here's a coin and that's an elementary probability. So you might think this is the chance property of the coin may not be 5050. Exactly. Or doesn't that mean it might depend on how I flip it, but let's say 5050. So I flip it, put it over, like cover it up. What's the probability this is heads? And when I do this all the time with audiences and school kids and they'll go, Oh. Bah bah bah bah bah bah bah. And then eventually somebody, some brave soul might say, 5050. Yeah, that's your property. And then I look at the coin, it's not mine. So what I've done there is flip from three chance to epistemic uncertainty. Once the coin split is covered up, there's no one. There's no chance left, no randomness. It's purely my lack of knowledge, but it hasn't changed. It's no 5050, so I don't look at it. So I think it's that it can be a useful distinction, partly because it explains the Bayesian approach, because the bases are just as happy with putting probabilities on epistemic uncertainty, what they happen not to know. As for future events, this is no distinction between those two at all. Whereas within a classical framework for frequentist framework, you're not allowed to put probabilities on on events [00:30:00] that have occurred. Let's do it.

Harpreet: [00:30:03] So. Would there be a difference, I guess, in the way that maybe a philosopher or a statistician would interpret probability?

David: [00:30:11] Yeah. I mean, I tend not to read philosophers writing about probability because I kind of think, oh, I can't be bothered. So I, I argued about this in the pub 50 years ago. I made up my mind in that sense. So. So I can't be bothered, you know, because there is, there are some claims. Oh, yeah. Probabilities are somehow out there, there's some underlying propensity for something to happen in the world. Well, show me it. I suppose I'm very much a pragmatist. I follow experts saying, I believe unless you can show me the thing, I'm not going to I'm going to treat it as a virtual quantity and use it use it all the time. But I'm not going to present it so that I and people will have different views. Instead, our Bayesian philosophers are probably there's all sorts of different views. But the point is that nobody there's no consensus about this at all. There's no agreed. No wonder people don't like discussing this, mathematicians or even philosophers, because there's no agreed idea of what probability is. Can you believe it? You know, I find it slightly. It's like a dirty secret. You've got to admit it to people. Oh, you do realize this entire world subject we built for that problem in statistics is built on unbelievably shaky ground. The very basic ideas, the mathematics is agreed, but the very basic ideas, what it means, there's no agreement.

Harpreet: [00:31:37] I think that's part of what makes it really difficult and unintuitive to grasp and to think about and use it to make judgments. I mean, probability theory, if I'm correct me if I'm wrong, kind of arose from games, games of chance. That's that's what birthday. And in those domains you really get six sided die two of [00:32:00] them. What do you expect to see on average those types of domains? I feel like it's we can intuitively kind of grok that with that.

David: [00:32:07] They're fine. They're fine, but they're all wrong. I mean, they're all just they will make assumptions about the coins. So when I demonstrate case, I carry two headed coins with me and just say, look, sorry, you're wrong. You didn't think that, did you? So there's probably isn't a half or whatever and it's not going to convert to anything. I fiddled it. So it's all based on assumptions about the model in the process. And then you can draw some mathematical conclusions, but the coins don't come up exactly 5050. There's all sorts of ways these things just don't. As I said, apart from possibly at the subatomic level, they don't exist as as as verifiable numbers. Yeah. And yeah.

Harpreet: [00:32:48] I kind of use it as a way to make decisions. It's kind of a weird way I'm going to be saying this out loud, but I think about any action I'm going to take, if I'm going to take an action, if I'm going to do something, you know, what would the result be of this action in, let's say, 1000 parallel universes? Right. Is it a favorable outcome, 20% of that time? Favorable outcome X percent of the time. But do you ever only see one reality? So it's like, yeah, it's kind of hard to explain if that makes sense, that's great.

David: [00:33:15] No, I love that metaphor. I use that I convert toward metaphorical probabilities and I really love that interpretation. It's the only one I really like. I think, you know, if you are going to have some sort of mechanistic apart from just pure subjective belief, I like to think of that. Well, you know, if I'm talking about a future, I think, well, there are all those there's 100 possible ways things might turn out. For example, they're all equally likely all these little paths going out into the future. In what proportion of those possible futures is the coin going to come up heads? I'm going to be alive in ten years time. The world will will cease to exist in a century, etc., etc.. All these and some of them end in catastrophe and others carry on. Okay. And I think that's a fantastic metaphor and multiverse type metaphor. [00:34:00]

Harpreet: [00:34:00] Yeah.

David: [00:34:02] But the bizarre, the thing that I've been criticized for others who do this is that you end up you frequency interpretation of what actually is a Bayesian subjective judgment. So if I think some if my probability for the world ending by 2100 is 0.1 or something or is 1% the my pure judgmental probability. What I mean, one way of describing it is out of 100 possible futures, it's going to happen in one, which is a frequency interpretation which is very valuable. Fantastic. When we when I when the stuff we designed for patients to use, when they're discussing cancer risks and everything, we never use the word probability, chance or any any of those things. It's always described in terms of out of 100 people like you, what would we expect to happen to them in the future? And we don't use out of 100 ways things might turn out for you. And partly in those circumstances that don't think it's that's not the right metaphor because we know the numbers that we incorporate. Not what we use and not personalize. It's always factors about the individual that will be different. So when we talk about these risks, it's not your risk because we've only put a few things into a formula, into an algorithm. So the correct embedding of that to say out of 100 people who tick the same boxes as you would expect, so many to be alive in ten years. 60 to be alive. That's the appropriate metaphor, I think, for communicating that problem, that judgment. So I think this is really important and I think people are not stupid and that a good metaphor like that, we can really work with people and I much prefer I don't use chances, I don't use coin analogies anymore for any of these things. I use it all the time and it's just stopped. Yeah.

Harpreet: [00:35:49] Yeah. Usually when I start talking about a hundred possible parallel universes, people start looking at me like I'm crazy, but I'm glad that, you know.

David: [00:35:56] I love it. Yeah, but yeah, but people do think you're crazy, so some [00:36:00] people really get it, you know, 100 possible futures for you where you think and then climate change because there's only going to be one planet. So you can't say out of 100 planets like this, really, you have to stay out of 100 possible futures for this planet. This is what we expect to happen. And I don't think that's especially if you I've seen some lovely drawings where you see this sort of, you know, here we are at the moment and there are all sorts of ways we could have got here. So you've got all these sort of tangled web of possible causal paths. And then from now on there's this tangled web of possible ways in which things might turn out with the multiverses or whether they all happen simultaneously. Because another matter, we don't have to go into that because we're only going to see one of them and we don't know which one it is. So I think this idea of a sort of spaghetti of possible futures, one of which is going to happen, is a very powerful image. And of course, you see it. That's what Monte Carlo analysis does explicitly, is draw spaghetti plots for the future.

Harpreet: [00:36:55] There's a saw this beautiful image that that captured everything you talk about is it was a it was 100 different paths leading up to one moment in time with the with the barrier a line. And then it said, all your possible future is like going for it. So to me it was like, have you seen that?

David: [00:37:14] I made one of those by killing a picture of a squid and cutting it in half and putting it back. So there's the squid going up and the squid coming. Oh, that's really cool. I'm glad others are doing that.

Harpreet: [00:37:26] Yeah, yeah. But I'll find out. I'll send you, send you an email, but.

David: [00:37:30] I'd like to steal that one.

Harpreet: [00:37:31] When I see that, it's just like, Oh, okay, well, that means the future is a probability distribution. Obviously, the past is determined.

David: [00:37:38] It's determined, but we might know, not know which path was taken to get us here. That's true. That's true. With causation. We ask to attribution and things like that. So which is another whole problem, particularly, of course in criminology and where the probability is all epistemic uncertainty, because the uncertainty is about what happened and you're trying to judge the possible [00:38:00] paths by which this particular event happened. So in crime, everything is to do with the path that led to the current situation. It's certainly about those.

Harpreet: [00:38:12] That's. At best, I don't have any intelligent response to that one. So let's get into Bayesian stuff.

David: [00:38:20] Yeah. Yeah.

Harpreet: [00:38:22] Oh, sorry.

David: [00:38:22] Go ahead. No, that's why we're working on on on education for lawyers and things that to do with it. And it's all Bayesian, but we can't use that. We don't use that word.

Harpreet: [00:38:32] That's right. You're not allowed to use Bayesian approaches in the UK courts, right. Yeah. So I guess talk to us about that. Like what's the Bayesian approach all about and why is it that courts in the UK are banning it or have banned it?

David: [00:38:45] Well, one of the points by the Bayesian approach is that it explicitly introduces judgment and that is far from being something to be embarrassed about is something to highlight. The fact that it is it's a method of of learning, meaning that it's not just what does the data tell us. It's it's given what we thought originally, how does the data how what is it reasonable to believe after we observe the data? So you have to specify what you thought beforehand is prior distribution and which is an enormously valuable thing, but it is introducing judgment into the analysis and that's considered by many people. And you've got to therefore be very clear about what you're introducing to justify it. You're going to do sensitivity analysis, different assumptions, much better to have lots of people's opinions feeding through and so on. So it's got to be done. You have a huge responsibility in doing that, but in the end, I believe it's the right thing to do in court. You're not allowed to do that because you can't put the prior distribution in that's actually saying, Oh, this just almost prejudice, I think, or this person is more likely to have done it than anybody than somebody else. No, you can't say that. Yeah. So you're only allowed in the basis in the court is to put in what's called the likelihood ratio, which is the way the deductive aspect [00:40:00] is the probability of the evidence given either guilt or innocence or some other pair of hypotheses. That's the likelihood, the central part of the Bayes theorem. So you're not allowed to do the whole Bayes theorem business. You can only do the data bits of it and you are allowed to do that in court.

Harpreet: [00:40:16] So how is this different from the frequentist approach to viewing probability? What's the central difference?

David: [00:40:24] Yeah, there is a central difference is just what is probability. In frequentist approach. You assume probabilities that to do with long run frequencies of repeated similar events. How if I keep on doing something, how often something will happen. So that's that is what probability actually means. Whereas the Bayesian thing this is relevant but it's not, that's not what probability means at all. So according to a frequency thing, you couldn't have a probability of a particular horse winning a particular race. And because that race, it's unthinkable to think of that race being repeated again and again and again. In what proportion of times that might in that particular race, that horse might win? Now, as I said, in a way, the paradox is that in explaining a judgmental probability, it is actually quite useful to think of 100 races like that. In what proportion? So as a metaphor, it's actually quite a good thing to think about in these 100 possible universes in which the race is won. In what proportion does this horse win? But it's not. What the probability actually means is that is purely a way of communicating. The magnitude is not what it means, and that is the crucial difference within a frequentist framework. That is actually the only way to define what a probability is. And that's why when frequentist go through this business of turning assumptions about distributions into uncertainty statements, they have to do this convoluted confidence interval business instead of just saying, well, this interval is 95% probability that the true I know based on my assumptions is 95% [00:42:00] probability that this interval contains the true value. You have to say, Oh, if I repeated this process millions of times in 95% of time, this random interval would contain the unknown fixed quantity. They often say, you know, really it's a.

Harpreet: [00:42:15] Complicated way of it's.

David: [00:42:16] Not so incomprehensible and everyone gets it wrong. There again, I've taught I've taught it for years so I can move between the two I'd Hampel and in fact, of course I'd never use I would never use it when I'm actually doing my communication work, I couldn't care less. I call them all uncertainty, whether they're Bayesian because in covert at the moment some of the confidence intervals based on classical analysis, but most of them are Bayesian intervals because all the modeling is Bayesian pretty well encoded, so they're credible intervals. These are Bayesian uncertainty intervals and I don't use either word, I just call them uncertainty.

Harpreet: [00:42:52] Patients and frequentist fundamentally dislike each other.

David: [00:42:56] No, I don't think so. They used to be I mean, they used to be real. When I grew up as a statistician, there's huge ideological arguments because people were trying to fight to try to construct, in a way, a universal theory for statistics, people. Given that up, I mean, that was a. Doomed, I think. And I came in I learned at the tail end of that attempt to have a great unified theory. And that's just going out the window now. Everything's much more pragmatic and ecumenical. I move between them all very, very, very casually. It's just that you should understand what you're doing and you should be able to see what's going on and to understand the limitations of what you're doing. But there's no correct way of doing it. There's no correct theory of statistical inference. Particularly as whatever you do. I mean, this is why I always get cross with everybody, because it's all dependent on the assumptions. And the classic example I got is that in the UK there are eight different teams, great, really good building models. To estimate [00:44:00] this magic quantity are three, you know, the current number of average number of people that somebody will infect if they get code and and they come up with they do all their analyzes. They are all trying to estimate the same quantity, using largely the same data. And they all come up with completely different answers. You know, their intervals don't even overlap. They're working on you can't all be right. If your intervals don't overlap, some of you must. At least some of you, if not all of you, must be wrong. So what that means is that because these intervals are dependent on assuming the model is true and the model is wrong, so that that you think, well, thank goodness, is eight teams doing it much? Just one. And that really reveals to me, you know, it makes me very cautious, if not skeptical about a lot of statistical modeling, especially the intervals that come out of it, because they're all understatements, the real uncertainties. Whether that pays in or it doesn't matter.

Harpreet: [00:44:55] It seems like the prior distribution is something that makes base them so controversial. Why is that?

David: [00:45:03] Oh yeah. No, I was just saying. And of course it does. And it's what makes it so powerful but so controversial because it is a mathematical expression of judgment. And I know there's no avoiding that. And it's something to wave a flag. Yeah, I my judgment has gone into this analysis and to some extent, it may just be a judgment about perhaps how smooth the underlying curve might be. So maybe something I'm not saying where I think the curve is, but just how smooth it might be. So it might be just some imposition of a certain amount of smoothing in the model just to make it estimable or something like that. The point is that within the Bayesian framework, these things should be made very explicit and critiqued and justified. Whereas when there's always whereas in a classical framework, there's huge judgments being made about the structure of the model, but they're just sort of swept under the carpet sometime by just saying, Oh, that's the model we're assuming. Well, hang on, why it's wrong. This is a judgment you're making. And I'd [00:46:00] much rather this was this was a lot more explicit that you are bringing judgment into your analysis. And it's the pretense that somehow these statistics happens as some automatic process, that this is the correct way to do it is nonsensical. There's always, always judgment.

Harpreet: [00:46:18] But it seems like Bayes Theorem is like the scientifically correct way to change your mind when you get new evidence, right? So why is that the case if judgment is is so controversial.

David: [00:46:30] But because it's only internal Bayes theorem only assures internal consistency. Given your initial assumptions and some data, it tells you what you then should believe. But you might have been wrong in the first place. Everything might. You might be completely deluded. So it just assures internal consistency. And that's why something I believe very strongly is that that if anybody making any judgments is amazing or otherwise probabilistic, judgment should constantly be checking them against the real world. And this whole idea of scoring rules, which are the mathematically appropriate ways to check how good your probability distributions are. If you keep on giving tiny probabilities to events that happen, then you shouldn't be taken very seriously by anybody else. You should. You should question what you're at or what you're doing yourself. I mean, that's really shown so strongly in the work of super forecasters that use a particular scoring mechanism. They put they don't say what's going to happen. They only use probabilities. And and you find these super forecasters by their schools being better than other peoples because their probabilities are more reliable when they say something. So they get something a 70% chance. Then out of those times it happens roughly 70% of the time. But they also use they you can't get away with just doing that. You also have to have probabilities that at least sometimes are near a hundred or naught, because otherwise you're not being very helpful. So super forecasters can combine those two things. They can discriminate, but they can [00:48:00] also have reliable. And that's all based on judgment. I mean, there are some modeling perhaps, but mostly it's judgment. So I think that's a really good demonstration of these ideas that you want internal consistency, but you want also empirical validation with the real world. Yeah.

Harpreet: [00:48:18] I've been reading a little bit of David Deutsch lately as well, and he's having some qualms, I guess, with with Bayesian ism. He says that Bayesian, you know, it Bayesian ism becomes controversial when you try to use it as a way to generate new ideas or judge one explanation against another. I guess how do we reconcile that when we're faced with some epistemic.

David: [00:48:41] Yeah. I mean, I get very suspicious of Bayesian approaches, for example, to giving probabilities on scientific theories or something other. I don't I really don't like that. I don't like that. I don't even like probabilities on models, particularly because again, definitely says that you should only give probabilities to things that are empirically, at least in theory, empirically verifiable. In other words, there is the possibility that they you will find out whether which one is true. So in crime, yeah, someone either did it, did something or they didn't. You know, it is there is some underlying fact there. But when you come to scientific theories, it depends what you think a scientific theory is. Is it really a fact about the world or is it just an adequate explanation? It's useful to assume for a while. So I I'm very cautious about using it in those circumstances where it's used to put probabilities on on things that are not empirically verifiable. So I like to use it, I'd like to think is being used on facts about the world, verifiable facts about the world, or that might be verifiable in the future, not more. I don't like it using it for more abstract ideas.

Harpreet: [00:49:51] About using it to help us in our everyday lives to make better decisions. How can we use Bayes in that context?

David: [00:49:59] Well, we do it all [00:50:00] the time. We've got Bayesian brains. We're doing it whether you like it or not. Our brain, our brains are constantly everything's Bayesian in the brain is that it doesn't rely on data. It doesn't rely on what we're seeing in front of us. You know, what we're seeing in front of us merely serves to adjust what we expect to see in front of us. If we had to constantly just see everything and you know, we just couldn't cope with that with that overload. So the brain is we are Bayesian, that's it. We update our expectations of what we see. And that allows us to navigate the world just like just like a self-driven car then that they're all Bayesian completely. They all got a model for how the world works. They see new bits of new data and that revises what they believe. So and they all they have to work that way, but we work that way. So that's what we do. And we may not do it very well. We might have delusions, we might have not see things well. We can be misled in all sorts of ways, but that's what we're trying to do.

Harpreet: [00:50:58] So we've got just a few minutes left. I want to ask a couple of formal questions before we jump into just a real quick lightning round. Are you going to stay around for about 5 to 6 more minutes? Is that okay?

David: [00:51:07] Yeah, fine.

Harpreet: [00:51:08] Awesome. So there's this quote that I absolutely love from the Professor Risk video. I would love for you to to elaborate on this. And it's one of the biggest risks is being too cautious, I guess.

David: [00:51:21] Yeah, absolutely. And that's what I taught school kids quite a lot. And I always say, yeah, take risks, take risk, don't be reckless, don't be stupid. You know, think of the consequences of what you're doing. But if you don't, if nobody takes any risks, oh, for God's sakes, what a dull life we're going to have. You know, you won't get out of bed. That's pretty dangerous as well. So, you know, no, life is a risk. We don't know what's going to happen. Thank goodness. God, this may be awful if we knew what was going to happen. So. No, no, no. There's we've got to take risks in all ways. And I'm not just thinking of physical risks. There's so many other risks we take in terms of friends and jobs and enthusiasms and just yeah, I, [00:52:00] I'm quite cautious in many ways, but also I do try to be as bold as possible. And because that's the only way you learn, usually by failing, but then you learn. So boldness, not recklessness, can't be reckless to be stupid. So I won't get on a motorbike, for example. I'm pretty scared of that, but I will do most other things. And so I got my own little and we've all got it. You know, I don't regard it. I don't respond to risks in a completely rational way. It's all my emotions come into it massively. I'm sure I'm not completely rational about lots of things I do, but I do genuinely feel that you just, you know, if you're going to experience, you're going to learn, you have to try new experiences. Yeah. And that's one when I talk to school kids, I say that, but just don't be stupid. Don't do things, you know, that could have really serious consequences. Unless you. Yeah. Yeah. Just be really, really careful about things that have got big downsides.

Harpreet: [00:53:01] Yeah. I mean, yeah, you got one life on this planet. Why not try to do something big, right? Why not?

David: [00:53:05] Yeah, yeah, yeah, yeah, yeah, yeah, but. But, but make sure you got some insurance, I think. Protect yourself against a big downside.

Harpreet: [00:53:15] Yeah, yeah, definitely. I used to be an actuary for a while, so just make sure you're not paying too much for the premium. So last formal question before I jump into what I like to call the random round, it is 100 years in the future. What do you want to be remembered for?

David: [00:53:30] Oh, goodness. Oh, Chris, I want to know. I don't nobody will remember remember me at all. And I think, oh, that's really good. Oh, I think somebody who in a way one of the developers, what am I call performing statistician, they're trying to put statistics onto a public stage and realize that this involved an element of of acting, a performance of personality coming into it. And I think I just thought of that. I'd never thought of that [00:54:00] before. But I think that's rather than actually what might do is remember some of the statistical stuff I do, which seems to be surviving quite well in terms of citations. I still getting vast numbers of citations every year. So so the work might continue, but in fact, that's not what I care most about it.

Harpreet: [00:54:17] Step into a random round here. First question is what do you believe that other people think is crazy?

David: [00:54:24] Sorry, what do I believe. Oh, with the other. Oh yeah. Oh God yes I did see that. I thought I don't know. Oh I've got quite a lot of crazy things, I've got crazy friends and I quite like crazy things. So Reiki treatments where people just put their hands on and don't even touch me, have their hands on me. So I like that. I know what other things I do, but I'm, I'm fascinated by, by religious ritual and I will take part in religious rituals. And I think they're interesting and I've got a lot of respect for them and really no real strong, probably bonkers stuff.

Harpreet: [00:55:02] What do you what are you most curious about right now?

David: [00:55:06] Oh, what am I curious about right now? Oh, I think the thing that fascinates me is about misinformation. I think it's about how it spreads, what we might do about it, particularly statistical misinformation. Because I just I have a real fear. It's my big fear, I think, with the breakdown of normal media and normal social or social relations, just how easily people are influenced by influences and by by bonkers ideas. Now I've got my bonkers ideas, which I'm quite fond of, but there are some, but I don't think they're very dangerous and there are a lot of really dangerous, bonkers ideas out there and they seem to be taking spreading. So I think that's what I'm curious about is what actually can be new. How can we try to slow that down?

Harpreet: [00:55:55] What are you currently reading?

David: [00:55:57] Oh, well, I read terrible things. I know I'm reading [00:56:00] a great book on luck hasn't come out yet by guy so and he's a gambler. He's a serious poker player. So nice book on luck and I'm fascinated by luck. I dunno what I think is quite a good radio program. On Luck I learned so much from a philosopher, Angie Hobbs. She's a philosopher of luck. Oh, good luck for different types of luck. And I was brilliant, really.

Harpreet: [00:56:23] The four different types of luck is it from this book Chase Chance and Creativity, where he talks about luck, type one, type two, type three, type for.

David: [00:56:30] There's oh, I don't know, maybe. No, she had a different list, but the one I really like was the constitutive luck, which is just the luck in who you've been born at, which I think is just brilliant idea because it's one it's one of which one has absolutely no control whatsoever because lucky think of it something you have outside of your control. But the biggest thing is so your control is who you are, where you where in time and space and family and you have been born. And because I think I've been hugely blessed with that which dominates so much, and I think that's it. And you certainly can't give yourself any credit for who you are. So I think that's fascinating.

Harpreet: [00:57:14] That's I definitely would want to check that that book out. Yeah, because I'm also fascinated by by luck as well. I mean, that's data scientist of sort somebody who loves probability theory. In this book he talks about the four types. There's type one luck is dumb luck. Blind luck where you have no control over it. Yeah. Type two luck is the luck that happens to you just because of your own actions and things.

David: [00:57:38] Yeah. You put yourself. Yeah. I mean that's why you put yourself in situations where you allow it to happen. You make your own luck.

Harpreet: [00:57:44] Yeah, exactly. Yeah. Yeah. And that was the Winston Churchill quote. We make our fortunes and we call it fate or something like that.

David: [00:57:54] Yeah, yeah, yeah, yeah. Which is not completely true at all, because it depends who I mean. I think I think [00:58:00] constitutive like who you're born as dominates.

Harpreet: [00:58:02] And just that who's the name of that that author. That's right. In that book that you're Angie.

David: [00:58:07] Angie Hobbs. Anyway, the link is a radio program. It's got archive on forum, I think. And I interviewed the poker player as well, whose book I'm reading.

Harpreet: [00:58:16] So I'll definitely look into that. Yeah, I want to, uh, want to definitely check that out. Let's, let's go ahead and open up a random question generator. Get a few questions in here. Here we go. First question. I love.

David: [00:58:28] It. All of the dark. Yeah, the question.

Harpreet: [00:58:33] What do you like most about your family?

David: [00:58:36] Oh, I love them. Well, I know a couple of things. First of all, they put up with me and and that's the best thing of all. But actually, I the humor, I think, and the humanity. But the the humor is very important. Yeah.

Harpreet: [00:58:53] What was your best birthday?

David: [00:58:55] Oh, goodness, that's incredible. It's probably when I was six or I was 65. Yeah, I think actually the one my 65th, I had a wonderful party in a friend's garden and we had entertainers and music and it was it was just lovely. It was just lovely. I quite like being old. I quite like being old. That's the point. I don't mind at all. Oh, goodness me. Well, that's such a good question. When was the last time you changed your opinion about something major? Oh, no, that's very difficult just to think of. Oh, COVID. Covid. Oh, God. Yeah. Last March. Oh, I was. I was on March 2020. Oh. Oh, poor Flo. Why? What's all the fuss about? It wasn't quite that bad, but really, I was deluded. I know I should understand exponential growth, but. No, I was thinking. Oh, it'll. I don't know. I don't know what I was thinking, but I certainly wasn't keen on me. People I thought were fear mongering about how awful this was. Must do something. No, they were right. [01:00:00] No, I had to change my mind, so I was totally deluded. I'm glad nobody was listening to me. I mean, I didn't say anything in public, but I'm glad.

Harpreet: [01:00:09] And for those listening, Professor Spiegelhalter has another book that his most recent book is called COVID by the Numbers. So definitely check that out. Haven't been able to check that one out yet, but once it's released in Canada, I'll definitely be taking that one. So Professor Spiegelhalter, how can people connect with you and where can they find you online?

David: [01:00:26] Oh, yeah. I mean, on Twitter is handle a D underscore speaker. So I'm easy to find on Twitter. I got since COVID I got my followers have increased rather a lot and you can easily find my email address and my website. Just Google the name there's not not that many active Spiegelhalter is around because it is a stupid name.

Harpreet: [01:00:49] And those listening. I highly recommend the book The Art of Statistics. Absolutely love this.

David: [01:00:53] To the plug.

Harpreet: [01:00:55] Absolutely. Thank you so much for taking the time out of your schedule to be on the show today. I really appreciate you being here.

David: [01:01:00] It's a real pleasure. Honestly, it's it's not often you get a chance to ramble on about your enthusiasms for now. So thank you very much indeed.

Harpreet: [01:01:09] It is my absolute pleasure. And my friends remember you've got one life on this planet, so why not try to do something big? Cheers, everyone.

David: [01:01:15] They might be bold, but don't be reckless.