PRE-ROLL: Whether you're working on a personal project or managing enterprise infrastructure, you deserve simple, affordable, and accessible cloud computing solutions that allow you to take your project to the next level. Simplify your cloud infrastructure with Linode's Linux virtual machines and develop, deploy, and scale your modern applications faster and easier. Get started on Linode today with $100 in free credit for listeners of Greater Than Code. You can find all the details at linode.com/greaterthancode. Linode has 11 global data centers and provides 24/7/365 human support with no tiers or hand-offs regardless of your plan size. In addition to shared and dedicated compute instances, you can use your $100 in credit on S3-compatible object storage, Managed Kubernetes, and more. Visit linode.com/greaterthancode and click on the "Create Free Account" button to get started. REIN: Welcome to Episode 216 of Greater Than Code. I'm your cohost, Rein Henrichs, and I'm here with my cohost, Damien Burke. DAMIEN: Hi, and I am here with our guest today, Professor Julie Shah. Julie Shah is an Associate Professor of Aeronautics and Astronautics at MIT and director of the Interactive Robotics Group, which aims to imagine the future of work by designing collaborative robot teammates that enhance human capability. She is expanding the use of human cognitive models for artificial intelligence and has translated her work to manufacturing assembly lines, healthcare applications, transportation, and defense. Before joining the faculty, she worked at Boeing Research and Technology on robotics applications for aerospace manufacturing. Professor Shah has been recognized by the National Science Foundation with a Faculty Early Career Development award and by MIT Technology Review on its 35 Innovators Under 35 list. Her work on industrial human-robot collaboration was also in Technology Review’s 2013 list of 10 Breakthrough Technologies. She earned degrees in aeronautics and astronautics and in autonomous systems from MIT. Julie, welcome very much to the show. JULIE: Thank you so much for having me. REIN: So Julie, you were pre-warned about this. We ask all of our guests the same question to start the show. What is your superpower? JULIE: Yes, I don't think of myself as really having superpowers. I've mostly just sort of worked really hard [laughs] to do the things I've done. I'm an MIT professor, maybe I shouldn't say this, but I don't think of myself as especially brilliant; I just work really hard and I'm really passionate about what I do. I do think of myself as really especially good at is I am very at maintaining focused attention on something for a very long period of time and that got me pretty far in life. That got me through a Ph.D. because mostly you just sit quietly at a desk as an AI researcher and program, on your own, really, really focused. Then I became a faculty member and was quickly told and taught that that job is totally different. It's a juggling job. If you look at any one of the balls too long, the whole thing falls apart. So over the years, I have trained myself to juggle, but really at heart, my superpower is just focusing on one thing for a really long period of time. REIN: Okay. So then the second half of this question is how did you acquire it? Do you think that was nature or nurture? JULIE: I'm positive, it was nature because I have a 2-year-old and a 4-year-old and my four-year-old is exactly like that. He was just born that way; my 2-year-old, a little less so. I think it's just a personality thing, but who knows? REIN: I have ADHD so I am also like that about 10% of the time. [laughter] DAMIEN: I have to say, I really love that answer. What I heard was, “I don't have a superpower, I just work really, really hard.” I've read enough comic books; that's a superpower. JULIE: [laughs] There you go. REIN: Yeah. It's not what I have so I can agree with that. That's super cool. That's the main thing that I know about you is that you're into robots. JULIE: Yes, yes, I am and I mostly feel that same way every day, too. Pre-COVID, going into my robotics lab, I can't believe this is my job, this is totally amazing. I think one of the outstanding things about being a professor is that my whole job is to envision the future and then try to make progress towards realizing that future. I work in AI. I'm an AI researcher and a roboticist, but I specialize in developing the artificial intelligence and the software. I've worked to deploy collaborative robots that work alongside people in factories to build planes and build cars and work on translating what we know about effective human teamwork into robots so that they can sort of jump in and work alongside people in an equally natural way. DAMIEN: Okay, that sounds scary impressive. In my experience, robotics and manufacturing, for instance, mostly collaborate with humans by following a very careful script instead of behaviors that the humans are aware of and can work with, but it sounds like that's not what you’re doing. JULIE: That's exactly the opposite of what I try to do. Carefully scripting the robot has its place. There are industries in which we have successfully, widely, deployed robots, but the automotive industry isn’t one of those industries. Most people think about in the process of building a car, it’s this long line of robots. it's building up your car, but it's actually only half the build process of a car that is really done by robots and it was that half of the build process that could be carefully scripted and is highly repeatable. But the final assembly of the car like the wiring, the cabling, the installation, it takes up half the factory footprint and half the build schedule. So about half the build process and it's still done almost entirely manually today. It's a lot of work there that is still very, very hard for robots to do, but that's only because we can't sort of cage it. We can't pull out the work, separate it from the human work, cage it, and structure it. What I aim to do is to think about rather than replacing the humans’ role in doing that work, how do you develop an intelligence system to assist or augment a person in doing it? Once you sort of take a step back and say, “Well, our end goal is actually just building a car more efficiently, more increasing the productivity,” then another concept is to deploy a robot that's almost like a surgical assistant to the human associate doing the job that's still very hard for robots today. If that robot can just hand over the right part at the right time and save a little bit of that walking distance between the cart with the material and the car on the assembly line, that non-value-added work that we don't think about is actually a very large portion of that build process. A robot that can help a person paralyze tasks and really be like that surgical assistant can drastically improve the production of a build process. But it requires a system that you can't script a robot to work with a person, as we learned over the better part of a decade, because people do things differently from person-to-person. But also, from day-to-day and shift-to-shift based on their personal preference, based on whether they have to hurry because that part of the car got to them a little late. So it really requires that a robot work more flexibly with people and that's the AI I focus on is actually enabling those systems to more flexibly, change their robot programming or their plan in response to what's happening in front of them while working with the human. DAMIEN: I'm going to ask a question that sounds very simple and I understand that it is the bulk of your life's work. How do you create that flexibility? JULIE: Yes! Okay, so! REIN: Just summarize your dissertation in about 5 minutes or less. JULIE: Okay. No, it's even worse than my dissertation because I've been – now, if you don't pass the dissertation, I've been doing this for the better part of 10 years, but I figured out how to summarize it briefly for you. So what I can do is summarize decades of research and human team collaboration and coordination and human team training like, everything we know about how to train pilots, to work effectively with co-pilots and a cockpit, or to train nurses and surgeons to work together, or to train some military teams to work together. There's a whole science behind that that we've been working to be able to translate to develop computational models for robots that can learn with people and be equally good teammates. I'm going to have to retire this joke now, but I'm going to do it maybe one or two more times, then I'll retire it. But I can summarize all of that work, those decades of research with usually in my [inaudible] Tom Brady and I say he is an example of an outstanding team leader because I'm in Boston. So there you go. It'll have to be retired soon, but it's okay. They say he sees and he knows and that's what you need to know about what makes an effective human partner. You need to know what your partner is thinking, you need to anticipate what they'll do next, and then you need to be able to make fast changes in response to disruptions that occur. The AI that we work on developing for robots is they do exactly that to infer human mental state to be able to develop predictive models of how people will behave, and what they'll do, and the timing of those actions, and then take that into a techniques for dynamic tasking and scheduling. So the robot can very quickly sequence that work, to choose different motions, or do different parts of tasks based on what's actually occurring online. DAMIEN: So I'm not explaining this, Julie, this is more for our listeners, but this combines decades of research in a huge number of fields like cognitive systems, engineering, human and machine cognition, joint cognitive systems, human factors and ergonomics, just a whole bunch of stuff has to be synthesized to make this work. JULIE: Yes, exactly. You got it. Yes, it's a number of different fields that are inputs to developing systems that are capable in this way and we're only at the beginning of the team science of human and machines working together. One thing that my coauthor, Laura Major, and I raise in the book is that to get this right, it's not just about making a system that's more intelligent, that's more capable, or more human-like. It requires systems and processes to change what we know about robots, what we think about them, how we're going to learn over time to interact and collaborate with them, and what we're aiming to do is translate those insights. Those sort of hard-earned lessons from aviation, from designing a partnership between a pilot and an autopilot, or from the industrial sector, where I've worked in collaborative robots, and figuring out how we can take those insights and make robots in our homes, on our streets, on our sidewalks, in our workplaces more capable and be able to provide value for us. DAMIEN: Wow. So one of the things you said that this requires, this incredibly impressive work, is a mental model of the humans you're working with. At that point, it sounds like you've solved artificial general intelligence. JULIE: Well, yeah. Okay, so. [laughs] I have not solved artificial general intelligence, just as a spoiler alert. In contrast, rather than thinking about these systems as sort of artificial human intelligence, a key unlocking point for me was to – something we've thought deeply about over the years is, well, what are the natural strengths of humans versus machines or humans versus AI? It was recently pointed out to me, although, I never actually made the connection, the Turing test. The original test, how do you know if something is really artificially intelligent? You want it to pass for a human. My lab has used or thought about variants of that type of Turing test in a team setting. In computer environments, we've paired virtual agents with humans to play games or to compete, and then we ask the human, “Were you playing a human or were you playing an AI?” Really, what you want is for them not to know, right? You want it to be random chance that they get that right. But the very conception of that Turing test is you're aiming to replace the human’s role. You're aiming to look at what a human is doing, pull a human out, and put an agent in. In many ways, that's missing the point of the complementarity between humans and AI or humans and intelligent robots. The way I think about it is unique human capability is our ability to take an unstructured problem and structure it. Machines can't do that. That's our unique human ability. Once we've structured and defined a problem, AI or a machine can crush it. The question we have to ask is, well, what are all the ways restructuring the world for AI now, or for intelligent robots now? If we're not structuring the world for them, or we're doing it in ways that are sort of implicit well, then let's open up that design space and let's think about okay, well, how do we structure the world for these machines so that they don't just do the things that are easy for them, but that are actually valuable for us? Oftentimes, the way you change machine learning today is through labeled data and that's how we structure. A machine's knowledge of the problem in the world is through us painstakingly labeling data for it. That is a very poor way of translating our unique human insight [chuckles] on how to structure an unstructured problem and so, what are the ways we can expand that and create the sort of richer communication between human and machine? DAMIEN: I'm sorry, I have very strong opinions about AI and I really, really love this. I'm really glad I got to be here for this. One of the things I like to tell people is the reason we don't have machines that think like humans is because we don't want them. Do you know how humans think? You don't want your computers to do that. Occasionally, sometimes on accident, we get that anyway and nobody likes that. The other story I like to tell is the best chess players in the world are not people, the best chess players in the world are not machines, they're people-machine collaborations. It makes my heart expand two sizes to know that this is the work you're doing. So thank you. JULIE: No, that's exactly right. That key that the two together can achieve more than either can individually. Building off of that comment, in my lab, we’ve seen that over and over again. Especially in intelligent decision support, there are many tasks that we can't really in code for machine to solve on the timescale that say, like a military operator, or a nurse, or a doctor would need a solve in say, running a hospital unit. It's actually very few of us as humans that can do it that well. So for example, we had a project in a local Boston hospital where we were developing a system to support the resource nurse or the nurse supervisor that runs the labor and delivery floor. When we looked deeply at the task they were doing, the job they were doing, it's actually that of an air traffic controller. It's actually computationally more complex than that of an air traffic controller and they do it without any decision support. There's no structured training process, there's no manual or rule book, there's even no objective they're given like, here's what we want to achieve on this labor floor in terms of utilization of beds or other measures. Some people are naturally very gifted at solving those types of air traffic controller problems and some people are not. But if we use machine learning to reverse engineer the rules or heuristics the people use, these high-performers use, and we give that to a machine to see the optimization process, the machine can find a solution, the computer can find a solution very quickly. That's even better than the solution a human expert would find. This is just another example of from chess where you were saying, together, if we intelligently leverage our relative strengths, we can achieve a lot more. In fact, if that's your end goal, you might actually want to design an AI system, or a predictive system, or an automated system that's actually not perfect. In a safety critical context, say like a TSA agent looking for a threat in your luggage or a doctor looking for cancer on a scan. If the system is too good and if say, 90% of the time, it's giving the person the right answer. Well, why do you have the person in there? The person's in there to catch the 10% of cases that the machine is going to mess, but there's these natural human biases [chuckles] that we bring. We are not perfect. We are not necessarily the thing that you exactly want to emulate, even just as you were saying. One of these biases comes up in that a human will over rely or over trust the recommender systems, the system that's recommending. Is there a threat here? Is there cancer here or not? Not exactly in those applications, but in some control studies, researchers have shown that there's a sweet spot. You actually want the system to be something like 70% accurate at its prediction, because it keeps the person and the person is better able to catch the situations that the machine fails at such that the human-machine team overall performs better than either the human would have on their own or the AI would have on their own. So by suboptimizing the AI, you can better optimize the human-AI team. Isn't that interesting? DAMIEN: No, that is brilliant. REIN: Yeah. There's so much going on here. The last thing that you just said is fascinating to me, which is that by suboptimizing the AI, you can make the system better. Russell Ackoff has a saying, which is, “You can't make a system better by improving each part separately and in fact, sometimes you have to make specific parts worse to make the system better.” JULIE: Exactly. Yes. In that long bio you read for me—now I'm going to make you include the sentence of it. But in that long bio you read for me, I’m faculty in the Department of Aeronautics and Astronautics at MIT. I moved into computer science and AI for my Ph.D., but my undergrad, I studied aerospace engineering—and that's still my home department. One of the things we teach our undergrads in aerospace engineering is exactly that. So you can optimize each individual component of the system, but end up with a highly suboptimized overall system. We illustrate that to them in the context of these capstone courses, where they have to design a satellite or a Mars Rover, and then there's teams that breakout like this one does avionics, this one does the materials and structures, this one does the thermal. Each team on their own optimizes their own sub system and then they put it together and it's a disaster. You have to sort of work through, it's an interconnected system and to choose something that's going to work, you would need to make these trade-offs. AI on its own is not the goal. We want it to be. Intelligent robots on their own are not the goal. We want them to be embedded in society and in helping us in the way we do work and so, that requires different engineering and design considerations. DAMIEN: Can I ask a favor? Can you go to Tesla and explain this to them? Because these people who are building Level 4 self-driving systems are going to kill a lot of people just because again, they're heading towards exactly what you described a system that works very well, almost all the time. JULIE: Yeah, and one of the key differences between the Tesla or a car and our roads and an airplane actually – okay, there's a few key differences and it's just going to horrify us all even more. A pilot is trained not just to how to fly the airplane, but on the automation in that airplane for thousands and thousands and thousands of hours. Highly trained, right? Driving, where we got a license to drive the car, but we're not trained on these new automation systems that are being deployed and certainly, not at that level. In an airplane, you have thousands and thousands and thousands of feet to troubleshoot your problem, a long time, minutes to troubleshoot your problem. When something isn't going right, the pilot is actually taught to slow down. Slow down, take your time, and understand the full state of what's happening and then to be able to take the right action. Whereas, in a car, we're putting the person in as a safety net with automation that will definitely fail because it's not trained, it's not validated on every possible scenario that we're going to see on our roads, and then we're asking, in a split second, for a person to be able to jump in and be that safety net. Meanwhile, there's good studies that it actually takes a person 7 or 8 seconds to be able to safely transfer that control authority from your vehicle and to build up your own mental model to be able to take over and take the right action. But we would never be given 7 or 8 seconds on a road and in most of these circumstances. There's a lot here where we're setting ourselves up for failure, in a way. A lot of the lessons that resulted in plane crashes over decades around weaknesses in humans, understanding our human situational awareness of the aircraft, or in mode confusion; that understanding what mode it's in and then taking the wrong action in response to that we're seeing in these new vehicles on our road. There's a seminal researcher that did the research on pilot situational awareness and the root issues that result in many plane crashes where a pilot doesn't know what mode the system is in and takes the wrong action. Just maybe 2 years ago, she published a beautiful journal paper where she bought a Tesla and she said, “The software updates on this Tesla regularly, whereas, an aircraft, it doesn't.” Every time the software updates, there's huge training around it when her car is updating software relatively regularly. She self-documented issues of mode confusion in her own car over the course of a six-month period and it was about a dozen or so incidents. It points to the issues that are sort of key threats to safety and aviation are cropping up again in these vehicles and there's things we can translate, but there's things that are working quite against us in terms of timescale, in terms of training of people on the automation itself. Who's going to sit in their garage and undergo 30 minutes of training after each software update before they go to work? Nobody would stand for that. So it's something that we need to build into the design of these systems, these human feelings or weaknesses. We're good at other things, though. REIN: You mentioned mode confusion. This is also called mode error. For our listeners, the original exploration of this was there was a plane that had two levers next to each other; one lever controlled the flaps and the other level controlled the landing gear and those levers were the same piece of hardware. They looked the same. Can you imagine how these planes were crashing? That's how they were crashing. JULIE: Yeah, and highly-trained and highly-skilled, but under duress, under stress, people revert back to their previous mental models. There's also an example of as the model of airplane shifted, there was a change in where those levers were and pilots have reverted back to their prior training and the previous aircraft. Yeah. REIN: This also gets back to what we were talking about earlier, which is there's been a paradigm shift and originally how we thought about automation and now, in your work. This actually started happening decades ago. So the old view from the 40s is that you make the machines, do the things that machines are good at, and you make the humans do the things that humans are good at. Machines are there to replace humans at stuff that humans are bad at. But the newer view is typified by a paper, which was “Ten Challenges for Making Automation a ‘Team Player,’” which is how do you make machines a team player in a joint activity? You do that by figuring out how machines are constrained. They're good at understanding things that are within their ontology, but not good at understanding like the larger context. So you get people to point them at the stuff they should be looking at and then you figure out what humans are good at, but where they need help. They're good at understanding context, but they make machines to inform them of new events. JULIE: Exactly. REIN: So that's the kind of paradigm shift. JULIE: Exactly. That's exactly right. Yes, and I think what's interesting is it reframes the AI problem. If we are narrowly defining the AI tasks, the tasks for AI, it requires a different problem framing to be able to design AI for that type of use case. It's not just a human interface problem is the thing. On the other hand, it isn't just an AI problem either. There's a lot we can draw from our prior expertise in human, our prior knowledge in human systems integration that's really not that different. So when we introduced a radar in World War II to be able to detect submarines and there were differences in how you want to tune that radar based on your end goal or who the operator was. There's high consequence if you think there's a submarine there when there isn't, but there's high consequence. Is there a submarine there and you don't see it? Turns out, you want to tune that sort of threshold based on the operator's skill, or expertise, or even their fatigue level. That spun the area in psychology, signal detection theory, that now influences how we design threat detectors for TSA agents. All of that still applies. Just because it's fancy AI, it doesn't mean that doesn't apply anymore. So yeah, you're exactly right. I think getting this right is our key challenge right now, but it's also, the really big opportunity. REIN: Are there things about robots specifically that present new challenges compared to the work on automation in the 90s? JULIE: Yeah, yeah! REIN: And how did those differences play out? JULIE: Yeah. So there are ways that it's different. One of the things that's more challenging is we look to bring the newer, deep learning techniques into work products, into applications, but also, into intelligent robots is it's no longer easy to understand how or why the output is the way it is. These models are inherently uninterpretable. Whereas, before, if we have a closed loop control system, that's something that we can characterize very, very well, mathematically. We have a rule-based system, that's something that is deterministic and that makes it amenable to analysis. REIN: You can open up the box if you need to sort of a thing? JULIE: Exactly, exactly. What are the issues with that that might be similar or different? So one is when we're using these systems and we're relying on them, if they're providing us a recommendation, or an output, or a prediction that then we're using in some downstream process, then a key task for us is to be able to calibrate when we should rely on that output and when we shouldn't. Without an interpretable model or a model in that machine learning system that corresponds to our human mental model, we're undermining our ability to do that. So as a concrete example, this is an experiment we conducted in the lab, but you someone directly controlling a UAV and scanning a video feed for threats or concerns in the environment. If the system in two or three experiences makes a mistake in alerting you to some threat or obstacle in the environment and it's dark in that image, you as a person will say like, “Ah! Okay, this system makes mistakes when it’s dark because I make mistakes when it's dark and I can understand why the system would be failing.” But meanwhile, that system is using an entirely different sensor suite than we have as people. Very possibly and the underlying model [chuckles] is inherently uninterpretable. It's like, you don't really know what feature or what combination of features resulted in it to succeed or to fail. The problem is that we as humans will very naturally, very quickly build some mental model for the explanation of the behavior of a system. In the absence of being told, we're being trained on how the system is going to behave in different envelopes or different regimes, we will build that mental model. But it undermines our ability to know when to rely on it or when to not rely on it in a new context. So this problem of supporting the person in building some calibrated trust in the system becomes much, much harder. In terms of human systems integration view, that's something that's fundamentally different. DAMIEN: So it sounds like one possible solution to that is to build computers that model things the way people do, but we already established that that's also a problem. JULIE: Yeah. Well, so in terms of a generalized approach to modeling things the way people do, I mean, I'll never say never. I don't know. But I agree with you like that really probably should not be our goal for many applications or in many settings. However, machines are fundamentally different from us, but that's not to say that there isn't a place for changing the structure of the machine learning model so that it better corresponds to our human mental model. So I'll give you another example from very recent work from my lab. I'm originally from New Jersey. In New York, the subways run uptown and downtown and the behavior of that subway is to go to the end of the line, turn around, come back, go to the end of the line, turn out, and come back. But in Boston, the subway is run inbound and outbound. That's how we named the movement from some arbitrary central point Park Street. So if you're going towards Park Street from any external end, you're going inbound and then once you cross Park Street, you're going outbound. But based on my external observation of the two subways, what I see is exactly the same behavior, but the model I hold in Boston is different than in New York. Corresponding those miles could be important for generalizing the behavior of the system or in this case, just communication; being able to communicate the state of the subway to someone else in that community. But what I could do is – so this is what we did in the lab. Instead of taking an unsupervised approach to trying to learn the behavior or the state switching of say, the subway or some model of more complex human, if I just tell you when that state changes and I say the change point is at the end of the line or the change point is it Park Street, I don't have to give you any names for the behavior of the system. But I've provided a way to incorporate structure about the knowledge of the world that I hold as a human that can allow the system to learn a structured model of the world that now corresponds to my model of the world. So we're still sort of leveraging this semi-supervised approach. We don't want to hand label absolutely everything because that's super hard for us to do, but we don't want it to just learn some random explanation for the behavior we’re seeing, because actually there's physics in the world, or there's some mental model we hold as humans that's useful for communication. We actually have preferences over the structure of the model that it's learning. How do we give it just the right foothold to sync those models, to align those models to our human mental models without this very laborious process of labeling data? We don't need generalized intelligence, what we need is the ability to strategically open these lines of communication between a human and machine so the system can lock into models that are useful for us for specific tasks at specific times. So the person in New York or the person in Boston could use the same AI model, but sync it to their own mental model in a way that's useful for them as they need. REIN: Let me see if I understand the challenge with that model. In the model, there's a discontinuity. At one point, there's a vector pointing this way and at some point, it flipped signs to point this way. But the underlying reality, the actual subway car just keeps moving continuously in the same direction. Does that sort of? JULIE: Yeah. So the mental model I hold of the behavior of the subway is not linked to its velocity vector. That's what you're saying. The way I talk about where the way the subway is going, it’s [inaudible] linked, but in another one, it's not. Yeah. REIN: Yeah. At the point where it stops being inbound and starts being outbound, that's a discontinuity in the model, but actually, the thing just kept moving in one direction. JULIE: Exactly. That's right. Yep, yep. Which I found moving from New Jersey to Boston, many years ago, very confusing. So we don’t need artificial general intelligence, we just need a way for the system to follow our breadcrumbs and be able to learn in a way that's useful for us in some particular context, for some particular kind of task. That's my view. We can accomplish a lot with that without waiting for or being concerned about generalized intelligence. REIN: Okay. I got it. So I think what the team paper says about this is important because the challenge is relevant here is adequate models. JULIE: Yes, yes. REIN: Not perfect models of humans. JULIE: Adequate models. REIN: Models that are suitable for the task. JULIE: That's exactly right. Yep, absolutely. DAMIEN: I want to switch change directions here because there's a topic you suggested that I fear we're not going to get to when I really want to get to it. Can you talk a little about the interaction of robots and humans in public spaces? JULIE: Yeah. So this problem is fascinating to me because it brings almost everything we've discussed so far kind of together. I think the autonomous vehicles on our roads, it's just the first example of the type of robot that's going to be all around us, on our sidewalks, in our office places, security guard robots, the news stop and shop grocery store robots that are going up and down aisles looking for spills. These systems are equally dangerous as autonomous vehicles just to motivate why this sort of pedestrian type robot is so important for us to think about it and get right. A few years ago there were headlines, I don't know if you remember. These always end up being kind of funny headlines, but they're not really funny. A security guide robot in a Palo Alto shopping mall collided with a toddler. That's like a 300- or 400-pound robot like, that can kill a toddler and that's equally tragic as a car killing a pedestrian at an intersection. So why did it run into the toddler? The toddler did not – as most kids do, they do not conform to the human model you would build if you're looking at adults. The toddler, instead of moving out of the security guide robot’s way as it was approaching, as the security guard robot would had expected, the toddler went towards it and then went towards it faster because the toddler was excited to see the robot and wanted to get up close to it, not really thinking that it would pose a safety hazard. So this idea of who's interacting with these systems and what training do they have is made much worse by these robots on our sidewalks and on our streets, because now it's not even a person driving a Tesla that has opted in to drive the Tesla. It's bystanders who are really just trying to go about their business and their work interacting with these robot system and they hold a mental model of the robot. That toddler held a mental model of the robot that it's like a friendly robot from TV and it could go and say hi, who knows what that toddler thought. But actually, the mental model that toddler should have had is that is a very dangerous robot and it's unpredictable [laughs] around you and you should give it space. REIN: Challenge three, predictability. Human age and team members must be mutually predictable. JULIE: Must be mutually predictive, yes. Directability is equally important; being able to direct the behavior of the robot. We need bystanders to also be able to direct the behavior of their button, not just a supervisor and operator remotely overseeing it. A parent should be able to wave off that robot as their toddler is going to it. REIN: That's challenge four. JULIE: That’s challenge four. [laughs] DAMIEN: And to be able to do that without any sort of training or previous knowledge about the robot. JULIE: Yes, yes, and it becomes pretty nightmarish because each of these startup or each of these big companies—there are a number of different manufacturers of these robots. So while there's been AI challenge to make the system more capable of distinguishing toddlers from a person or me pushing a double stroller from someone else riding a bike, it's equally important we think about the other ways we structure our world for these robots; for safety reasons and just to make them more capable for us. Let's see so for example, when we rent cars, the standardization for the controls for driving a car, we rely on that to be able to rent a car at some new place and be able to just drive it without any training or looking at a manual. But with different manufacturers in an uncoordinated way, you're going to have different beeps, you're going to have different signals, you're going to have different ways that the robot communicates with people from manufacturer to manufacturer, a robot from this sidewalk, or this block to that block. It's really untenable for bystanders to learn the ways that they can safely direct the robot when they need to if the input to each of these robots is different. So standardization is another tool in our toolkit here to make these systems safe and easily integrate them. It's not fancy AI, but I think an argument can be made; it's arguably even more important. REIN: So these agents need to present signals about their status and intentions and humans need to be able to interpret those signals to understand what the agent is trying to accomplish and what it might do next. JULIE: Exactly. DAMIEN: So we've kind of had this problem before, right? We have trains and they're on rail, so you know where they're going to go and they have crossing guard. You mentioned subways earlier. Subway is another thing. I live in LA now; we go out to Boston, we go to New York, people haven't seen subways before like, what is this?! But they know not to step off the ledge, I guess, because there's a ledge there and then there's yellow paint in the front and there are the signs. So we've dealt with these issues with all sorts of industrialization and dangerous tools and it seems like infrastructure might be another tool we have when dealing with robots in public society. JULIE: That's exactly right. It is another tool. It's a key one, in my view. These systems are not going to be capable without us investing in infrastructure for them to understand us in the world. But in addition to that, there's also this question of how we want them to exist in our world. So for example, what is the mental model we want to hold up these robots? Are they bikes, do they exist in bike lanes going down our street because then suddenly, I know how a bike is supposed to behave and if these systems are behaving like bikes, that could work okay, or are they sidewalk robots and are they really more like pedestrians and that's the model we're going to hold of them and then they should behave differently than bikes, right? But then we can build a model of okay, this is how a pedestrian would navigate around me, this is what it would do. Or is this something where just we eventually installed our bike lanes in many cities for safety of people who are riding bikes, are they going to need to be something fundamentally different and need their own lanes or are they going to be more like cars? Infrastructure is a key aspect to making them usable, but there are also these larger questions of what is the right way to integrate them into society even and that flows back to have implications for the technology design as well. DAMIEN: That's such a great example. If you put a mobile robot in a pedestrian space, we’re going to expect it to behave like a pedestrian. We have expected to move our pedestrian spaces to yield or not yield the way pedestrian do. So just putting it in that space does that. But then there are also technical questions that I don't know a lot about the mechanics of robotics. How do we answer this question? How do we want our robots interact with us in society? JULIE: Yeah, it's all trade-offs, it's just like the designing the space satellite example. If the robot is going to be a pedestrian robot well, then it can move a lot more slowly and it can stop in a shorter timeframe, but there may be more variability. Sidewalks weren't designed for wheels. [laughs] There may be less predictable entities on the sidewalk with the robot that has to be contended with. Whereas, a bicycle lane, you may have a more limited set of types of interactions it would have at various intersections or going down the streets to work with, but it's moving faster so it has a differences stopping distance, it has different signals that would have to attend to know when to slow and stop. The design problem is deeply interwoven with the question of how we want to think of these systems and what's going to be better for us in terms of how they're integrated and it may be different from neighborhood to neighborhood or city to city. I think in the book, Laura and I, we originally set out to write this book as a textbook actually for roboticists and for AI researchers. That was our goal. We spent a while working it through like that and then came to the view or understanding that so many of these design issues are so interwoven with these larger societal considerations that we need to work with that it’s actually really important for everybody to understand that these are the questions we need to be asking that needs to flow back. We shouldn't just wait for the tech companies to stick a robot on our sidewalk. That's going to end well for nobody. We need to be a part of the discussion right at the beginning about what we want of these systems and how we conceive of them integrating effectively. So we aimed to write a more widely sort of accessible type of book. DAMIEN: Yeah. That just brings to mind like I've long thought about my pizza delivery robot. Why not have a pizza delivery robot? Well, a pedestrian robot couldn't get me my pizza fast enough and a car robot couldn't get to my door. JULIE: Yes! [laughter] DAMIEN: Do I need two robots to deliver my pizza more? JULIE: Yeah, yeah. That's a part of the design space, too. Maybe you prefer not to have robots gumming up your sidewalks, but then you need a different solution to do those last steps and actually get it to your door. REIN: Well, people beating the shit out of robots with a baseball bat is an externality. That's not something that we need to include in our model. JULIE: [laughs] I wonder if there are any examples of road rage against autonomous vehicles, yeah. REIN: Oh yeah, people are beating the shit out of robots with baseball bats. JULIE: Oh, yeah? [laughs] REIN: Yeah. JULIE: There you go. Yeah. [laughs] DAMIEN: There's also a great theory that because autonomous cars are very safety conscious, that human drivers will learn this and will bully them. You don't get to merge because you're not going to hit me. JULIE: Yes! Yes, yes, yes. So I, very unfortunately, have not yet had the opportunity to drive a Tesla, but what I understand is there is this knob you can tune it on sort of like the aggressiveness of the Tesla exactly for this and use the drive or tune it. You need to set it differently depending on where in the country you're living or maybe based on your own personal preference. Yeah. REIN: Yes, what is the aggression threshold that will allow you to merge in New York versus San Francisco? [laughs] JULIE: Oh, there's definitely a Ph.D. in that topic. DAMIEN: The [inaudible] implications are absolutely fascinating, right? JULIE: Yeah. [laughter] REIN: This is so much fun, I [inaudible] about this. JULIE: Then there's larger considerations, too or privacy considerations, how does that information then flow back and get used for your insurance premiums? REIN: It's also entirely possible to go back to what I was saying before that making the world's best automated car won't make society better. That improving that particular part isn't going to be the thing that makes society better. It might even make society worse. JULIE: Yes. Yeah. Definitely. For any technology you're developing that can have wide reaching societal implications. Small decisions along the way both, technical but also, social or policy decisions could result in very different futures. If there’s a world in which it’s safe and very, very inexpensive to be transported in an autonomous vehicle from one point to another, what about our societal investments in other mass forms of transit like subways? Does it completely replace that and over what time scale? Laura and I started this book obviously, a long time before a pandemic. I think even a year ago, we didn’t even – these issues, I think are imperative and are very urgent for us to be and considering writing out. I don’t think we considered that on a day-to-day basis, there would be an imperative like a safety imperative to be able to address risk and harm for essential workers. I think there’s no reason to say what if we had done things differently or invested differently, but let's do it differently for the next pandemic. Let’s make these technologies that are actually useful to us in our workplaces and in society to be able to address something like this that happens the next time around. REIN: Yes. Let's do that. JULIE: Let’s do that. [chuckles] REIN: Yes. JULIE: But there’s implication for future of work, too. Again, not just wanting to replace people or being thoughtful about the implications of deploying any new technology and there needs to be a larger discussion and investment in the long-term. I guess, my understanding is, I've been embedded in MIT’s Work of the Future Task Force for the last 2 or 3 years, which is this is amazing effort with economists and social scientists and roboticists and AI researchers trying to understand what's coming and how we shape technologies and how we shape the future that we want to see. We have a lot of opportunity to shape it, but my crude understanding is that long-term, everything is going to be okay. In the long-term, technology creates new jobs. But what about the short- and medium-term? There needs to be an active effort to make sure that people have livelihoods and that we make it through a time of disruption and so, that's an equal part of the conversation. DAMIEN: I think Frederick Koch has said, “In the long run we are all dead,” JULIE: Exactly. [laughs] One way or another, what will happen in the long-term so. REIN: There’s a particular facet to this that we kind of touched on earlier that I'd like to dig into a little bit more, if we could. I’d like to get back to the case of intersection with the bikes and the pedestrians and the cars and talk about joint activity. There’s an old, the classical model of communication, the Shannon model. Claude Shannon is I say a thing, you interpret that thing, you say a thing, I interpret that thing so there’s sequential blocking communication, but intersections are not like that. Everyone isn't waiting for everyone else to make a move; they're all moving at the same time and so, joint activities are activities where every agent is constantly acting, interacting, and perceiving, and doing and it’s categorically different kind of system than the ones that engineers have historically tried to build. JULIE: Absolutely. REIN: So the prompt that I’m trying to get at there is how does that impact your work? You can’t just build client server systems where your machine blocks waiting for other machines to act or humans to act. JULIE: Yes, that's definitely true. If you’re analyzing one of these systems in a multiagent setting, you'd ask all sorts of questions like stability, [chuckles] do people or entities get to where they need to and in some amount of time and so on. The thing that's really interesting to me about robotics or intelligent robotics is that systems are continuously taking in information, processing, making decisions, and acting. There's concurrency to it that clearly introduces another level of complexity. Actually, one of the classes I teach at MIT is real-time systems and software and so, building systems, concurrent systems, and concurrent programming techniques and approaches and modes of analysis. But on the other hand, that is the way the entire world functions. We want to design concurrent systems because it mirrors the concurrency in our real life and so, there are challenges, but they're not insurmountable. The thing is, we don't need guarantees necessarily. [chuckles] We have many ways to get around deadlock at an intersection; it’s not something we worry about actively with people, but with agents and with cars, it is something you worry about practically. So there was this really interesting story or at least, I found it interesting and it stuck with me. When autonomous vehicles were first being rolled out and tested—I guess, there's no reason to name a particular company whose car it was, [chuckles] but – REIN: Let’s assume they are all bad. JULIE: Let’s assume, yeah. A natural thing to do is to have the autonomous vehicle defer to a human at an intersection like, that seems like a very reasonable goal to put in place because the person will kind of sit there for some amount of time, wait to see what the car is doing, and then get impatient and then eventually, go and you've broken deadlock. But then who’d have thought these systems are being tested on the roads. Two of those cars came to an intersection at the same time and rather than the rules of the road where you defer to someone on the right, the rule was to defer to the other and then there was deadlock. The two cars just sat there waiting for the other one to go. That's clearly an issue and so, that was eventually addressed, but those were two cars even from the same manufacturer. Now you have different cars across different manufacturers and some explicit communication and coordinating among them and explicitly between people and these entities is absolutely necessary and it needs to be designed. It should not be an emergent system; that will not work very well for us either. DAMIEN: At some point, you should probably tell us about your book! JULIE: [laughs] Sure! Yes! DAMIEN: That’s not in the bio. You should add that to the bio, too. Make it a little bit longer. JULIE: I should add that to the bio, make it a little longer. Why not? [chuckles] The book touches on many of these questions and themes. While in my lab, I'm working to make AI more capable of modeling and collaborating with people who are very focused on making systems intelligent enough to be effective teammates with people. That’s only half or maybe even a smaller part of the equation or the considerations that are necessary for moving systems from an industrial environment onto our streets, roads, and workplaces. So the book is both, a look back at decades of research in these other industries to ask the question: what translates like issues of mode confusion, that translates. What doesn't translate, for example, designing safety critical systems that require operators to have thousands of hours of training, that doesn't translate. So the things that relied on there to make those systems work will not translate in these new contexts and thinking about the wider design space we have in terms of designing our infrastructure, outfitting our environments, and then thinking more at a structural level, too what's necessary to make these systems safe. Again, there is a lot that we can translate from aviation and other fields in terms of sharing of information between companies and through public private partnerships that can ensure say, a security guard robot at a Palo Alto mall that runs into a toddler is not a situation that occurs again on the East Coast. It’s meant to provoke some questions not just for engineers or technologists, but for us more broadly as stakeholders in society. Systems at point of scale, they're going to have wide-ranging impacts on all of us. Around an airport, whenever you're going to change the flight patterns, there's community input on that because the noise associated with that can impact property values. What is the equivalent here? Well, where are these systems tested? Where are they refined? But then, how are they deployed in practice? Which neighborhoods do they go through or blanket to succeed in your next day delivery target and which neighborhoods do they avoid? What times of day do they come through? These are questions that at a neighborhood and at a municipal and city level, we need to be involved in designing the solutions and it needs to be tailored for each individual community. REIN: So your book, which is, What to Expect When You're Expecting Robots: The Future of Human-Robot Collaboration, which is available in every place that sells books, I assume. It looks really interesting. I'm going to pick it up. I'm really fascinated by this topic, but most of my reading in this area is at least a decade old. So aside from your book, which people should read, what are the most interesting or influential newer works on human-machine collaboration? JULIE: Oh, great question. On a slightly different tack, I might recommend to people, a book called Girl Decoded. This is a book by an author also via MIT, just as an FYI. But thinking about maybe some different aspects than Laura Major and I cover in our book, Rana, the author of that book, is a leader in developing technology that can infer or establish human emotions. There's a company called Affectiva and the ways that emotional intelligence is an equally important aspect to our overall intelligence and how technology is evolving and can change, I don’t know, our future with technology [chuckles] via these new systems that are incorporated to look at our facial expressions, our language, our affect more generally. I would strongly recommend those, but pick up our book. Also, check out Girl Decoded as another really excellent, thought-provoking book in this space. REIN: I have one more question along these lines and this is an entirely selfish question for my own interest. What is your favorite, most seminal paper in your field? JULIE: So I would say that the paper that sent me down this path, I read when I was a Master's student at MIT. My background in undergrad was in aerospace engineering and I did a Master's in human factors engineering and it was only after there that I switched gears and did my Ph.D. in artificial intelligence. Much of my work is a bridging those two fields and thinking about what we draw from human factors to be able to design the systems that collaborate more effectively with people. A work that around that time that was really seminal for me was a work by Andrea Thomaz and related work by Guy Hoffman, who were both graduate students at the Media Lab and now our faculty respectively, at UT Austin and Cornell and our leaders in this field. So just go check out everything that they've written and published. But there was this one paper in which Andrea, I believe was looking at reinforcement learning techniques back then. This is circa 2006 or so. Looking at reinforcement learning techniques, like the way you train a machine learning. One of the ways you train it is by giving it a signal back about how it’s doing some tasks via positive or negative rewards. Similar to how you train a dog. If it does something right, you say, “Good dog,” and give it a treat or it does something bad, you give it no feedback or you give it negative feedback. Those reinforcement learning techniques were not developed, the model for them were really not developed considering what type of signal is easy or hard for a person to give to the reinforcement learning model. There was a now seminal study in my view, then that found that actually people were quite bad at giving the reinforcement signal that the model mathematically assumed it was getting. So when people will give positive or negative feedback, imagine training a dog, or, in this case, training an AI agent to do something, the system is asking for feedback on the action it just took. “The action I just took, was that good or was that bad towards making progress, towards achieving my goal?” But when a person gives feedback, they're thinking about all sorts of other things; they're thinking about what the agent had done in the past or what they might do in the future and prospectively giving some feedback about where they think the agent might be going or what it might be doing next and that messes up reinforcement learning. [chuckles] When people are not giving the signal in the way that the model mathematically assumes it’s going to receive it and use it. That mismatch really intrigued me and inspired me to think about in other contexts, where this mismatch exists and how you could redesign the machine learning model to better solve that impedance mismatch. DAMIEN: There are lessons in there for parenting, also. JULIE: Definitely! Yes. [laughs] REIN: Yeah, I was even thinking about how when you train a dog, you have to make sure that the feedback you’re giving, the dog knows what’s it’s for. JULIE: Exactly. It’s like all of the – REIN: You can’t just come back 3 hours later. JULIE: Yes! [laughs] That’s exactly right and it has to be timely. That's another thing, if a person gives feedback, if they thought about something the agent did a few steps ago and then they're like, “Oh, wait, that wasn't good,” and gives negative feedback. That temporal aspect to it is a problem as well. Actually, in the dog training books, they tell you that same thing, they're like, “Okay, your signal needs to come really, really close to what the dog just did. Otherwise, it's totally useless and confusing.” So yeah, there is definitely an anomaly there. REIN: What was the title of this paper? JULIE: Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance With Implications for Learning Performance. It is a 2006 paper right when I started my Ph.D. [chuckles] DAMIEN: Nice and you mentioned there’s another paper that you recommend to your students to read most often. JULIE: It's another paper by Guy Hoffman, the other researcher that I mentioned. It's a paper validating different ways of assessing team fluency in human-robot interaction. It's an incredibly important paper in the field because if you're going to ask well, how well a person and a robot, how good are they at working together? You're going to try to assess that via some measures, but then every researcher is making up their own measures, like [chuckles] what does fluency mean here, or effective teamwork mean here versus in this task versus that task. One of the things that's important for moving the field forward is to have common ways of measuring key aspects of the collaboration, but then have those measures be validated. There's a very nice paper by Guy Hoffman, who is now faculty at Cornell, on validated measures for human-machine or human-robot team fluency. REIN: That’s the Evaluating Fluency paper? JULIE: It is. Did you find it? REIN: Yes, I was actually already familiar with that one. JULIE: Oh, yeah, yeah. Awesome. REIN: So reflections, let's do reflections. I was thinking back to a thing that we kind of skipped over briefly, which is about being able to open up some of these black boxes and that you can't do that with machine learning and AI. There’s an axiom of cybernetics which is that you can characterize the variety of a system as a black box would have happened to get inside it and so, that is what makes control of these systems possible because you have to be able to account for that variety of the control of a system. So the thing that I'm thinking about is there may be a sense in which AI or ML systems are categorically different from the sorts of systems we tried to control in the past because you can’t characterize the variety of the system anymore just by observing its inputs and outputs. DAMIEN: Nice, yeah. This was a real blast. I also studied artificial intelligence at MIT but I stopped at my Bachelor’s degree so, it’s great to dip my toe in a little bit and to see that what's happening now, artificial intelligence is not human intelligence nor should it be. The Turing test is a fabulous thought exercise, but not a goal to lead towards and the goals are actually making systems and human lives better, not just making computer better. That's a very, very different task. That’s what I’m thinking a lot about more so, thank you. JULIE: It’s awesome. Yeah, so something I'm thinking about from this conversation that I hadn't quite put together in such a crisp way previously was that when we talk about needing those suboptimized parts of the systems to make an optimized overall system, that that extends not just to the human-machine team or partnership. This is definitely woven into the book, but I had never actually expressed it in this way that you helped pull out through this discussion that system is really at this much larger societal level as well and that the feedback goes much deeper back to the technology for what we want on our roads needs to go back and shape not only the fundamental survey I questioned but how we shape the direction of the technology more fundamentally. Now I'm thinking about what sorts of things, as I go about my day, do we want to try to suboptimize. [laughter] In what way, should we aim for mediocrity? [laughs] REIN: Yeah, my favorite Ackoff saying is, “A system isn't the sum of its parts, it's the product of its interactions.” JULIE: Yes, yes. Well said. REIN: Well, he said it. I just quoted it. JULIE: Great quote. [laughter] REIN: So we're done. We did the thing and you did great. Thank you so much. JULIE: Thank you for having me! This was really fun.