Track 1: [0:21] Hello, and welcome to Teaching Python. This is Kelly. I'm a teacher who codes, and I'm with Philip, who is... Professor who's super amazing in everything. Since I'm without a co-host, I'm just going to plug you in there for it, Philip. Track 3: [0:39] All right. Yes. My name is Philip Guo, and I'm a professor of cognitive science at UC San Diego. So despite the title, I actually am not a cognitive scientist. I'm more of a computer scientist and a computer science education person. It just happens that through happenstance that a lot of the people who do more human-centered and education-related at computer work. We have an academic home in the cognitive science department, which is actually very diverse here at UCSD. And one of the big projects I've been running for many years is this tool called Python Tutor at pythontutor.com, which actually works for a lot of languages like Java, C, C++, JavaScript. It's not just for Python, but since the Teaching Python podcast, well, it seems relevant. And I've been on the podcast before to talk about various other educational initiatives and such. I've been a longtime listener for many years. Track 1: [1:30] Yeah, you're like, you're one of my favorite. And I love it because even though the cognitive science part, I feel like I can always nerd out with you when it comes to teaching and I always learn something new. But before we get started, I want to talk about when's the week. So I feel like I need a reset. I've had a really busy week, but I'm going to make you go first. So if there's anything that's a one of the week in the classroom, Track 1: [1:52] at work, at home that you want to share. Track 3: [1:55] Yeah, so I think my win of the week is that I just hired a second undergraduate student to help me on the Python Tutor project. So for many years, as a context, I've been running this project for 16 years now. So it's older than all of your students, right? So 16 years since 2010 is older than all of your students. And for basically all that time, I've been doing it myself. It's basically a one-person show. I've done all the development, all the design work, all the just, you know, hacking it together with duct tape. And then in the past year, I hired one really talented undergrad to be essentially my visual and graphic designer. So she's been helping me basically revamp the site and the user experience to make it much more modern because the site looked like it was made by a graduate student 16 years ago with no experience in design, which was me. And it just kind of stayed that way. And now we're giving a much more modern look for a newer generation and stuff. So that's last year. So she's been working me for a year. And then I recently... Track 3: [2:57] Found another student who is going to be helping me do more user experience research work so more actually very very relevant to this podcast so she's literally last night the win of the week was that we went through kind of a draft email to send out to some some k-12 teachers who are using python tutor so that she can you know interview them about their experiences and about their desires for the tool and everything so i'm really excited for her to do that because i feel like she can probably do a better job than me because she is a brand new college student she just been through several you know high school computer science classes she knows what it's like from the student side it's all fresh in her mind now she gets to talk to these teachers so i'm i'm really excited for for her so that's a big win for me hopefully that's. Track 1: [3:43] So awesome i love that and and i'm gonna try to not talk about that but it's this is the whole thing i love the fact that the Python tutor, the code runs, and we can really start talking about understanding it. But I'm going to go to my win of the week before I jump in and get to it. This is why Sean is usually very useful for me because he keeps me focused to stay on the win of the week. Track 1: [4:07] So, gosh, lots of things. One, it's getting close to summer. I think we have like 19 days, school days left. And so I'm excited about that. So that's a huge one. But today was my last in-person 30-minute training of the school year, and I have done two a week since August, and that's a lot that's been going on. I can't remember how many sessions I've done, like, I don't know, like 75 sessions, because I do eight sessions every, I do four, One topic every week, eight times in a week, and it's been a lot. So I'm excited. So that's a win of the week. We were really celebrating and just reflecting on how far the teachers have come. We were talking about all kinds of things of mixing up Gemini with Canva AI, teachers making their own products, understanding how some AIs like Canva used to be really heavy biased and skewed, and now they overcompensated. And so now they've overcompensated on their biases. So now we're looking at that feature to something to take in. And it was just great having that conversation with these people who have stuck with me all year and have pretty much come to almost every single one of the sessions. So it was kind of like I sat back, I showed them a couple of things and then the conversations went crazy. That's awesome. Track 3: [5:32] Yeah, I mean, it's cool to reflect because it's like when you're in the midst of it, You're doing every single session, but then you think back, wow, it's been a few dozen of them, and we've actually gone through this collective experience together as a cohort. Track 1: [5:44] They couldn't even prompt before, and most of them had never even used or heard of, I mean, they heard of Chachy Petit on the news, but they were like, oh, it's too scary. So in a full year, I feel like they have come so far. We talk about context windows and really, truly understanding what's going on. And that's why I love like this topic with you of really, truly understanding what's going on behind something. Track 1: [6:08] Yeah, you like that. How something's behind the scenes. And when you first reached out again, of course, one, you reached out and said, hey, I have a question for you. And I was like, yes, come back on the show. And two, I started thinking about, and I know this was not something that you were intended, but I started thinking about the fact that I shifted a little bit in my eighth grade teaching, and we use a lot of AI, and I'm really trying to get them to understand what's going on, because, you know, with code, AI tends to either use a function that's not, or calls a function, but never uses it or does something. And so I thought, wow, you know, using Tudor and really showing them how it works is quite an interesting thing. Track 3: [6:52] Yeah, no, I think it's actually really good. It's a really good and appropriate topic. And very incidentally, I think I sent you, you can put in the show notes, this press release that UCSD made about my project because it's been going on for so many years and such. And the story that our writer, kind of our kind of PR comms person, kind of the narrative that she worked with me to write, it's actually it's uncanny how similar is what you're saying, right? Track 3: [7:23] Because she wanted to orient the story around AI, obviously, because it's a very newsworthy item. Everyone's thinking about AI and education and AI and software development. Track 3: [7:32] And the way she framed it, she heard about my project, and it was not originally an AI project, but she heard about my project and she was saying that it's a really good story because in this age where AI is generating so much of our code, what is our value at as both educators and as practitioners? And it's that, you know, we hope that there's still value in understanding what's going on in the code that AI is generating so that we could correct it when it goes wrong. We can debug it. We can know to do like kind of sniff tests. Like, well, this code seems kind of weird. I don't know why, but it seems kind of weird. I mean, the analogy I always kind of make is to like math, right? That's always a sensible analogy of like, why are we having kids do all this math by hand when clearly on the job, no one is doing math by hand. They're just putting in computer packages. And one of the kind of arguments always made is to get that number sense and intuition, you know, that enables you to do things that you might not otherwise do. So, you know, a simple example is if you have some data entry error in your spreadsheet and some number just seems kind of off, if you just don't have a good number sense of how, you know, three digit numbers add together. Track 3: [8:41] You just think, well, that's weird. That can't add up to like a 10-digit number or multiply up to a 10-digit number that just seems like there's something off about that number when you know if you don't have that sense of doing a bunch of multiplication by hand you just won't sense those kind of mistakes that would inevitably be made in any complex you know system and you know perhaps the same thing holds for for software encoding as well. Track 1: [9:04] You know, I would like to think that the sniff test works, but with the middle schoolers, we're not quite there. And I think that's what, for me, has been a struggle and trying to get them to visibly see. case in point today. I started this quarter with the kids generating an AI code because I was trying to show them that let's find something. And it's really hard with eighth graders. Let's find something that we're actually interested in, right? In the fourth quarter, they're already set to go into ninth grade. They're already checking out of middle school. And I finally got them to create something. And the kids, for example, one kid was making an app to check to store the fishing fish that they caught at certain waypoints on their GPS so that they can calculate how many fish they caught at each waypoint. And you can do this with a, you know, an everyday Garmin, but to make it yourself on your own app in Python is pretty cool. And what's cool about this, I can bring in dictionaries, I can bring in APIs, we can talk about reading and writing files. So it all sort of fits into that curriculum that I had always done, but I did it differently. Track 1: [10:20] And so we're walking through this code and I'm asking him, OK, well, great. It's saying, oh, I got so many points. I was like, where did this come from? Well, the code made it. And you start to understand that they really have no clue. Track 1: [10:34] And trying to get them to see these things and step through it and understand that this output that's getting generated, it really doesn't make sense. And I think that that is the key to what we want to do with AI and like, could Python Tutor be that, be that way? Track 3: [10:54] Yeah. No, I think that it's an interesting use case, right? Of, of, you know, I mean, you can imagine a thing in the system where the AI generates code, right? A code example, and then asks you to walk through it and ask you questions about it. Or even right now, what teachers are doing in the classroom, one of the big use cases, they create an example by hand right the teacher creates an example and they walk the class through it step by step and they see the diagrams being generated and you know in the interactive version what teachers have told us you know in terms of feature requests is that you know it would be cool to be able to ask questions about you before you hit a step what do you think actually happens to this picture right and right now they do it manually they're just like we're at step one i'm about to hit the button to go to step two what do you think is going to happen and then they pull the class and and such and then you know the students can be like oh i think that this number will increase or i think this arrow go there and then they step forward and reveal the answer so you can imagine doing that with ai generated code right you can just be like instead of having an example that's generated by hand you can have an example generated by ai and then you could have people trace through it essentially and try to try to understand it and yeah perhaps Perhaps the value of this tool is, you know, as increasingly writing our code is to just paste in code and help people understand what's going on, right? Rather than, it's not really a productivity tool in terms of trying to like help people write their own code faster because now AI can do that with good prompting. So, yeah, that's a good point. Track 1: [12:23] But we also have to, I think we have to take it a step further further. And this is some things that I've started to realize sometimes with the tutor and just AI code in general, right? The kids will walk through a code. Track 1: [12:37] Well, first, let's start with the biggest problem. They make this AI code, and if they don't really know what to ask for, if I don't say, hey, it needs to have dictionaries, it needs to write to a file, don't use classes, if I don't give them constraints, the code's going to be intense, right? They're going to bring in all too complicated. They're not going to be able to understand. So that's my first problem. Kids need the constraints. Track 1: [13:02] The second problems are normally the functions. They'll overcomplicate or overmake functions when an AI generates the code. They'll make a function like even smaller as here's an input function. And you'll notice like there's a lot of little functions. And then what happens is it's hard for the kids to see the functions being played out in the main part of the script. And so that a lot of the times is the first hang up. So we can get through that. I like that with the Python tutor, it's really good to show lists, to show assignments, you know, show how it steps through, goes through some sort of appending or whatever, writing, popping. And that really works well but what it would be really nice is if somehow it could we can and i know it's a lot more memory and a lot more work for you take on more of a more of a script and see how that bumps around because i feel like now with ai our basic 50 lines of code and here's how i make this is is is getting past its Track 1: [14:09] due date if that makes sense no. Track 3: [14:11] It's interesting so it's kind of like, you know, just for context for viewers or listeners who aren't familiar with the tool, right? The kind of mental model of how this Python Tutor tool works is that. Track 3: [14:22] Kind of takes in it's like a virtual whiteboard or blackboard right it's like you usually write an example as 10 or 20 lines of code at most and then you're tracing through it by hand on the board or drawing pictures and it kind of works at that level right so when you have 100 lines of code or 200 lines of code it could still technically work we have to scroll through a lot of code and it you know the arrow jumps up and down like you were saying kelly like if there's functions calls another function little function calls it just keeps jumping back and forth and it starts to get pretty confusing i mean i had two thoughts on this one thought is you can imagine a system prompt for whatever ai you are going to be using whether it's you know maybe it's when we built in a python tutor for an example generator and the system prompt could be something like only use you know don't use little function helper functions just try to write straight line code don't use advanced classes or other you know decorators or whatever only use these you know 10 basic python features that are in your classes with the hope that it generates more straightforward code and also another thing is like don't do like extensive error checking exception handling whatever right because if you want to generate production code it will be very nested like try accept handlers and catching different kinds of exceptions and all kind of robustness things so we may strip those away that that's kind of prompting the AI to make the code simpler. Track 3: [15:43] And then the other thing is, you know, we need to make the tool more robust even when AI or humans generate more complicated code. And you have had some thoughts about. Track 3: [15:52] I've been reading some research papers actually of people making different prototypes of just their own systems. And one especially interesting one, I think it's been published in this like small workshop. I don't know that PDF is online, but the idea there is essentially to, you know, kind of unroll the whole execution so that you just see one line at a time that actually executes. So it's sort of like, you know, instead of seeing, because right now, and when you're using Python Tutor or any other tool, you just see the source code, you have lines one to a hundred. And then, you know, if you have a function, you're on line 50, it goes line 50, line 51. But then if you jumped to another function on line 10, it goes to line 10 and then line 90 and then line 80. So it jumps all over the place. So then you get kind of disoriented. So their idea, which I might actually, you know, now that I see their publication, I might actually try it. In some kind of mode in python tutor is to just it's really it's an elegant idea which is just why don't you have an alternate view where it just shows the lines that are actually executed with some kind of context so it's like now i'm on line 50 it'll just show line 50 and then the next line is 51 next line is line two it jumps out of the function i'll just say you know the next line that happens is line two and here's like the function body and stuff so when you actually read through it it's actually like every line that executed rather than the actual source line so you can imagine alternate forms of viewing this but yeah nothing is perfect yeah. Track 1: [17:13] I i can see i can see like almost a transparency of the code. Track 3: [17:18] And that. Track 1: [17:19] Highlighted line i think that's going to help also because my biggest my biggest complaint i guess my biggest hardest thing to teach i think for kids is still functions and returning returning and imprinting the output and how it how you have to reassign it to a variable down in the main, and then they don't really understand how the parameters and arguments work because they can't see that connection because you've written it, you've defined the function early on in line 20, and now you're calling it or using it again in line 200, and they don't understand fully. And I feel like even now when we're generating the AI code, there's a bigger loss of connections. Track 3: [18:08] Yeah, because they're not typing it. because they didn't write it. Exactly, they didn't write it because, you know, if, If they follow good practices, that's a really great example, right? Track 3: [18:17] They wouldn't have instantly generated the 200 lines of code, right? They would be writing two lines first. They would write their little main function, and they would try to write the first function call, and then it would probably fail for some reason. Track 3: [18:29] And they would kind of ask you for help or ask someone for help until they got to the first function call where they call the function foo and then put the parentheses and put the numbers in. And then they see it on the other side and they see it print out something in the function and they see it return out and resign it it's like it's like if you had it in in the old day before 2023 or whatever when people wrote stuff by hand you actually had to hand you know seems kind of silly say but you had to hand write every you know piece of code but more than just typing the the letters out it's really in your mind you're kind of connecting together the components right so in main you know you you you set up a function call you put the arguments in and then what happens it ships off to the function and you're like oh i need to do some of the parameters and i need to return something out of it you build it up one little bit at a time and you test it and then when it works it works and then you do the next version but with the ai generate code it's it's sort of like if the teacher just hands you 100 lines of code and said do something because you didn't actually get a feel for it so in terms of the tool itself the python tutor itself you know it's structured using pdb which is the python debugger which is the standard one line at a time debugger so it doesn't actually go into the real details of like on this line you actually have to you know copy the argument into the parameters on this line and have to return out it does a decent job but i could imagine. Track 3: [19:54] If this was a focus you know you could you could have a more detailed drawing of exactly what happens when something returns out and stuff and it's i think it's just it's more implementation work and it doesn't come standard in the pd debugger so it's you know it's all it's a little bit less robust that way but um that's definitely something to consider just these i guess i'm trying to generalize some of your point which is like there are these touch points that are like real points of friction for students Track 3: [20:20] like calling and returning from a function or like you know pointers or you know something yeah. Track 1: [20:25] I mean it's really weird because and it's been i think computer scientists developers are going through this whole the people in the k-12 fields i don't know how it is in the universities but we're going through this whole process where we understand that ai is here right it's here even us we're getting lazy some of us and how do we still teach these foundational skills and understand that the kids are offloading a lot of that cognitive learning to AI. And I think it's a constant grapple of what we're giving up in terms of what we want them to learn. And I've felt lately this year that my biggest Yeah. Win. My biggest win for the kids is just getting them to understand that it's pretty cool and you can create a lot of things faster. And I feel that now that I can get them hooked, now I actually wanted them to have some sort of intelligence behind what they're creating. And it was coming to that story. There was just, I don't even know the name of the company, but some company in Silicon Valley where a person made some code with an agent and it totally deleted. Track 3: [21:41] Everything in their. Track 1: [21:42] In their companies. Track 3: [21:43] You know i think those are great stories that you know the news headlines put up and stuff and it's just explaining that as well i think that's no i like your thought about i mean this is getting a higher level you know kind of educational philosophy debate right of the top down versus bottom up right like this is the age-old debate of you know should we try to you know should we try to motivate using real world examples and authentic environments and the more kind of you know constructivist view of we want kids to be building stuff with their hands and doing more authentic things even if they're not understanding the fundamental physics or whatever building or engineering building blocks or should we go the complete other route of the very bottom-up route of here are you know here are the first principles of how code works or how math works or how whatever works and i like how you're advocating for kind of mix of like let's do the very authentic real world context things first you know in the first week knowing that you have ai you can all type in prompt and make you know an awesome game or something you can all make some awesome game and that's great you know that that's what you'll eventually be able to do if you want to get into computing as a career but then you know you open up the code it's like you have no idea what's going on it's a thousand lines of code and then what we're going to learn in this class is we're going to learn the fundamentals of how these building blocks work so that when you do use AI, hopefully have a better mental model. I think my own. Track 3: [23:09] I think my own bias, I mean, just a lot to discuss to you for K-12 is to really, you know, hone in on these fundamentals. I just, I mean, I keep making the math analogy, right, of, you know, if we have coding exercises and coding things like math almost, right. Track 3: [23:27] You're using a tool like Python Tutor or something else to really quiz students on like, okay, here's a five-line piece of code, ten-line piece of code. It exercises some concept. Can you walk through it and tell me what happens? When this with when this runs and you know a an even more kind of extreme version of this is you have kind of an oral exam right like which is kind of like a interview for a job right it's like here's 10 lines of code can you just talk me through what you think it does i'm just gonna ask you some questions and i feel like if a student really has a good mental model understands what's going on then they'll feel confident they can answer your questions but if they obviously just wrote the code with ai or they just you know just studied it briefly they won't have a very robust model in their head so like for me that's the interesting part because like you were saying the ability to generate impressive looking code with ai is now something that many people can do without really an understanding. Track 1: [24:26] It's funny that you say this because I've switched a lot to code discussions and I've done, I think what I believe is that I'm the first person in probably the history of K-12, maybe not a professor, to do a, I don't know if you've ever heard of a Harkness seminar or a Socratic seminar. Track 3: [24:47] Tell me about it. Track 1: [24:48] So in English classes, they do it a lot where you have to read a passage or read a book and they have an oral discussion in a circle and every child has to develop the thought based on the previous child's questions. So it's this whole method. If you Google Socratic seminar, they've also kind of put it in with AI of this is how you do a teaching moment where you're asking questions, going back and forth. And a lot of people have developed that. So I did this with our code. And I've been like I said, I've been using this AI code now for five weeks. And we've done here's a variable, put some comments in there. Here's a list. Here's a for loop. Now do version five and we're adding list. So we had a Socratic Harkness code review. Last week and it was really interesting listening to the kids come across and say well yes i've used that similar conditional statement in my code to to go through a menu and it was quite it was quite interesting to see how they could really talk about it so how. Track 3: [25:53] How big are the groups you do for this is like a group of five or like a group of 10 or three. Track 1: [25:58] I had 15 kids and normally english teachers will do two circles so sometimes they'll do a circle that watches versus like a smaller. And this is my first time. So I invited high school AP teacher and my eighth grade English teacher. And they helped me when, because apparently you're supposed to wait for a minute if no kid talks. And I'm not very good at pausing. And so the English teachers were with me and they don't really know a lot about, they don't know anything about code. Track 1: [26:25] And so it was neat for them to see that develop. But I'm almost thinking that kind of application, here's another thing for you, for you and your intern, what if it was you know we spoke into it to understand it because kids are also lazy. Track 3: [26:40] Not only for. Track 1: [26:42] Typing but all you know what if they could speak through it and say okay this is what's going on kind of feature or. Track 3: [26:50] Yeah i mean the voice recognitions are getting really good i think the voice interaction is really interesting because you know already i think, having ai explanations for while it's stepping through would be useful right some some students have i mean i want to run this by you right so right now the site i mean i haven't been giving context on the site for people who don't know as i just assume you know i mean you could put the link up but right now what happens is on the site is that you know you write code and then it steps through one line at a time and you see the diagrams of the stack and the global variables and local variables and data structures and stuff, just like how a teacher would draw on the board. And, last year i added an ai chat on the bottom so if you scroll down the bottom there is an embedded chat that knows about your code and if you're running the code it knows about your visualization and knows if you have an error so the hope is that people use it for you know debugging the errors or explaining what's going on it's like essentially what is what is this picture showing me right because then the the it passes in kind of a representation of the visualization into the. Track 3: [27:56] And so that it can answer questions like what is happening in this picture and then be like oh this picture shows you know two arrows pointing to the same object and and you're on line five or there's an you know an out of bounds error and it's because you you know went one past the end of this list or something so there's a normal chat interface but it's not super integrated, but you know one concern teachers have with seeing a chat interface obviously is that students may use it to, you know, generate solutions or help them too much. You know, they could always go to ChatGPT for that, obviously. They can always place in their code every day if they want. And when someone suggested, I've heard a few people suggest that they might be more okay with an AI if it was more like inline help. So like if you click on an object or you click on a variable or you click on a line of code, you only have certain choices for it to do, which is explain what's going on here or help me debug this error. So it's not as much freeform input because if it's freeform, then students might ask, oh, you know, give me the answer or like help me solve the problem. But if it's posed more like kind of a smarter tool tip, like just click on this line and give me a explanation of this line. Track 3: [29:06] Then teachers might be more okay with having that because it's sort of like asking the teacher to explain something rather than free-form just having to have GPT chat with you? I mean, just in general, what do you think about the role of AI in these coding environments for education? From not having it all to having all of it. Because I'm debating that still, Track 3: [29:26] right? What is appropriate to put on the site? Track 1: [29:28] So the thing is, is no matter... Which tool you have, whether it's lockdown. So we have a tool at our school that's made only for kids under 18. And it's got a lot of guardrails, and it's very learner-focused. So if the kid goes to a chat and says, you know, I need to talk about code, I need this code, it'll say, yes, but the point of coding is for you to learn. Track 1: [29:55] What do you want to do? And it's really supposed to be positive and it's supposed to promote the learning process well anything about cognitive science if a kid wants to answer the kid's going to get the answer whether whether they'll just go to chat to bt exactly on their phone or something if it's blocked exactly and there's there's kids that open up open up even you know google lens or something and then they'll copy the answer from ai overview into the flint just to get it to get past the assignment because you can make assignments with this so over the past three years that i've been using this i've become even more to the fact that i don't even do these assignments where it locks down i write the assignment so it says whatever the kid asks you give them the answer but after you've given them the answer check that they understand and then i try to get them to do through the process and that's how i came up to where i am right now with giving them this ai code and doing a reverse engineering the socratic kind of discussion yes and and always come in i've been touching this code we've now like i said for five weeks is are they learning are they learning what i had originally thought i don't know but i'm also fighting with that cognitive kind of thing that's happening with them an ai so i'm not sure that. Track 1: [31:16] When you make, you know, coming long story short, I'm not sure that even if you did put it in lockdown, that it would really keep the kids who wanted to find the answer from finding the answer. I think there's a tool that we also use where the kids have to speak out and explain their thinking. I think that, and that's what I liked about the Python tuner, it kind of shows visible thinking happening, right? But it's visible thinking coming from the computer generated. Track 1: [31:44] So I think as long as that, student can explain those arrows and going through it. Maybe it's a recording while you're doing it in there and they have to talk it out. I think that would be a more useful tool for teachers, right? So as I'm clicking next, next, next, it's showing my arrows and the student is recording their process going, okay, at this point, the list is starting at index zero, it's going through the loop, et cetera, et cetera, And you can use either an AI generator to verify that voice to diagram. And then the student, even if they script it, you can tell when the kids script, they can run it through Python Tutor the whole time. Do you use Google Photo Lens to get an answer? You can tell the difference when they're reading it and when they're thinking visibly, speaking it. Track 3: [32:41] That's so interesting. I mean, I could imagine, I mean, the low-tech solution for that is to just have the Python tutor up with an example and then basically use it as the basis for a one-on-one dialogue or a small group dialogue. Track 1: [32:52] Or screencast it. Track 3: [32:53] Or screencast it. Yeah. So then, you know, the more scalable version is on the site itself. You know, perhaps we make a teacher portal where they can create an assignment, let's say. And that assignment is just record yourself talking through this thing for a minute. And then they can collect student submissions that way. That's, yeah, it's so interesting. Some of my colleagues at UCSD, they have a. Track 3: [33:12] They piloted this in their class with, this was a few years ago. I mean, during the pandemic, obviously, where a lot of this stuff was becoming a concern, right? I think, you know, I think early pandemic was before all the chat, GBT and go-to-copilot stuff. But it was that age where, as you remember, because everyone's doing school online, like people, you know, the cheating was a lot more rampant because even without the AI stuff, people were able to, you know, find answers online and whatever, right? So I think they instituted, i have to find this paper but then they wrote a paper about this uh work when they kind of did a deployment in their classroom is that they institute essentially like a video exam so like you know every student you know they give them a topic and they have to talk about it for a minute you know explain this how this data structure works explain whatever and you know there's the challenge here you know at a big school is you have hundreds of students in a class right so you can't give a one-on-one oral exam so they wanted to scale it up by just having students do one minute and then they distribute the grading to the TAs. And it's, you know, it's not as perfect as, you know, the instructor talking to everyone, but they try to approximate it and they kind of give some experiences where I think it matches your intuition, right? Like you can kind of tell when someone is actually just talking through it and they know what they're talking about rather than, you know, reading from a script or really memorizing something beforehand. So that's cool. Track 1: [34:29] What does a cognitive science say? Like I fully believe if you can talk about it, if you can have a conversation about it, your brain has has locked it in and you know i remember just trying to to study for pre-med and i would record myself on a tape recorder my little handheld tape recorder and just read and talk through the the whole thing what is what else. Track 3: [34:54] Yeah i mean i think just the thing that comes to mind no pun intended that comes to mind is is the whole retrieval thing right the idea that you know i mean you've had many guests on who talked about this way the idea that if you're just passively reading and highlighting in text, right? Highlighting is not super effective because you're just kind of going through it. But if you force yourself to retrieve that information, if you're making flashcards for yourself, or if you're, you know, covering up the second half of the page and making yourself guess what's on there and verifying it, that helps. And I think there's a bunch of stuff about like, self-explanations or giving explanations others but i yeah i think i think it largely matches your own intuitions right of if you have to retrieve it from your brain and articulate it in some form that's coherent to explain to someone else that is that that's really good and i think an even harder test is actually having to write something out right like orally is one thing but if you actually had to write a tutorial that's pretty succinct so maybe that's a interesting student problem except i don't know there's an ai exactly yeah that's the thing right so right there's an ai for everything you can see them right in front of you or something but yeah so yeah yeah but if someone can actually write out a good explanation then that really you can really tell to understand right Track 3: [36:08] but obviously yeah you can write all that too very well funny. Track 1: [36:11] Funny story not to kind of switch topics. Track 3: [36:13] But i have. Track 1: [36:13] A kid who's not a good coder at all but he's a brilliant ai user probably one of the better eighth graders I. Track 3: [36:21] Know. Track 1: [36:21] Right? And so here's my thought process. I was like, okay, well, Track 1: [36:24] I'm going to use Circuit. I'm going to bring back hardware. I'm going to bring back hardware. We're going to do some hardware in robotics class because you can have perfect code from Adafruit with CircuitPython, and you can copy the code into the editor, but getting the stupid NeoPixels to light up, you can have... You know, the wrong cable. You can have the library missing. You can have everything, right? So this one kid, I was like, okay, we're going to have a challenge. We have NeoPixels working. I've showed you how to light one up. You guys understand how to do lists because these are eighth graders. I was like, now I want every other light to light up a color. Then I want it to switch off and I want the other four pixels to light up. So you have it kind of like flipping back and forth and he goes i did it. Track 3: [37:11] He's really good at prompting he's just really good at really. Track 1: [37:16] Good at prompting so i i mean i think we're working with a new school skill here of no matter what tricks what tricks a teacher is trying to do the kids who get this error are going to get the answers whether there's a the old form of learning or not and and i'm wondering is that bad i mean as long as they can understand why it's. Track 3: [37:43] Happening yeah no that's interesting i mean i'm trying to think of you know pre-ai days right so you know even pre-ai days right like 10 years ago you know there's you know in in college you know students did a lot of these hackathons right where they would build apps together in 24 48 hours and a lot of that was looking on stack overflow right looking at stack overflow for code examples looking at tutorial sites you know it was really mashing together pieces like oh we have this idea let's connect this library pair with this one you look at the website for the api you try to put together you try to you know go to stack overflow something doesn't work you you put the error message in google and then there's some stack overflow answer but you paste it in you try to adjust it it works and and i think that in the end you probably don't understand how your app works very well but you know there is a skill to like what you're saying like the equivalent of prompting ai now is 10 years ago was the skill of googling around and looking at stack overflow and integrating examples like it's that sort of almost like. Track 3: [38:42] Craft or kind of, I mean, we call it hacking, right? We call it just putting stuff together. It's like the skill of being scrappy and putting stuff together. And it's a less rigorous skill than, you know, the mathematical thinking, the very formal, I understand how things work from first principles, but it's nonetheless really important. Right. And it's, I mean, I would love for someone to have both, right. That, you know, if you only have the very formal reasoning and kind of mathematical thinking, all that, it's very hard to do anything practical, right? Because practically what you're doing as a software developer is putting different APIs together. Track 3: [39:16] Debugging weird errors, like, you know, the pin isn't stuck in or like you didn't reboot your computer or like you have your cache turned on in your browser and it's not reloading the data, just stuff like that. That's not like, that's just stuff you learn from experience and that AI can help you with. So I don't know. It's a much broader topic. I mean, I'm thinking a lot about the, Because of this Python tutor, I'm telling a lot about the very kind of clean fundamentals, right, away from all the stuff that really is necessary to build software, right? So, you know, I'm kind of trying to live in the world of the equivalent, the math world of you're in math class and it's like, you're learning algebra. This is a lesson you're learning. We want to manipulate these symbols really well and have you have a really good understanding of that. Whereas the applied math world if you're in real life you know work on an engineering project we're using math you're using you know matlab or mathematica or whatever and you're simulating these things at a very high level and you're never doing Track 3: [40:13] the math by hand so yeah i don't know it's a. Track 1: [40:15] It's a hard question i want i want here's another one i want i want the tutor to really step through you know what would be really cool is like the deployment and everything that goes in, from you know at this point of of my life of coding i'm a i'm a one script kind of wonder right i can make i can make a couple files talk to each other i know how to do it i understand the concept you know connect apis but you know what happens when you compile all these scripts together and And then you want to deploy it somewhere. And how does that work? And what does that look like? I think you can, as a non-computer science major, I can read it, but I don't. Track 3: [41:00] See it yeah yeah no i mean this is something where you know someone like sean would come in and i think you know he would have a very detailed mental model this that's what he does professionally now i think it's great to think about like you know obviously the tutor right now is for very zoomed in on the whiteboard 10 lines of code what is each line doing but you can imagine these same ideas zoomed out and that's really kind of understanding computer systems right so even from you know i'm just thinking when i was growing up in the 90s you know i wasn't a programmer i didn't do programming but i i did a lot of computer stuff i grew up in this pc generation where we built pcs we play games so like i developed this understanding of how files work and how folders work and how installing programs worked on windows right you double click and you install something that goes into some directory and you set a path somewhere because i you know i know from all this experience you know installing games or shareware apps like. Track 3: [41:56] Something doesn't work and then you know this is even before you could google for anything or before google or but you would ask your friends and you kind of developed this side and then when you started programming you know when i started programming in college you know understanding this kind of linux or unix command line environment was the more predominant environment and that's all the i think environmental stuff around the code right it doesn't have to do with the lines of code it has to do with what happens when you run a compiler it generates another file right and when you deploy this somewhere to the cloud it's just somebody's linux computer somewhere and whether it's a you know virtual machine or some some you know platform as a service thing yeah you need to develop this mental model of it and then there are these concepts like i meant fairly of like caching and naming things and other things where you know i think part of it is just you've gotten bitten by it so many times you've done it so many times so you've developed this robust mental model but i mean this gets back to i mean the thing you brought up is just zooming out even more is that. Track 3: [42:57] I see that there are students who can go through a computer science curriculum in high school and then college and do very well in classes and they do build up a very academic mental model of the code, but they don't have that hands-on experience of, I need to go, what service do I choose? What you know do is amazon google cloud microsoft whatever once i choose a service how do i get my stuff on how do i get my website on there how do i start a database how do i know what happens when you know going aside and there's an error like is the database failing i don't know do i need to reboot something do i need to like pay extra money because my credits ran out like all these things that are around the code yeah it would be really cool to to see that in action and i think now the the chat GPTs and Claude's, I think they do a decent job if you kind of ask for an explanation to kind of trace you through the steps, but it would be nice to have something personalized to you. Track 1: [43:53] It would be nice. I mean, because that 1% like you and Sean, that still exists. I still see the kids that are saying, do I have to use AI for this? I'm like, absolutely not. This is, this is, you write your code, you do your 14 versions, you code along and you happy go lucky. I have the kid that was tech was not texting me, but emailing me back and forth because he decided that he wanted to try Lua or something. And he was complaining to me like, seriously, this is so stripped away of everything. That's not, you know, C++. He was complaining. I still have that 1%. But if the if AI is generating for the other 99%, right? So the other 99% can be creators. I think somewhere down the line of having that mental model of really understanding what's going on, it's going to help. The fight against the future hackers, right? So they'll be able to get that sniff test a little bit better of what potential issues you might have. Track 1: [45:02] So it's pretty cool like that. But not to switch gears, we have about 10 minutes real quick. Any news, any other new research projects coming your way? Track 3: [45:13] Yeah. So I think that, I mean, I guess unrelated to all this I've been talking about, one of my PhD students that, yeah, I guess it's related to AI. So we can talk about that. So one of my, my PhD students, like, you know, her dissertation work is around how do we have scientists and help scientists and other sorts of people who have to do complex work in the real world leverage AI better. And I'm excited about that direction because it kind of goes with your, you know. Track 3: [45:41] Plugging in, you know, raspberry pies and whatever, you know, the physical debugging things because you know ai has been really good at things that are purely on the computer right so like even when people are talking about science you know ai and science they really are talking about stuff that's purely computational or knowledge base right they're talking about you know your protein folding or drug discovery or things that are just crunching a lot of data right like you you input in all the stuff from your protein database your genomics database and ai can find out stuff about it or ai can help scientists do a literature review understand research paper synthesized research paper all very knowledge heavy work we call that like at the desk work right it's like any work that you can sit at your desk and do ai will be able to do very well because it's like very good at crunching stuff on the computer and our research direction the paper that. Track 3: [46:29] That we worked on it was called beyond the desk okay so it's like thinking about all these tasks beyond the desk that you know like that real scientists need to do and and they're very mundane things you're there there are things it's equivalent now that we've time back the equivalent to all the stuff that's required to deploy something on aws or google cloud or all that stuff it's that but physical right so you know what are scientists doing in the lab they need to pipette stuff they need to you know you know manipulate stuff with their hands if they're in the field you know they work on animals they need to know how to you know interact with you know they're observing monkeys out and you know with cognitive scientists who are observing monkeys out in the wilderness they need to know how to you know build a rig to be able to like safely observe their their animal subjects and do things that they do like it's very hard right now to just ask chat to be to help them with because it's very physical it's very embodied there's a lot of tacit or unspoken knowledge behind it. So we're doing interviews and observations of scientists who work beyond just like sitting at their desk typing on the computer. Track 3: [47:34] And we want to try to design AI systems that can be more multimodal and that can help in these tasks. And, you know, on the very extreme, like people are talking about humanoid robots, right? Like in a far future where we have humanoid robots do all this stuff for us, then that could be good. But for the time being, you know, lowly humans like us will have to be the ones doing the kind of physical work of plugging in the pins i mean the plugging in the pins is actually a great example because in one of our studies it's like a a neuroscience lab right they have these complex instruments where they're measuring stuff on people's heads and stuff and a lot of their protocol is you know making sure you don't forget the pins and plug in the right pins at the right time and activate this machine and click this thing on and you know It's impossible for AI to help with that now because it's so complicated and multimodal. So can we design some kind of interface to help people deal with that stuff? So that's the big thing I've been thinking about with this other PhD student of mine who's completely unrelated to Python Tutor stuff. Track 1: [48:35] That's cool, though. It just came to mind when you're in a Zoom call now, you can change the view of the camera so it looks down at the desk. Track 3: [48:46] Wow, interesting. Track 1: [48:47] So have her look into that and see if that. Track 3: [48:49] That's interesting. Yeah. So you can see your hands. I mean, another inspiration we get is really from like live streaming, right? Like people who do live streaming of crafts and like, you know, people who do the kind of maker stuff, like they're live streaming themselves, you know, like on their hands, you know, knitting or crocheting. And yeah, a lot of that I think is relevant knowledge. Yeah. So it's you're thinking about AI, but away from the desk is the kind of frontier. Track 1: [49:13] So not to cut now. Are you coming to PyCon? Track 3: [49:16] I i unfortunately it's really hard for me to travel nowadays due to family stuff and such but i would i would love to i would love to one i've actually never been it's funny enough that in the early days of python tutor i submitted a python tutor talk thing but those are very early days i didn't get accepted because you know there was that was a time when it was it was 2012 right when it was just growing a ton and a lot of people were getting it myself i submitted one and then my colleague who, is teaching the large introductory class at UC Berkeley, the one that Quincy Tennyson talked about, right? So Quincy talked about, you know, he made kind of a high school version of that, CS61A, that very rigorous Python-based computer science class. So my colleague who made that class, I think submitted a talk that year, and neither of us got our talk. That was the time when we, I think it was probably in the Bay Area where we were too. Track 1: [50:07] Well, you can always come to the Ed Summit. You can always submit a talk, and I'll let you talk. Yeah, that'd be great. Track 3: [50:12] Right. Track 1: [50:12] Yeah. Track 3: [50:12] No, I heard the open, you know, the open mic thing this time, right? Well, at that. So, yeah, I've been keeping up with that for years. And like I told you about, there was a SIGC conference, which is the more academic version of that, where, you know, it's much more professors. And I don't think SIGC has as much K-12. It's mostly college, you know, college level instructors. So that's kind of like the academic version of the more industry oriented PyCon. But yeah, hopefully in the future. Track 1: [50:39] Hopefully in the future. Anything else you want to share? Track 3: [50:43] I think I'm good. I mean, I would love, you know, if any teachers are listening to this to, you know, try out the Python Tutor. It's funny because we didn't really talk about the tool because we assume people are familiar with it, so we can put a link to it. Track 1: [50:54] Well, we've been, yes, we'll definitely put a link, and then we're going to put the past episodes because if they don't know, then shame on them. They should know. Well, thank you. Track 3: [51:03] Yeah, my only call to action is if there's any, you know, teachers listening to this, you know, try out the tool. There's a teacher mode that you can sign up for that has some enhanced features. And I am, like I was telling you off recording earlier, like, I'm really, really interested in the K-12 teacher population at this point. Because I feel like the college instructor use cases I've already kind of hashed out pretty well because those are the people I know. But K-12, I think, is so interesting because there's such an expansion of computing classes in K-12 in the past decade, right? Because of efforts that co.org and others have made for advocacy of putting it into state standards and into various requirements and recommendations for states and then around the world as well. So I'm hoping, my fingers crossed, like, you know, very self-serving for this tool is that I hope that even in the age of AI, when professionals are using AI full-time to code, which is happening now, right, that we still will find value, especially at K-12, of knowing what's really going on, the nuts and bolts, the fundamentals. And if any teachers are interested in talking to me about it or trying out the tool, I'd love to hear from you all. Track 1: [52:14] And they can reach you at? Track 3: [52:16] My school email is, I can just tell it's pgmyinitialsatucsd.edu. Also, I'm famously not on any social media on anything else. Track 1: [52:25] No, you're not. Track 3: [52:25] I am famously, I cut off all my social media around pandemic times, so purposely hard to reach. But yeah, you can just go to the Python Tutor site and just sign up for the teacher course. Link, and then I'll email you. Track 1: [52:37] That's awesome. That's awesome. Well, it's always a pleasure. I love nerding out with you. I'm sorry we didn't talk more cognitive science next time. Track 3: [52:44] Oh, no. Anytime. Anytime. I'm happy and such. Thank you so much, Kelly. Track 1: [52:48] And so, well, then I will, no updates from myself. It's weird asking myself, are there any updates? No. And then I guess I'll just sign off. So for Teaching Python, this is Kelly. And then you can say, there you go. Thank you. Thank you. Track 3: [53:03] Fill in as a co-host for the day. Track 1: [53:05] Thank you so much. Signing off.