0:00 Hello and thank you for listening to the mathematics teacher educator journal podcast. The mathematics teacher educator journal is co sponsored by the Association of mathematics teacher educators, and the National Council of Teachers of Mathematics. My name is Eva Anheuser, and I'm talking with Tony lamberg, who is an associate professor at the University of Nevada in Reno in the College of Education, and Linda jillette Cohen, who is a PhD student in mathematics education at the same place. They co authored this paper with Diana moss, who is an assistant professor at Utah State University. We will be discussing the article supporting teachers to use formative assessment for adaptive decision making, published in the march 2020 issue of the mathematics teacher educator journal, we will begin by summarizing the main points of the article and discuss in more depth the lessons they shared in the article, their successes and challenges, and how these lessons relate to the their other work to Rooney. And, Linda, thank you for joining us. 1:02 Thank you for inviting us, 1:04 we are happy to be here. 1:06 So let's just get started by giving us a brief summary of the innovation you're talking about in the article. 1:15 There are two frameworks. One is for pd facilitations of anyone who is doing professional development, this will be helpful to think about how do you support shifts in teaching? And then the second one is the whole class discussion framework, which is answering the question of what teachers would ask what do I need to do in order to implement these ideas? So there's two frameworks, one for the teachers and one for PD facilitators, 1:46 okay, and then the professional development framework is like a larger framework and the class facilitation framework is more a smaller framework for class. Okay, cool. Can you give us a brief summary of the article, including the results, 2:03 what we really looked at was, this article came out of a larger project where we had to give content based PD to teachers and kind of think about how do we shift practice. So we kind of looked at formative, we know that formative assessment is critical for supporting learning. But the problem is, even though this research exists, it's not being used in practice. So we were trying to figure out how do we embed formative assessment meaningfully into content based professional development in order to support teachers to improve their teaching so that they can affect student learning. So that was kind of a goals. So in this article, we kind of looked at what shifts occurred as teachers engaged in professional development with regard to formative assessment. Again, this was part of a larger project. And then, to our delight, we are happy to report that shifts did happen, and that the shifts were like from teachers thinking about formative assessment in terms of right and wrong answers and procedural knowledge, to make pacing decisions, it shifted to thinking about how do I develop conceptual understanding in students, so it changed what they looked at, and how they use that information. So that's kind of a brief synopsis of the article in what the readers would get out of that. Okay, thank 3:34 you. So who should be reading this article, really any 3:37 group or individual who wants to support teachers in shifting their practice, that would be from PD facilitators, or teachers of inservice, teachers or researchers. So it really focuses first on teachers use of formative assessment. But it also addresses that ideas for designing PD to support those shifts and practice to happen beyond the scope of the PD. Okay. 4:03 So what is the important problem that you're addressing with these two frameworks. 4:11 So in one, it was focused on teachers use of formative assessment. And we know that formative assessment increases student achievement when used effectively, but that teachers have limited use of integrating formative assessment into making their instruction decisions in the classroom. And so we looked at specifically where that break in the formative aside cycle is happening, which was between gathering the data to using the data and then how to address and adapt instruction or PD so that they would be able to increase their effective use of formative assessment. 4:52 So I think you're referring to the formative assessment cycle that you talked about in the paper. Could you explain that Briefly, 5:00 the formative assessment cycle in the paper, that building on the work of attero. And poplin, is really looking at the say effective professional, effective formative assessment looks at assessing student thinking, of interpreting and being able to use. So that's the cycle. So assess interpret. So we 5:24 have like collecting data, interpreting data, and then using that information to design the task or to revise the task. 5:35 Absolutely. That's exactly what that is. 5:38 And you're talking about, I think you said initially, when they collect data, they would look for right or wrong versus understanding. So changes happen, potentially, in what they collected, but then also how they interpret it. 5:54 Yes. Or even if they interpreted and use it, what we find in the research literature, first of all, they may not even collect data, and some of the teachers who collect data might just look for right or wrong answers. And what we found is that they might use that for pacing decisions, or just being able to go, Okay, why does this data mean? And then because that is the different skill to going? Okay, I learned this above my students, but what does this mean in terms of my curriculum, the standards? And then the other part is going, Okay, how do I use it to actually use that information to make decisions? So this is no natural. 6:37 I'm just trying to like if people aren't as familiar with the idea of what formative assessment means. I'm trying to just clarify. So what are the differences? If I'm listening carefully to you could be that after I taught a class, I give my students an exit card. And then the next day, I say, hey, only three of you answered correctly. So let's go over the material again, or something like that. That would be a pacing decision. Yes. Is if I would collect exit cards and look at, you know, for us the strategy five, use the strategy seven, use this strategy. And you know, the strategy is really useful here. I don't know. I'm trying to make something. 7:20 It's kind of like, how do you build on that, right? So so the teacher who looked at it and said, okay, three children? Got it, I need to reteach it the same exact way. Whereas what we are, when you're looking at a more effective practice is going, three children guarded. But what was the other kids struggling with? The way I approach teaching? It didn't make sense. How do I read assign the task, or maybe I need to go back, they're not ready for this task yet. And backtrack and kind of fill in some gaps so that I can get to that. So that's a lot more sophisticated decision than going. This was my lesson on page 29. I'm going to repeat it, and I keep repeating it, and the kids still don't get it. 8:06 And that is what we would call the difference between I think summative and formative assessment, right? One is just giving you an answer of how many got it, and the other one gives you more information. So I'm trying to backtrack to what you said earlier, one of the things that you said is often teachers will collect, sometimes they don't even collect info, but if they collect it, sometimes they don't use it. 8:33 birth, right. What's interesting is a lot of the teachers would say, you know, when we kind of introduce this topic, oh my gosh, you know, we don't have time to do this. It's not feasible. That would be some of the challenges that would come up. When doing that. And we actually, in our professional development, the article kind of focuses on one site, we had hundred teachers bringing in student work, and then we have them categorize them in within five minutes, they were able to look at the work and put them into piles. And they go, what we found was typically, there were two to five different ways the students were thinking about it. And then we ask the question, is this doable? If you can do this in five minutes? Can you use the information without spending extensive amount of time grading and recording and things like that? So our motto is work smarter, not harder? 9:28 Yes, I like that. So you addressed not only what and how to look at it, but also how to make it feasible to do so? Absolutely. 9:39 I think that's a critical part of actually any innovation going into the classroom, regardless of what it is. So in terms of building on the research literature, so we looked at what the article so the formative assessment cycle, but then we had to go back into the research literature and look at and say What about formative assessment? It's really important that we address with the teachers. So in the research literature, they talked about disciplinary knowledge, that formative assessment is not something generic, to really be effective, it needs to be targeted to the discipline like, so what does the student understand about the fractions? versus just looking at General strategies? So that requires specialized knowledge about what does it mean to understand the fractions one half. So if the teachers only have procedural knowledge, they can transfer that into the classroom. The other body of research we looked at was on professional noticing. So teachers need to kind of notice students strategies, but they can do that if they don't understand what they're looking for. So that's, those two 10:50 are kind of addressing both knowing the mathematics well enough to be able to see it, but then also being able to see it when it's happening and knowing what you're seeing. 11:02 Absolutely. Because if you're not seeing it yourself, you're not going to notice that in your kids. The other part is the learning trajectories research, in terms of how do you use How do you support learning of students? And how do you support the learning of a whole class of students? And then there's research on test design. So all these bodies of research are kind of like pixels are not necessarily even in the research literature, we were looking for frameworks there really doesn't exist. In as you said, it's much easier when you look at most research looks at you know, one aspects of these things, but it doesn't really address. How do you pull all this together in practice? So that's kind of where we were going with our work is green in a pragmatic way, how do we support teachers to improve student learning? So that's the bottom line. 11:58 So let me try to see if I can summarize. So you drew on content knowledge and specialized content knowledge that teachers need, you drew on noticing, and you drew on task design and formative assessment literature and trying to put all of those together to help your teachers learn how to do formative assessment within, like maybe done one lesson, but also within the whole school year? 12:31 That's right. And so the innovation is kind of looking at the PD framework, in terms of how do we design professional development that actually integrates this, give 12:43 us a quick rundown of that, 12:45 what is 500 kind of give you kind of an overview, and then I'm going to go a little bit more in detail, and explain what that PD framework is, and why it's necessary to kind of think about it. Like, as I mentioned, all those different pieces of research that I mentioned, are like pixels of knowledge. But if you're trying to it's kind of like instruments in an orchestra. But if you're trying to create music, you need to know how these all fit together. So when we in the Nevada math project, what we had was we had the entire team come together for researchers, professional development coordinators from the district, so we had like a roomful of graduate students. We had a roomful of 25 to 30 people sitting there and going, what is how do we define professional development so that it integrates all these pieces together? That process actually is documented in a book that I wrote on leaders who lead successfully and conducting productive meetings about getting that knowledge together, the PD framework that we came Wait, what 13:51 is the book called 13:52 leaders who lead successfully and conducting productive meeting? So it's really kind of looking at that book actually came out of the Nevada math project is, how do we all work together from researchers and practitioners to improve teaching? How do we work together? So that's like a different, it's not part of this paper, but that contributed to the design of the framework. So I had to credit the team for coming up with this framework for this is a collective work. So the professional development design, when you look at it is if you're looking at a terrorist and poppin cycle of assess, interpret and use. This is really complicated. It's not just going in teachers to look at student work. As we mentioned, the research before you need to figure out first of all the disciplinary knowledge, and teachers need to have a reason why do I need to change what I'm doing? Yes, right. Like for a teacher, I'm already overwhelmed. Why do I need to do this? So somehow they need to know that their life would It's so much easier if they do this, because it's gonna kids are gonna learn faster. And for them to be get that buy in, we needed to kind of put them in a situation of a learner. That's why in the framework, we say, teachers learner, so that they experience learning from the perspective of a student. And during that time they explore what conceptual understanding is what disciplinary knowledge is. So that's where the content based speedy helped from that they need to look at, okay, if this is what it takes for me to learn, how do I look at student learning? Where does learning look like? And that's where the professional noticing of strategies. So our PD focused on the Common Core standards. So we kind of designed it around that content, then looking at at after teachers explored the mathematics themselves going, Okay, where does this look like with students? That's simply to assess student reasoning. However, it doesn't give them anything else to go, Okay, how does this fit into my curriculum? And how does this fit into what I'm required to teach? So then the second part, he talks about interpreting? So teachers have to make connections between the disciplinary knowledge the student thinking, to how does this fit into my standards? How does this fit in with my curriculum? So there's that bridge between those two, that doesn't naturally happen? So that's like a different body of knowledge. So looking at standards, how does that fit in with the disciplinary knowledge? What do I do if the lessons doesn't fit? where the students are at? How do I adapt? The one of the biggest fear is like, well, if I don't cover everything in my curriculum, you know, they're not going to get it. So kind of being able to step back in the curriculum and go, if I spend a little bit more time, say, on place value here of when they're doing a, you know, two digit addition with regrouping later on. It's just they're going to get it faster. So it really becomes making coming to pacing decisions, and giving themselves time going, it's worthwhile to spend time doing this. So it's about being able to adapt the curriculum, then following it. prescriptively. But 17:28 that requires, like a fundamental understanding of how things build on each other. 17:34 Right? Yes. And that lays the foundation for learning trajectories approach. If you do not know where your lessons are going on how they build on each other, then that information you collected on formative assessment is not very useful. So that's like a critical piece, what tools do I have? Do I need to supplement it? Is that adequate? 17:57 Yeah, let's quickly shift to the other framework, and then we'll get to your research. So I 18:02 would just want to add, the other piece of interpreting is the use. So how do I use this to test design? So going to the other framework, it's pretty much that framework is laid out. So if anybody needs to use it, there's a book called work smarter, not harder that I it's a revised version of what we use with teachers. But the beauty of that is its simplicity. That framework, or that book has tools that are research based, that addresses the process of teaching. So it's how do you set a framework is also attached in the article as an appendix, right? That's right. And they can download it. And there's links to the blogs and stuff with resources to it. The simplicity, so you do all this stuff. But I have to say, when I first started working with teachers, I literally have pictures of car loads. We had a couple of cars with car loads of crates of journal articles, this travel. That was my academic upbringing coming, I was so proud of myself for that. I'm going to get there and read as many articles as I can. But the reality is teachers don't have time to process and then they don't know how to use you don't have time to read all the articles. Basically, they would say this is fine and good, but what do I do? And so if you really wanted to change practice, you really have to tie it into the daily work of teaching which is setting up the physical space, developing classroom routines, they can all relate to that lesson planning, facilitating discussions, and reflecting now these are things that teachers naturally do, but how they go about doing this things and tweaking it is what makes the difference. 19:53 I have to say when I read through that framework, I loved it, the inclusion of the setting up the space I don't know that I've seen that in a lot of other frameworks. And I was wondering because you say there's routines Do you in your book? Do you have a routine specified if 20:10 everything is, like, if you're going to the book, it gives strategies, they can go to the end of the chapter. Because one of the things I learned, working with teachers to get it, I tried to write the thinnest book possible. They want to know, okay, bare bones, why do I do? Why do I use and that's taking that research and kind of these strategies has been fine tuned by teachers over 15 years, I would give an example like for the whole class discussion, we have a framework of three levels of sense making this is easy to remember. First level, get kids to share their thinking, second level, analyze each other solutions. Third level. So what's the big idea that gets should get out of it, just in these three steps is simplified, really complex body of research into something that's doable. And I think that's something really critical when working with teachers, is not to overwhelm them with too much information, but to simplify as much as they can so that they can go, Okay, I can do this. I can try this. 21:23 Yeah, I really enjoyed reading about your three levels of sensemaking. And they map really, I think the research talks a lot about the need for private thing time for like small group sharing, and then for whole class sharing. And with that aligns to your three levels. Let's get to your research questions. And the evidence that you had to report on your answers. 21:46 Our first research question really focused on the teachers themselves? How are they describing what their use of formative assessment is? Because really, it comes down to what is their voice in this, we have to understand their practice from their perspective? And how are they interpreting what is going on? So it's also giving voice for the teachers? So it's their voice coming out in this, how they understand formative assessment? How are they defining these different aspects of it? And what do they see as the use of it? And the second question really focused on what are the shifts that teachers are relating in terms of formative assessment? And that comes down to is the PD framework being effective? So those are the two questions that we really asked. I'd have to say design based PD. So the first question was influencing throughout the PD, the next steps of decisions of what are teachers understanding? And then how will we adjust to adapt to that? And so just to talk about one aspect was, we know from formative assessment, you need a mathematical strong math content knowledge, in order to be able to even know what to assess what are the big ideas. And so the summer session with the PD spent a week developing the content knowledge and the pedagogy. And then when they came back for one of their sessions with formative assessment, and each time, we're putting out these teacher questionnaires in order to understand where they are and how they're understanding aspects. So after our first follow up meeting in August, they brought in their own assessments, we did give a common assessment because teachers were spread out and 23:34 had different when sorry, when you're talking about assessment, you're talking about student work right at the back. 23:40 Yeah. And so what the design had happened was we had a common assessment that they were giving to their students, knowing that they hadn't necessarily taught that section yet. But it would give common work for all the teachers to bring in and look at together within their grade levels. So with the math content knowledge being built, when they came back in and looking through video and listening and doing the field observations, they were not, they were still stuck on analyzing right and wrong questions. So this child had three wrong, this child had seven wrong so he's mastered it. And it was that PD design then that had them going back that Rooney had mentioned, where they went back and they sorted them into piles, looking at just one question in particular, and coming out with now we have three misconceptions that we're looking at three different ideas and looking at changing that perspective from right and wrong answers to focusing on what do the students understand about this mathematics and what ideas and concepts are they bringing with them to tackle these problems? And that was an interesting analysis as we're coming back and understanding the teachers going math content online. did not support this shift happening and their use of this tool to be more effective. So we went on and those questions informed some of the design decisions to really do you want to talk more about that. 25:14 So because shift that happened with the procedural to conceptual adaptive decision making, one of the things that was challenging that we had to address was a managing time, lack of common curriculum for planning. Those are some of the things that we had to kind of work through in terms of how in a design research approach, we, we expected all the teachers to come into the PD session, even though they were from the same district and be able to talk about the same curriculum. And we found that was not the case. 25:50 Yes. 25:52 And so that was something we needed to figure out how to address. And that's where we ended up developing some common assessments for them to talk about. 26:00 So summarizing your findings, was that most of the teachers shifted from looking at right or wrong answers to understanding why students gave the answers they gave. 26:14 Yes. And Linda, would you be able to say the percentage of 26:18 Yes, so our pre PD, we sent out a teacher questionnaire, and on that one, about 24% of the teachers really focused on conceptual understanding, and post professional development at 1%. were mentioning, really focusing on conceptual understanding. And another interesting one, too, is is still using this formative assessment into your planning, like making it part of your teaching cycle, not something that's done at the end, and then being hit with Oh, my goodness, what do I do with this one, my pacing has to keep going forward. And so a lot of the teachers in the beginning with the formative assessment, it was mainly used as pacing the teaching, they were already planned on doing so that was 49%. Only 13%, were really using it with that whole unit aspect within their lesson design. But by the end, they had post PD, we have 57% of the teachers were mentioning, using it within it to get an idea going into a unit, what misconceptions they were going to be addressing. And they were talking then about planning the intervention for the misconceptions they might encounter before beginning a unit so that they were not backwards planning or not. It's not backwards planning, but trying to take care of something as it was happening in a reactionary mode, but more being proactive about what they were doing what their students. 27:41 So if I were to summarize what you said there was a shift in what they looked at, and shift in when they looked at it and how they used it. Yes, I don't think that we have in our conversation yet talked about the length of the PD, if I remember correctly, it was a summer session, like a one week summer session, and then three meetings throughout the year. 28:05 Yes. The for the purpose of the study. That was the data we collected for that year, but it was part of a three year project. And also, the study reported on one site, but we actually did the entire state. And what's really interesting in terms of anecdotal information was that as a state, the student achievement scores went up according to nape. And Newsweek reported us as the fastest improving state, we still have a long way to go. I can't claim credit that it was a project, but a project was the statewide intervention at that time that was statewide, and we had representation from across the state, which is kind of exciting news. 28:54 That's super exciting. Okay, so I was just asking, because if somebody goes and get the article, there's really a few things that one could gain from that. One is the structure of the PD, which I think the for the purpose of the paper, you looked at summer workshop and three follow up meetings. Yeah. But then there's also the framework of what to do within each lesson. And for both of them, you give kind of an idea of how might one might implement it. And I think you mentioned throughout our conversation, several other sources that teachers could turn to, to get more information. So maybe let's close up with what are the new contributions of the paper or your work? And if anybody wanted more information, after they read the empty paper, where could they go? 29:47 I think the biggest contribution is that if you really want to change practice and student achievement, you cannot focus on a pixel. You need to think about the symphony how all these pieces fit in. And it needs to be done in a manageable way. So you want to make it all connect and simple simplicity is the key to being able to do define effective PD and support teachers shifts. So I think those two frameworks really makes it simplifies it for the PD facilitator, it could be I actually use this in my methods classes as well into my methods class design, where you can get information Nevada math project.com has a bunch of resources and links to that. There are several books that came out of the project that can be helpful for others trying to do similar work. One is work smarter, not harder. A Framework for Teaching and Learning, it came out of a joke going, what's smarter, not harder. Yeah, 30:54 it's a good title. 30:56 Because we were watching a video from the term floods and study of this teacher really struggling to support student learning. And it's really exhausting to watch as to how much effort he put in, and then we can trust it with another video that the students are doing all the work and talk about which one is there more learning taking place? And, you know, can you work smarter. So that's, that's like a takeaway or a contribution for professional development. And for teachers. I also developed a blog around this framework. So this actually has over the years expanded to a lot more resources. So if anybody has resources, they can email me and I will link it somewhere. What's the blog? It's math discussions.wordpress.com. And that's linked into the website, too, with the mte fact. 31:53 Well, thank you both so much for joining me today. I had a great time talking to you. 31:59 Well, it was a pleasure talking to you, too. 32:02 Thank you for having us on here. 32:05 Okay. For further information on desktop. You can find the article on the mathematics teacher educator website. This has been your host, Ava Sennheiser. Thank you for listening and goodbye.