Eva 0:00 Hello, and thank you for listening to the mathematics teacher educator journal podcast. The mathematics teacher educator journal is co sponsored by the Association of mathematics teacher educators, and the National Council of Teachers of Mathematics. My name is Eva Thanh Heiser, and I'm talking with Amber Candela from the University of Missouri St. Louis and Melissa Boston from Duquesne University. Hope I said that right. We will be discussing the article centering professional development around the instructional quality assessment rubrics, published in the June 2022 issue of the mathematics teacher educator journal. We will begin by summarizing the main points of the article, and discuss in more depth the lessons they shared in the article their successes and challenges, and how these lessons relate to their other words, Amber and Melissa, can you briefly introduce yourselves? Amber Candella 0:53 Hi, Amber Candela, Associate Professor of math education at the University of Missouri St. Louis, which are also called M Zol. So if I say that, how we say our university, Melissa Boston 1:04 Okay, Hi, Melissa Boston. I'm a professor of education at Duquesne University and thinker. And also currently on the NCTM. Board. Eva 1:16 Excellent. That's an interesting job. So let's jump into this article. Can you give us a brief summary of the article including the results. Amber Candella 1:27 So what we shared is how we took rubrics Melissa's instructional quality assessment rubrics, and which they had typically been used to rate teachers for research projects, we wanted to use them with teachers and professional development. So we share how we took this research tool and use it as a professional development tool with teachers to support they're thinking about implementing tasks, asking questions in the classroom, how they had students talk to each other, how they talked to students. And then we sort of talked about how they changed it was over the course of one year, and how the teachers changed, which was actually quite varying per teacher, some teachers improved, some teachers did not improve some declined, and some stayed the same or improved on one and and not the other rubrics. So Eva 2:18 let me follow up on this really brief summary. Melissa, could you give us a really, really quick focus of what AI QA is all about? Melissa Boston 2:27 Sure, sure. So the AI QA rubrics follow are built very heavily on the work of Peggy Smith and Mary Kay Stein at the University of Pittsburgh and some of the quasar work that kind of showed the importance of of a task and how if you follow a task throughout a lesson that the impact that that has on student learning, so the AQa looks at the level of cognitive demand of the task, the level at which that task is then implemented in the classroom. And then certain other things that impact the level of implementation, more specifically, so the level of rigor of teachers questions, of students responses, and any, you know, work displayed by students. And then also a construct called accountable talk from the University of Pittsburgh Institute for Learning that looks at how teachers then follow up on student responses. And in the IQ A, we break them down into two groups linking so how students are connected to each other? And press how the teacher tries to build more connections amongst the mathematics. Eva 3:32 So when you say linking, does that mean students mathematical ideas, linking to each other? Or what exactly are you referring to there? Melissa Boston 3:42 It's more about people. So it's more about the building the accountability to the community. So sure, they're talking about mathematics. You know, they're not rather than the you're talking about what they had for lunch. But really, the linking moves are focusing on how either the teacher is providing opportunities for students to connect with each other, or students are making those connections. So things like, you know, I disagree with Johnny, because it's mathematical. But it's really looking for those connecting pieces. Cool. Eva 4:13 Thank you. Who do you think should read this article? Or who did you write this article for? Amber Candella 4:20 So one of the things that was really important to us was really getting that teacher voice in this. So like, yes, we brought this tool to the teachers, but we also wanted to understand how they experienced it. And so part of this was really to write it for math teacher educators and people who are providing professional development in schools to think more about like, how are we interacting with teachers? How are we engaging with teachers around these ideas that we as researchers think are important to do the teachers think they're important also. And then how do they take these things up? So thinking about like, here, we have all these rubrics, and then they were kinda like, they really liked the questioning rubric. So it's like, okay, you know, we want to bring all of these things, but maybe we can bring all these things to teachers, and they can pick one or two. And then you do that for a year instead of, I mean, we were very ambitious with what we tried to do in a year. Eva 5:16 So I feel like that's always like, hey, all right. Well, always ambitious. Would you say that the audience then is people who work with in service teachers are also potentially pre service teachers, Melissa Boston 5:29 to potentially pre service teachers. Also, our focus was in service teachers. And Amber needs to take more credit for this, you know, having this idea that often classroom observation research is something done to teachers, you know, we go in with rubrics, we observe, we collect video, we leave, we write our papers. And so she really wanted to not only engage the teachers with the rubrics and the constructs to see what they would find important and how they would implement it, but also like, how did the teachers feel about it? You know, how did they feel it helped their practice or connected to their practice. So I think that was a really important piece. I think I'd also add people who the article kind of shows rubrics used in a small scale study, where you know, that even though you can do stats, and you know, quantitative analysis with it, there's still something really important qualitatively and I think that that comes out in smaller studies, when you're really able to look at the lesson across the rubrics. So I'd say math teacher educators, also people who might be doing smaller scale studies, who might not think about using a quantitative rubric in ways that would tell you something qualitative. Eva 6:38 Okay. So what is the important problem, or problem of practice or issue that your article addresses? I think, Amber Candella 6:49 one, we really tried to highlight, if we're going to use these rubrics to rate teachers, we really should use them with teachers first. And so thinking about, you know, there's a lot of observational rubrics and math education. And I think it was a like, I think one of the unique parts about this was we used it as professional development with teachers. And so originally, this started from I did a dissertation study way back when, and I use the IQ waiver rubrics to assess whether or not my intervention worked. And so then, during my defense, one of my committee members was like, Hey, did the teachers have any idea of what they were being used to, like, rate them? And I was like, No, they did not. They had no idea what I was using, what the scale was I was using. And they were like, Oh, I think that that could be an interesting study. So then, when I was at my first university job, I was like, Oh, this would be a good study. And so that's actually when I reached out to Melissa, and I said, Have you ever used your rubrics with teachers and professional development? I think this would be a cool idea. Let's try it. And so I think that's sort of like, again, you know, teachers do feel like professional development is so often done to them and not with them. How are we working within our communities of teachers to think through what's really beneficial to them, because we do have ideas of of research, and what we think works best the teachers classroom, but teachers realities, and what's really happening might not be what we also think so I just think finding that teacher voice and our research is really important. Eva 8:28 You said a lot of things so long, I'm sorry. That's okay. But I think I'm going to try to summarize what I can remember. And then please add. So one thing that you said is that you took an observational rubric and changed it from just being an observational rubric to actually being the intervention or working with the teachers on it, which, in a sense, seems like, if we carefully develop these rubrics, which I'm sure this one has been carefully developed, right, Melissa, then then, then it seems like a lot of that work is already in the rubrics that we would want to do with teachers. So that seems like one of the big ideas. And then I feel like I'm also hearing from you listening to teach your voice. And I'm wondering, Can you expand a little bit how that plays in? How would Would that then inform the instrument? Or would it inform what our understanding of what they think is important? Amber Candella 9:30 It was never to inform the instrument, it was more to inform how we use it. And so thinking about so the I QA has a number of rubrics and thinking about, okay, maybe we don't want to use all of them with teachers, but which ones did they find the most beneficial? And in this case, it was the questioning rubric. They loved having that reference of the questioning rubric to think about their questions that they asked for students. And so maybe in the future, we would change it so we just did some interventions around the question and using the questioning rubric. And so I think just thinking about, we like to have these big things. And all of these rubrics are still great to rate teachers with. But as far as using them with teachers, some are more what from what our participants said a little bit more what the teachers find useful in their daily practice to impact what they're doing and shape what they're doing in the classroom. Melissa Boston 10:25 Just to add on to that, you know, what the teachers questioning rubric one of the teachers was commenting that they they're told ask good questions, or they just get a lot of general feedback around asking good questions, but they liked that the rubric had types and categories of questions and also numbers of questions. So you know, thinking about to get like a three and four kind of the threshold of higher forms of practice on the rubric. So so that that, you know, that gave them a threshold and something they could plan and incorporate both in the type of questions and the number of questions. Eva 10:59 Yeah, it almost seems more honest, right? To say, Hey, here's what we're using to rate you with, and give them a shot to actually know. So that's cool. So my next question is, how does this article build on existing work in the field? And what particular theories or previous articles do you work did you build on, Melissa Boston 11:22 but I think it builds a lot on the Task and Task implementation research. So again, pick Smith, Mary Kay Stein, a lot of the work that came out of the quasar project without silver, a lot of work since then, with the five practices with the GI Bill and things that they've done through the Institute for Learning. So I think the professional development an idea there, certainly professional development going on around tasks and task implementation, and, you know, five practices and questioning, but I think the the part that this adds is specifically using those rubrics in the professional development sessions, so you know, having teachers read them, having them write tasks, having them write videos, you know, rate questions from videos, and then using them on each other. And Amber was very intentional about building up like a kind of a developmental progression of having teachers watch each other's videos, so that it was it was more safe. And by the end, that was one of the things that they really liked. And in fact, they said, We wish we would have, you know, throw, we wish you would have thrown all the rubrics at us early. And then we could have continued to practice and watch more videos with them throughout, where we were kind of intentionally being, you know, developmental like a few at a time and letting it build up. So I think that's what we had. So we've built on a lot of the Task and Task implementation and PD research. And the piece that's added is was the specifically using the rubrics? Eva 12:43 And I think in the article, you're also you're building on other PD that has been used as our sorry, on other observational rubrics that have been used as PD, like Mary bicycles work, and then you expand a little bit on what you did that they didn't. Can you add a little bit of that Melissa Boston 13:02 the piece about? Is it like the reflecting with teachers, so having so getting teachers perspectives, and thinking about how to use that? So so that's another important component. So not only using the rubrics, but also getting teachers perspectives and thinking about I guess our next step would be then thinking about how do we use what we've learned now about their perspectives and what they found useful to, you know, to better support teachers or to, to have teachers engage with the rubrics that, you know, in ways that they found helpful. Eva 13:36 So I'm gonna lead into the next question with a more specific question. So the larger question is, tell us more about the innovation you created? And how does it address problem of practice? And more specifically, I'm thinking, if you could lay out because I, as I was reading the article, you just talked about how you brought in the rubrics one at a time. And then I think you did something where they watched each other's video, but not in the So can you just talk through kind of what the intervention was in summary? Yes. So Amber Candella 14:08 we started the summer and fall semesters with introducing the rubrics to the teachers and we had a monthly meeting. And we like so that so the plan, which we did stick to the plan was to introduce the teachers to a rubric every month. So we would watch videos, rate videos of teachers using the rubrics, and then the teachers were supposed to go that next month with that one specific rubric in mind and thinking about their practice with it. And then in the spring semester, we started to have them record themselves teach, and we started to do paired observations, reflection videos, and we started it within grade level. So they all had to come with a grade level teammate except for the middle school, there was only one teacher per grade level. So the middle school kind of acted as their own team, and then each of the three through five I have teachers, there's two, three to four and to fifth grade teachers. And so we really wanted to get into that self reflection, but also peer to peer reflection with each other using the rubrics to watch and rate the videos. And I did it more, I didn't want it to feel like it was an assessment. So it was very informal, like they went and they watched the videos together. And then they came back and talked about it, because I didn't want it to feel like I was evaluating them, or anyone was evaluating them, because it's very hard to watch a video with yourself and your peers to talk through it. So I was really trying intentionally to be like, you all, just go do this in classrooms, I'm gonna give you the space and time to do it, and then come back and reflect. So the data that I have from that is just their reflections. And they did talk a lot about how it's hard to critique each other. But what made it easier as they could point to lines and their rubrics and say, like, this is what it would have been to get a three, this is what it would have been to get a four, this is what you did. And this is how you could make it better. So that was really beneficial to them. And so and then the other thing was they wanted to start watching outside of their grade level, they wanted to start learning from other teachers. And so one of the positive aspects that they said from the professional development, was getting to know people outside of their grade level team watching how different grade levels taught, because they were really able to learn from each other and that reflective piece. And so they're like, people always talk about good practice. But now we can see it and watch it and then talk with our peers and teammates about it. And so I just wanted it to be a space where they could be reflective. But then also like, people were like, do you have the data and I was like I don't, because I intentionally didn't want to collect that data. But then a critique we did get from the teachers was they also did want to watch the videos with us because they wanted the like, person who was really trained on the tool to also give them feedback. So like they liked the peer to peer reflection, but they wanted peer to researcher reflection, which we did not do, because we didn't I didn't even think to do it. So the next iteration, there'll be some more like teacher researcher sessions together. Eva 17:11 That sounds really fun. Sorry, Melissa, go ahead. Melissa Boston 17:14 Just wanna say I think it's important that Amber had a pretty close relationship with the school at the time of the study, I think she had spent this was at least the second year, right where you were going into classrooms pretty regularly. So you were a presence there, you know, the teachers were used to seeing you. And I do think there's that tension between like what we want to collect as data, collect and evaluate as data and giving the teachers the space to grow and not feel as though you know, they're being assessed and evaluated. So I've had some other projects now, especially over COVID, where we haven't collected data, because we just wanted teachers to, you know, feel free to be in the space. And certainly we want the data as researchers, but building that relationship sometimes is more important. Eva 17:59 Yeah. And also, one of the things you said is currently happening in one of my grants as well is where we usually separate out elementary, middle and high school, and we brought all the teachers together. And the conversations that happen, were amazing. And they really want start because they want to talk across grade levels. And that's I think, something that maybe we haven't paid enough attention to yet as a field. I'm not saying like always across, but at least sometimes. So that rings true with me what you're saying that learning from other grade levels. Amber Candella 18:36 Yeah. And that's, again, we wanted their voices. And so when we were asking about what they found helpful and what they found not so helpful, you know, I mean, we're getting critiqued as researchers, which I think is good. I mean, we be critiqued, critiqued below, we need to get feedback on what we're doing, especially, you know, as we continue to do it, and so that was like, huh, yeah, you're right. And they were, you know, they wanted more feedback in the moment because they did they were doing this to grow and learn as teachers, and they wanted more of those opportunities. And like, and Melissa said earlier, they also were like, Look, you we didn't want one a month, we wanted everything. And then we wanted to be able to pick and choose as we went through it had more time to practice with it. And I was like, Yeah, all right. Heard. Eva 19:21 Yeah. All right. So let's get to the researchy part of the study what what are your research questions? And how did you study or how did you show the effect as TIPNIS of this innovation or what did you find? Melissa Boston 19:36 So one research question was changed in practice? So you know, did participating in the professional development using the rubrics lead to changes in practice? And as Amber said, for some teachers, yes, very much. So, our group have out of the group of eight for which we had so we had 11 people total we had eight for which we had pre and post observation data for various reasons. But I think out of our group of eight, we could make an argument that five of them showed pretty important shifts to stayed the same. And one of the ones that stayed the same was already kind of at a very high form of practice. And one one did not. And so, you know, maybe that was just the lesson that we happen to hit. But in any case, that's, you know, that's the, what the data had. And, and here's where I think the qualitative part of the rubrics comes in is, you know, when you can look across and say, Well, sure, it may have only been a two or three point change. But what did that really mean in practice, and so then the AQa, if it's a change in points between like a one and a two, that doesn't really matter all that much, you know, it's still lower forms of practice, it's still not happening very frequently, or, you know, with a level of demand, or, you know, that's engaging students at higher levels, but it changed from a two to three or even or you know, or two to four, or whatever, even three to four means something very different was occurring. And so that's what we tried to So sure, I think, statistically, we did have a significant increase in point and like, overall points, if you looked at that, but I think what's more important was looking at those individual teachers and being able to say, well, this is how their practice changed. Amber Candella 21:18 And then our second question was around how they felt about their experiences. So that was really qualitative and what was their voice. And we've shared a lot of it as we've talked through this, but they gave us feedback on the structure of the professional development, how they wanted it, they wanted more feedback from us. And they liked like the questioning rubric, and the implementation rubric to that sort of, they liked the rubrics that came with sheets that had like checklists, because it really felt like they could use those checklists and practice and then they liked being able to watch the videos, we had example videos of teachers of different levels. So they liked being able to watch, you know, a teacher at every single level and make those comparisons and then reflecting with each other. Melissa Boston 22:01 And I'll say, just in our training materials, some of the videos of of teachers at different levels. We had we at one point had asked teachers, please, you know, please teach your procedural lesson, because we need a video of a lesson at a level two. So you know, it was the videos were collected, the sample videos were collected very respectfully. Eva 22:21 So I think you said five teachers made some pretty significant changes to didn't, but one was kind of topped out already. And one, change the other direction. Do you want to share a little bit more maybe an example of one of the five or some that shifted like so we can get a sense for what that shift looks like? And maybe some ideas why you think that one teacher didn't, Amber Candella 22:50 we used six of the rubrics to measure what they were doing. And I will say that actually, the so we talked about like the potential of the task or implementation, the student discussion, the teachers questions. So there were things based on teachers and things based on students. And so for a lot of the teachers who improved, that big improvement piece had to do with the student responses and the student discussions, there was more of that as they improved. And like the teacher questioning also was one that went up for most of the teachers. And the teeth, there was a teacher who declined, but it went, they were at a four and they went to a three. So they were still they stayed high level. Eva 23:32 Okay. So it might have just been like the particular lesson, right like that. Amber Candella 23:39 But then we had a teacher who overall declined across all of the categories. And so that was, you know, something that we had to work with. I mean, that this is where like, you analyze the data after the fact. And so it's like, I would have loved to have gone back to that teacher specifically and talk more about like how the PD went, and things like that. But again, we were trying to get lessons at the biggest lessons at the beginning of the year, and we got lessons at the end of the year, and the teachers told us when to come into the classroom. And it just could have been, you know, the day the lesson and, and really thinking through this, like one of the things I think we learned is, instead of just being these one point in time lessons that they need to be more of a sequence of lessons at the beginning and the end. But this was like a stilt like, you know, I was doing most of it in Missouri and St. Louis. And then Melissa flew up one time to interact with the teachers. And she was more consulting me on this. And it was like, yeah, if we had a big research team, of course, we would do all of these videos of all these teachers in the beginning and all the teachers in the end, but yeah, like this is my second year in the school my second year as an assistant professor. So it was a very ambitious project as it was. So I think that we did learn is that we do need the sequence, the sequences of videos to really show a more holistic view from pre and post so that you get, you know, because also, I mean, they're inviting us into their lesson. So they're, of course, you know, we hope that they're trying to show us their best lessons and their best foot forward and things like that. And so if we had like a week or three days, it would show more of like this, this happens regularly. And this is what's Eva 25:17 Yeah, practice. So I know with the M Qi, that is one of the things that they actually suggest is to have at least three lessons is that typically something with the AQa as well. Melissa Boston 25:29 So yeah, so for larger scale research, we usually suggest two consecutive days at each time point. And then that and even with a goal of only having one complete set of scores, but especially at the middle school, and high school level, if the lesson carries over to the second day, you guys started plan for those two days, or we take this out of scores that represents the teachers best foot forward. So if you do end up with two complete sets of scores, you take the one that represents their best effort, Eva 25:59 okay, Ambar. I'm Blake, as a second year professor to study seems like oh, my gosh. But that being said, if one were to redo it, maybe a second lesson could be interesting. But that was also not necessarily your main goal here. Right? Your main goal was working with the teachers, not the pre and post. Amber Candella 26:21 Right? Right. And that's the whole thing, right? Like, we learn as researchers how we're doing things like this was my, like, I wrote this grant my the end of my first year at will, and I got, you know, funded to do it. So I did it in my second year. And Eva 26:35 I And it's amazing. Amazing. Amber Candella 26:39 And so but it's like, you know, that's, I think that's the other thing, like, as I'm talking about this, I was like, Yeah, we're kind of showing you all the things that didn't go well. But I think we have to talk about though they do like, and we try to be very forward in this article. And yes, one of the teachers data we didn't collect, because I didn't have the sound on in the video. And did Eva 26:57 that happen to me in my dissertation study, too. And I think that's, like I liked it, you saying we need to talk about that, because that happens all the time. And people need to hear that. And, you know, then you just don't have that. And that's part of what it is like to or, you know, like I've showed up to schools and the campus was closed, or this the to school teachers were on a field trip, or you know, they forgot to ignite, there's just so many things that can go wrong, Amber Candella 27:25 right. And now, I always make sure that the volume is connected, and I'm back now before I do videos. And so this was a learning process for me as a researcher, you know, as I went through these things, and we are, we try to be very truthful about that. And again, we always write these things up years after they were done. But I still think it's important to talk about some of the hiccups that come along the way as you're doing this, because research is messy, especially in schools and teachers, and we're trying to honor teachers, and we're also trying to like talk about, we work with teachers whose their reality is very different. And, you know, yes, we had some different things go up and go down. And it just, you know, we also want to honor the teachers that we're talking about in our studies and say, like, this is what happened, and we're not sure why this happened, or why it didn't happen. But you know, but these are the voices we also bring from this. And these are the lessons that we did learn from this, because and really, yeah, Eva 28:22 by the way, I just want to say, it didn't feel to me that we're just talking about all the hiccups that fell to me, we're talking about an amazing study, where you know, something's, you know, like, in retrospect, you can always do things better. I mean, I think I'm a pretty decent interviewer. And every time I listen to an interview, I go, like, oh, I should have asked this, or I should have asked that, you know, it's just, it is how it is. Amber Candella 28:47 Right? And no, and we did learn things, and the teachers learn things, and we're very thankful. And I continue this relationship with the school, and those teachers that were in the study. And so yes, like, we, we did have some very positive aspects. And we do, and we talk about that. And that's what's so great about getting to do research, and then write about the research that we did, because we can share what we learned positive and we can share learned not so positive, and we all learn from it. I mean, that's why we read articles is to learn from each other. Melissa Boston 29:15 And I think it's gotten so much easier to video record lessons now post COVID Well, and even with you know, with the swivel and the our cameras and having teachers record their own lesson and send it in or even, you know, some variation of using zoom to record, so I feel like it's it's become so much easier and less intrusive, less labor intensive. So I think that's encouraging for no current junior faculty or people starting out who want to take on these ambitious studies. I think the other thing too, and this would have been interesting to talk with teachers about is that how Amber was saying, you know, their scores around their student what their students were doing improved. And so looking at how their improvements and questioning coincided, we won't say generate Did but you know coincided with these opportunities for their students to do more math and talk more about math. And you know how those those two things interacted? So I think that's it was a great takeaway from the study. Also, Eva 30:13 that is really cool. I noticed that to me when you were sharing before that it's both the students and the teacher side. So let's wrap up with summarizing what is the contribution that your article makes? Amber Candella 30:27 I think our contribution is really how we share our story of using something that has so often been used for only research on teachers to be used with teachers. And so this was an observation rubric. But there's a lot of things that we do in research, that it would be nice to see people start to do things more in conjunction with teachers and getting more teacher voice input into some of these professional development tools in how we're thinking about interventions with teachers and finding using their voice to help us as researchers also reflect on what we're doing. Yeah, I Melissa Boston 31:04 agree. I think sharing these things with teachers, like they said it gave them it, it gave them kind of a pathway for where to focus their time or energy and even thinking about, you know, if you're conducting an eight minute discussion, you don't need 20 linking moves, you need three or four. So even, you know, putting in perspective, and then being able to use that for planning. So I think just the contribution of what we learned from teachers, and then also how they found the rubrics helpful for their practice, I think makes a nice contribution. Eva 31:36 Well, thank you so much for joining us today. Amber Candella 31:38 Thank you for having us. Melissa Boston 31:41 Yes, thank you for having us. For further information Eva 31:43 on this topic. You can find the article on the mathematics teacher educator website. This has been your host Eva fan Heiser, thank you for listening and goodbye. Transcribed by https://otter.ai