0:00 Hello and thank you for listening to the mathematics teacher educator journal podcast. The mathematics teacher educator journal is co sponsored by the Association of mathematics teacher educators, and the National Council of Teachers of Mathematics. My name is Eva Sennheiser. And I am talking with a lot of people today, so I'm going to let them introduce themselves later, we will be discussing the article design principles for existing student practices. In a technology mediated environment. Published in the June 2020, issue of the mathematics teacher educator journal. We'll begin by introductions and summarizing the main points of the article and discuss in more depth the lessons they shared in the article, their successes and challenges, and how these lessons relate to their other work. Jennifer, Allison Lara and charity, thank you so much for joining us today. 0:51 Thanks for having us. 0:52 Thank you. Yes, 0:53 we're excited. 0:54 So again, I just go in author order to make it easier and have you each really briefly introduce yourselves. 1:01 I'm Jennifer. Love it. I'm an assistant professor at Middle Tennessee State University in the mathematical sciences department. 1:08 Hi, I'm Alison color. I'm an associate professor of math education at the University of North Carolina, Charlotte in the Department of Mathematics and Statistics. 1:17 Hi, I'm Laura dick, 1:18 also an associate professor in the math department at Bucknell University. Hi, I'm 1:23 charity Katyn, I'm an associate professor of mathematics education at East Carolina University in Greenville, North Carolina. And I have the longest department name, I am in the department of mathematics, science and Instructional Technology Education. Wow, that's a cool department. 1:40 Oh, and even though we didn't in the same cohort, three of the four of the authors are stars. So we'll plug STAR program also. 1:48 Thank you. Can you explain to us really briefly what star is it stands for service 1:53 teaching and research. And it's mentoring first through like second year new faculty. The program is sponsored by a mte. And it's a wonderful program to help new faculty form a community and help us understand the requirements of what it means to be a professor and these mentorships in terms of getting our research and our teaching and service off the ground. 2:18 That sounds fantastic. So if anybody's interested in this, where could they find some information? 2:24 I'm sure you can find it on amts website under probably the professional development tab at the top. Awesome. 2:30 Thank you so much for that. So let's jump into discussing the article. Can you really briefly just start by describing the innovation that you wrote about? Sure. 2:41 So the innovation is actually something we've been working on for a while now. We were in the midst of writing a grant proposal a few years ago. And it is the grant itself is about developing pre service teachers, knowledge of students thinking, understanding and learning when it comes to using technology. So when we were developing this grant, we were going to create learning experiences for pre service teachers and realized that we needed some kind of design principles for these materials of what we were developing what they would look like, and make sure that they were research based. As we were designing the videos, the tasks, what kind of videos we were capturing, that we wanted to make sure we had a set of design principles that was foundation founded in the literature. Specifically, we started with T pack, which is technological, pedagogical content knowledge. And nice is one of the main authors of T pack. And she specifically pinpoints four main things that when you're talking about that inner overlapping section of her Venn diagram, she talked about what four things all encompass t pack. And one of those is really just developing this knowledge of students understanding and thinking. So that was like our foundation that we were building from with our innovation. There is when we were designing these modules, learning experiences, whatever you'd like to call them. That was our goal was to develop that knowledge and our pre service teachers. And so we do that through studying video cases, and through the act of professional noticing. So that's kind of where the innovation got started. Specifically, in this paper, though, you'll read about how the design principles were used for something we're calling technological, pedagogical function knowledge, so TP fk so yes, more acronyms for you. But we think it's important as a group, that T pack is very general. And so when you're talking about t pack, if your T pack could be different for algebra versus geometry versus statistics, and then inside have something in terms of algebra. There are many, many topics that your content knowledge and your pedagogical knowledge and your tepeyac could be very different. So in this paper, we present what we call technological pedagogical function knowledge about what would be the components that are in service or pre service teacher to be able to, to facilitate that type of learning in their classroom. 5:25 So let me follow up with some questions. It seems like there is a larger idea you're talking about, which are these design principles, which would apply to any kind of task. And then you have one specific one in this article that centers around functions that is kind of ready for people to use, and you're using the article, you're using this specific task to illustrate how you applied those design principles. Is that fair? 5:55 Yep, that sounds perfect. 5:57 So let's unpack a little bit. For those of us who are not as technologically knowledgeable as the four of you, you referenced a Venn diagram that you also illustrate in a paper so people can go and look at that in the paper. But what makes this knowledge different from function knowledge, so can you just say a few words on if I have an understanding of content knowledge for teaching function, let's say that way, what in addition to that is included in the technological piece, one of the main thing is in 6:33 our Venn diagram, instead of the traditional Venn diagram that you see on the paper, or@tepeyac.org, we view your content knowledge as foundational. So instead of these overlapping, circles, we are more concentric circles. So we view that function knowledge is the most important part. And then we look at your technological knowledge in terms of function, and then t pack or not t pack TP fk technological, pedagogical function knowledge as the innermost circle, 7:03 the function knowledge is foundational in then give us like an example that so that somebody could wrap their head around what his technological knowledge could be, that's in this middle circle. 7:16 So the technological function knowledge, and that's one that's actually kind of tricky to be thinking about. But it has to do with the wave function might be represented, using technology, the way that you might interact with function using technology. So in the case of the applet that we present in this particular paper, it is designed to act as functions or not functions, but the understanding of the what is underneath the technological design of that how that was actually built would be that technological function knowledge, 7:53 why not a good example for our readers that are used to the graphing calculator, an example of technological function knowledge would be understanding when you graph let's say, the tangent function, and your ti, 83, and 84, graph all these vertical lines, to understand that those are the asymptotes and not actually part of the graph of the function. So that's extra knowledge you would need to have about how the technology, the technology is functioning, using function in multiple different ways. Yeah, to understand what you were looking at and know the limitations of the technology. And in those early graphing calculators, 8:27 it's actually the pixels that make you see the lack of pixels that make you see what look like vertical lines when there aren't actually vertical lines there. So that's very, very technological. And then the next the inner circle then is technological, pedagogical function knowledge, which brings all the pedagogical pieces into it. So the knowledge you need to make sense of student thinking in those technological environments. So the ways that students interact with that technology, the ways that interacting with that might inform their thinking about something, the ways you might leverage that technology to build understanding of something. So that's where the middle circle comes in. So we just become more specialized as we continue to move forward. Any builds on the previous, 9:15 the middle circle, just to be clear, it's not about how to program the technology, it's how to use the technology 9:24 to teach to teach to support students. Yes. Okay. Oh, that's mean going off against example, me knowing to I'm going to choose a, we're investigating the graph of the tangent function, being careful the examples that I might choose knowing what my students might see if they graph those being very intentional about questions I might pose, knowing what I know about the way that's going to be represented within the technology. So pulling all those pieces together to make the pedagogical decisions that I'm going to make. 9:57 All right, let's jump into a brief summary. Have the article. 10:01 Sure. So in this article, as you noted, we kind of use, we start off by talking about why we need design principles to do this sort of work, and then share what ours are kind of the literature that grounded them, why they were important for us, and then actually set out are five design principles. And then we provide an example where a module that we've designed using those material, those design principles, just you can see what it looks like, in context. And then we have a small study that we did, keeping in mind that our goal through all of this is to develop PS T's, our service teachers, technological, pedagogical function, knowledge. So that's what our goal is. So in this particular study, we actually studied that from the pre service teachers perspective, what did they feel like they learned through going through this going through this module related to the different aspects of function, technological, pedagogical function knowledge, so it kind of had three parts in the article you'll start out with here are the design principles. Here's an example of what they look like in action. And then we have actually used this with students. And here's what students are sharing with us they've learned through engaging in tasks that have designed with that had been designed with these principles in mind. 11:14 And we'll get to get into each of those in a little bit more detail. Who do you think should read this article, 11:20 this article was written for any math teacher educator that is working with in service or pre service teachers, an ideal situation, they're working to develop their PS MTS or in service teachers, knowledge of students understanding in specific to this article function. The thinking that the there the peak, Kate, she's me, the middle school students in this article, the thinking and learning in those technology mediated environments. And as we have said, this is specific to a middle grades function concept task. But as Allison mentioned, the design principles, and you said that also earlier in the podcast, you know, they could be used broadly. But this would be a way to read and learn about them and incorporate something that's already been made around the principles within your work. And then think about how it might impact your work further, 12:13 thank you, what was the important problem or issue that you are addressing? By doing this work and writing this article, 12:22 following up with what we were talking about with math teacher educators, being a good audience for the article, I think we could agree that pre service teachers are in service teachers both need experiences and thinking about how technology tools and learning in with technological representations, how that supports students mathematical thinking. And there just is not a lot of that out there currently. And so one of the needs that we meet is we're providing an example of that using the design principles. And the design principles help us use the concept of professional noticing to think about students mathematical thinking and technology mediated environments. So it addresses a very big need. 13:06 Let's talk a little bit, you just said that it addresses a big need, how does it work on prior work? And you mentioned a noticing work quite a bit. So let's just summarize kind of what are the bodies of work that you're drawing 13:22 on most of these, you've kind of heard sprinkled throughout probably, but we have three really big bodies of work that we drew on as we were working through this, the starting point being tepeyac, or technological, pedagogical content knowledge, as kind of our overarching goal so that that framework was our starting point and understanding what are the specialized knowledge is that we need to help our pre service teachers develop, we took niece's description of T pack that was specific to mathematics, but mathematics generally. And we operationalized it for the concept of function. And we drew on hollow brands and leaves work with statistical knowledge, technological, pedagogical, statistical knowledge, to be thinking about what those components would look like if you go if you operationalize them for very specific content. So that was one big area. Another area as you can imagine, we we use video cases throughout so we wanted authentic student work, we felt that was really important. And we know from the work of Sharon and her colleagues that using video with teachers it's not it's not trivial isn't it seems like we should be able to just throw some video up there but it is not trivial and they've done a lot of work with their work around noticing and classrooms and also their video studies about how to select video cases and what makes a good video case. So their work where they talk in video clubs about recommendations related to video selection on related to the window, the clarity and the depth of student thinking and also how they went and connected to recommendations by Schoenfeld that if you're going to be working with video that it's a really good idea to look For places where there's confusion or surprise, so we drew on that large body of work to be thinking about, how do we even collect? What would it mean to collect a really good example of student work in this technology mediated environment if we want to use video clips, and then from there, once we have these video clips, what we do with them, what are the pedagogical skills we're wanting to develop? That's where we draw heavily on the work of Jacobson, Philips and professional noticing of student thinking. And we've thought really hard about what does it mean to attend an interpret in a technology mediated environment, it is different than it is with written work, we've drawn a lot on their work, and also Wilson Halla, Brandon Lee, they did some there's only other place that we've seen significant work related to professional noticing of student thinking in a technology mediated environment. And that work actually happened. It was published before the Jacobs at work. So it's not called professional noticing there, but it's very similar. The piece that you'll see come out in this article is their push for the importance of providing opportunities for pre service teachers to reorganize their understandings related to what they're seeing in and understandings. So that large body of work around professional noticing, we use that to to guide how the question prompts that we pair with those carefully selected videos, with the intent of developing that technological, pedagogical function knowledge. Those are our a lot. But those are the three really big areas that were really, really important to undergird this project. 16:39 It's a lot. But it's also nice to have to work around it in prayer work, 16:45 right so that we build on solid foundations, we wouldn't have been able to do this without the important work and all three of those areas, it was critical for us to even know where to begin. So 16:58 now we come to the most fun part for me of the podcast, which is discussing the innovation. And I'm wondering if you guys want to do two pieces for this question, one, really discussing those design principles, sharing them and then discussing them and then the other one, to give us an image of this function application that you have, because it's really cool. And I think it'd be nice to kind of get an understanding of what those are. So yeah, let's hear a little bit about the innovation. 17:31 Jen and I have worked on that vending machine together since 2011. The design of that alone predates this work by quite a bit. And it's been a really fun journey. So as I was just describing kind of the literature that guides our design principles, we have five design principles. The first being kind of foundational one that pre service teachers need need to observe secondary students engaged in technology based tasks, those tasks they need to be of high cognitive demand. And the technology needs to be positioned in such a way that it is intended to develop mathematical or statistical understanding. That's our first one that this is an important thing that we do. The second is around the selection of the video clips and the associated written artifacts because sometimes there are written acts that go along with the video clips, they have to focus on aspects of student work, were there elements of confusion or surprise, as I mentioned before, what suggested by Sharon and Schulman Furthermore, we're also losing Sharon's recommendations as we're narrowing in on those clips, as selecting things with paying attention to the window of student thinking, the depth of student thinking, and the clarity of student thinking that is available in the short clips that we do choose. And finally, we want to make sure that in those clips, we're capturing interactions among students, any gestures that they're using is often when students are working with technology and talking with each other. They're pointing at things on their screen. They're gesturing to explain things they're seeing on their computer screen. So it's really important that that piece is also captured in the video. So we find that including things that have picture and picture along with screencast of what's happening with the technology is really important for providing that window into student thinking depth of student thinking and clarity of student thinking. We have to be able to see all of those things. At that point, we have a task, we have some video clips, and we start thinking about how are we going to put all this together in a lesson. We believe design principle three says that our pre service teachers must complete the task first as a learner, it's really important that they step back work on that task, discuss the mathematics and the technology involved before they step forward. To analyze student engagement with it. We also then guiding design principle number four is about the guiding questions that accompany the video clips that we're going to share with them. Those are designed based on Jacob's ELS frameworks. We want to make sure that we explicitly ask or provide prompts that suggest Some way, but we want our pre service teachers to attend to student thinking, to interpret it sometimes to predict their student engagement, which means they're paying attention. We're asking them to pay attention to not only what the students in the video clip are saying, or writing down, but also to what they're actually doing with the technology, and how that technology reacts to the things that they do. So what they're actually seeing on the screen, and then how they react to that as well. So our questions are there to, to support them engaging in those important pedagogical practices. And then finally, on principle number five is that it's really important that pre service teachers have an opportunity to reflect on their own understanding of tepeyac gist of the content but a larger picture, their pedagogical, technological, pedagogical content knowledge in that particular content, regardless of what it is, in light of their examinations of the student practices. So we want them to have opportunities to reconcile their observations and inferences, even with their own understandings. So those are big, five design principles. And I'll let Jen describe them with the innovation was what this looked like in practice for Awesome, 21:08 thank you. 21:09 So the innovation we call vending machine, which I'm actually going to correct, Allison, because it was 2013. She's making us all two years older than we actually are. So fix that one, because I wasn't a doc student yet. And so I was Allison's doc students, NC State, that's how we got started and building this relationship, working on the original PT Mt. Grant, which is preparing to teach mathematics with technology, which we'll bring up again, probably because we, that's our current grant also. And we'll plug that some more. So we were sitting in a room one day, and we were trying to come up with a task to probe at pre service teachers knowledge of function. And Allison when she was a high school teacher, and I did as well. And many of us did, you introduce function using some kind of idea of a vending machine or a guess your rule machine that you put a number in and a number comes out. And can you guess the rule, some teachers even build little vending machines, and they put an object in and something else pops out. So we were wondering how we could build that idea. And in a GeoGebra sketch, and then at the same time program, this vending machine to push at our pre service teachers understanding a function. So that's where it all got started. We have been through several iterations, there's actually three versions, it took three versions to get to where we are now for the pre service teacher version. And we also have a middle school version. So the difference is, is the middle school version introduces the concept of function, specifically aligned with the eighth grade standard and Common Core, the high school pre service teacher version is more of a refresher like you've learned about it. And now let's take a deeper dive into identifying and classifying functions. If you want to know more about our middle school version, we just recently had a mathematics teacher, teacher learning and teaching pk 12. That is a mouthful mt lt. Article come out in 2019. That details how we worked with the middle school students 2020. Or Yes, it's 2019 in our Google Doc, as I'm reading right off the screen 2020. So earlier this year, pre COVID, the article came out. So that's what it is. And so in this article, we're actually talking about that the pre service teachers have already engaged with the pre service teacher, one and push their own learning, like Allison was mentioning and design principle three. And then we have used the middle school version with students who had never learned the word, the term function before. They may have seen it in their textbook, but they hadn't actually explored what a function was. So we carefully videotaped many, many pairs of eighth grade students working together on the applet. And then our pre service teachers watch clips of two pairs of students engaging with the applet. So I go last night, what is also so yeah, is you're going to poke at that. So if you click on the link in the article, and it takes you to the GeoGebra sketch, it's a little elementary looking, but there are these boxes, and they have four buttons on them, red Cola, diet, blue, silver, mist and green do. And when you press one of the buttons, an image of a can appears, or no can, or sometimes two cans. And so you're asked to identify if the machine is operating as a function. The design is specifically built again, for the middle school students develop that. So in the beginning, we say, machine a is and machine B isn't. And so they're supposed to engage and come up with a conjecture of why a would be a function and B wouldn't be a function. And they do that for a few pages. And then after we've had a class discussion about their conjectures, the middle school students go in and explore even more machines. So what we mean by even more machines is we have a machine that produces all green cans. So no matter what button you push, it will produce a green This was to get out in the literature the misconception that constant functions aren't function. So when they see a horizontal line, and the literature students think that that isn't a function, there's another machine that when you press the Silver Mist button, it does not produce a can. So kind of mimicking when a function is not continuous. Because we found with our pre service teachers that when they're in their upper level mathematics courses, they kind of forget that functions don't have to be continuous because they have been working in this world of continuity in their proof writing courses. So there's the series of vending machines, I'll let you go on and play and in the article has the link to the different machines. And again, the MTL t just different ways that we could challenge our students understanding a function, and then just what is the domain and the codomain trying to develop that knowledge and simultaneously developing that TP fk we've been talking about as well. 26:00 So I have been playing with the applet while you are talking, just kind of seeing if I can find it when it's really fun. So I would encourage people to just click the link and play Sorry for interrupting you know, you're 26:12 good. We actually I'm gonna say this because Alison would want to plug it to last year right before we were presented at nctm. Nashville, a regional conference, there was just a teacher who tweeted about how they were going to use vending machine and their class of vending machine, not our vending machine, and their classroom to introduce function. And Alison had so much fun just pushing on the idea with that teacher, like what if two cans appeared? And so there was actually a long conversation with MTS that we're following the half MTB OS in service teachers, could that be a function and they were pushing their ideas of if it produced two of the same color versus two different colors? So we really like that that's one of our things that it's really fun to just dig into. And even someone with a PhD in mathematics, we've actually presented this to a math department before and just them sitting there thinking and like having fun trying to determine if they would consider it a function or not. So let's see where was I think it was on design principle. Four. Okay, so they then are gonna we develop tasks after we have these carefully selected videos, specifically, in this paper, we were looking at just the attendant interpret Laura, yes, the attendant interpret part of noticing you, we have to look like this, because you know how every research team is, you have three or four papers happening simultaneously. And you have to keep track of which paper we're talking about. Finally, the fifth design principle is we ask them after they've watched the video of students, after they've attended them interpret to them reflect on how their own knowledge has gained not only their content knowledge, through engaging with the function machine, so possibly thinking about what is the domain and the codomain? And what are what's possible, but also looking at how the students interactions help them. So for instance, as Allison mentioned, how you click your buttons, and your vending machine can tell us a lot about your understanding of function. So if you're checking for consistency, that you're checking that every time I hit red Cola, a blue can come out, you're telling us that you've developed that foundational idea of what a function is. And if you just check each button once and move on, you're telling us that you're missing that idea of consistency, and you may be looking more for colors to match and stead of the foundational idea. 28:41 So these videos that your pre service teachers watched, are those available somewhere, 28:48 they will be soon. Okay, those two particular are linked in that paper, I believe, then there's a few that are also linked in the MTL t papers just for some examples of student work. All of this is part of a larger project, where all of the videos will be housed on the project website, as well. So we are in the process of that. 29:08 What is the project website? 29:10 That's a great question. So this is part of the larger p TMT project work that Holly Lindley and Karen Khalid brand started well over a decade ago. And this particular piece sits alongside of the algebra materials that hollyland Karen and I developed back in about 2010 ish 2015. So our new project is a companion to the algebra and function materials. That's why our focus is on throughout these will be on algebra and function, the website, the portal will, where everything will be housed can be found@go.ncsu.edu, slash PT empty, 29:50 and we actually have three modules getting close to final draft form. And we're hoping for those three modules to be going live around the time of mte 2021 conference coming up, hopefully, this will bring this January February time period. Yeah. 30:08 So I just went to this website and it says register for access. 30:13 Yes. So the portal itself is part of the Friday Institute at NC State University. So you do have to register because they are keeping just track of who's engaging with it, it is free. It's just a short process to go in. But there is no charge at all. It's just making sure 30:29 if I plan on using this task, and I would like to use some other videos, I should be able to find them here at some point, 30:37 yes. And our intention is that they will be packaged in a way that you can easily download what you want to use with your classroom in nice sized chunks, or really large chunks depending on what you need. 30:50 Okay, so I think we wrapped our head around what your innovation is. Now let's talk a little bit about what research questions you asked to studied in innovation and what you found. 31:04 So our research question is focused on the pre service teachers reflection after completing the module. So we asked in what ways do the pre service teachers articulate their developing technological pedagogical function knowledge when asked to reflect on their learning as a result of completing the tasks? And so to study that we analyzed all of their written responses? And to the question, so the question they actually wrote an answer to was how has your own understanding of function been influenced by thinking about how the middle school students develop the concept of function, but they answered that again, at the end of all of all of the activities that they had completed in this module, and then we mapped their responses on to the four components of the technological, pedagogical function knowledge. And so of those, we found that 82% of them reported developing their own knowledge of the function concept, and 70%, directly related to their own understanding of the definition of function. So we know that engaging in this, you know, helped with that outermost circle of developing the pre service teachers understanding of function. But then 76% of them also reported developing a repertoire of strategies to use like really saying, I understand how I could use this tool, this technological tool to develop the understanding of function with my students, 53% of them discuss developing their understanding of student thinking. So they specifically reflected on how they are now thinking about students thinking with the technology. And then 42% talked about thinking about how the technology is supporting their thinking. So not only understanding the students thinking, but then thinking about how the technology itself is developing the student's understanding of function. And then the last component that what is in that framework is taking a critical stance about the technology 13% of them kind of took a critical stance within their reflections. But overall, 69% of all the pre service teachers reflections had two or more of those components. So we felt like that was definitely evidence that these design principles were working to help develop the pre service teachers, technological, pedagogical function, knowledge. So all that correctly. You. 33:26 Let's wrap up by talking a little bit about the contribution to the field. What do you see that this paper or this innovation contributes to the field, one of the big things that we've been working towards largely 33:40 in the last couple of years with this work is addressing the need to better understand what professional noticing a student thinking looks like a technology mediated environment. Part of that is there hasn't been a lot of research done in that area. And on top of that, there aren't a lot of authentic artifacts even use with your students, finding video exemplars where technology is being used, like math, action technologies are being used in mathematically powerful ways. They're hard to come by. So we have kind of those two pieces there. One hoping to really inform the field about what it looks like to notice in these situations. And we're continuing to work on that and other things we're doing now. And also by providing design principles for others that we hope will help us create some more of these artifacts for the field across grade levels. We need work like this so that our pre service teachers and in service teachers who are learning about how to engage with students in their classrooms have examples to look at and learn from and think about, you know, the way we pay attention to student thinking in this environment, these environments are different, we need need to actually have ways to work on those skills. 34:51 And as Alison mentioned, that there's a lack of videos, and that involves technology and mathematical tasks. Just as many of us know, there's a lack of balance. At the middle school and high school level period of students engaging with mathematical tasks, we know that there is a lot more at the elementary level to use and content methods courses. So even though we're focused on technology, we're just hoping to try to add to the availability of open educational resources that have student work and student videos at the middle school and high school level as well. Well, 35:22 thank you so much for joining us today. This is really amazing. And I feel like learn, in addition to reading this article, and the website is a really real resource. 35:34 So I just wanted to add that not only are we planning on putting things on that website that Alison mentioned, but if you go there now and register, you'll be a part of our newsletters. So when new materials come out, you can be informed of when we start, our goal is to have seven modules all together. So again, as we mentioned, three of them are getting ready to come out hopefully in the next six months or so. But if you're registered there, you'll get a notification. And you'll let you'll know when the other modules are coming out as well. 36:04 to piggyback on that, go look at our last newsletter, because in there was an example of some of the tasks from one of our modules, our module is focused on rate of change, where we have some tasks and video of students working around the working with the Desmos function. Carnival tasks are one that many of us have looked at. And if you haven't checked it out, it's wonderful. But we've got an example of one of our modules there right now that you can take a look at as well in the newsletter, Allison and Jen is has our information where they can also follow us on Twitter. 36:35 Yep, fantastic, full of new resources. And then you can use the task and tweet out about it and have a discussion with everybody. That would be fantastic. Well, thank you so much again, and for further information on this topic. You can find the article on the mathematics teacher educator website. This has been your host, Ava Sennheiser. Thank you for listening and goodbye.