Assessing Teaching with Beate Brunow and Shawn Simonson

In today’s episode, we dig into an important question for higher ed: How can we improve the evaluation of teaching? Researcher Corbin Campbell was quoted in a Chronicle article recently, saying, “Folks will say quality teaching is hard to measure. Quality research is hard to measure, but we do it.” I’m excited to bring a conversation with two academics who are contributing to efforts on their campuses to assess and evaluate teaching in more meaningful ways.

Beate Brunow is the associate director at the Schreyer Institute for Teaching at Penn State, and Shawn Simonson is a professor of kinesiology at Boise State University. Both have been involved in the development of new frameworks for defining effective teaching, and both are using those frameworks to change how teaching is evaluated at their institutions. We cover a lot of ground in our conversation, and if you care about teaching and learning in higher ed, I think you’ll find it interesting.

Episode Resources

·       Penn State’s new Faculty Teaching Assessment Framework, https://www.schreyerinstitute.psu.edu/assessment_of_teaching

·       “Establishing a Framework for Assessing Teaching Effectiveness,” Simonson, Earl, & Frary, 2021, https://www.tandfonline.com/doi/full/10.1080/87567555.2021.1909528 

·       “American Value Good Teaching. Do Colleges?”, McMurtrie, 2023, https://www.chronicle.com/article/americans-value-good-teaching-do-colleges 

Podcast Links:

Intentional Teaching is sponsored by UPCEA, the online and professional education association.

Subscribe to the Intentional Teaching newsletter: https://derekbruff.ck.page/subscribe

Support Intentional Teaching on Patreon: https://www.patreon.com/intentionalteaching

Find me on LinkedIn and Bluesky.

See my website for my "Agile Learning" blog and information about having me speak at your campus or conference.

Transcript

Derek Bruff 0:06
Welcome to the Intentional Teaching, a podcast aimed at educators to help them develop foundational teaching skills and explore new ideas in teaching. I’m your host, Derek Bruff. I hope this podcast helps you be more intentional in how you teach and in how you develop as a teacher over time.

Last September, the Chronicle of Higher Education published an article by senior writer Beth McMurtrie titled “Americans Value Good Teaching. Do Colleges?” The subtitle was telling: “The evidence doesn’t look good.” In the article, McMurtrie explores a number of signs that colleges and universities don’t really value good teaching. Most college professors, she writes, don’t receive much training in teaching during grad school or on the job. The instructional work force is dominated by contingent instructors with heavy teaching loads and little support. Teaching is valued in word, but not in deed, with research counting more during promotion and tenure reviews for faculty. And even when those reviews value teaching, they lack meaningful methods to evaluate teaching, relying instead on problematic student evaluations.

That’s a hefty set of challenges, but it’s also something of a roadmap for colleges and universities who want to take their teaching missions seriously. How can we better prepare and support faculty for what is, in most cases, their primary professional responsibility? How can we improve the labor conditions within higher education so that instructors have the time and resources to teach well? And how can we improve the evaluation of teaching in ways that promote good teaching? I’ve been fascinated by McMurtrie’s roadmap since it came out, and I’m going to spend some time exploring that roadmap here on Intentional Teaching in 2024.

In today’s episode, we’re going to dig into that last question: How can we improve the evaluation of teaching? Researcher Corbin Campbell is quoted in the McMurtrie piece saying, “Folks will say quality teaching is hard to measure. Quality research is hard to measure, but we do it.” I’m excited to bring a conversation with two academics who are contributing to efforts on their campuses to assess and evaluate teaching in more meaningful ways. Beate Brunow is the associate director at the Schreyer Institute for Teaching at Penn State, and Shawn Simonson is a professor of kinesiology at Boise State University. Both have been involved in the development of new frameworks for defining effective teaching, and both are using those frameworks to change how teaching is evaluated at their institutions. We cover a lot of ground in our conversation, and if you care about teaching and learning in higher ed, I think you’ll find it interesting. 

Well, Beate and Shawn, thank you for being on Intentional Teaching and glad to have you on the podcast today and talk about how we assess teaching and maybe how we could assess it more effectively. Thanks for being here. 

Beate Brunow 2:56
Glad to be here. 

Shawn Simonson 2:57
Yeah, thank you. 

Derek Bruff 2:58
Before we jump into the main topic, I'll start with my usual opening question. Can you tell us about a time when you realized you wanted to be an educator? 

Beate Brunow 3:08
Well, I can jump in. 

So I think for me, I've always been drawn to spaces that are somehow about learning and curiosity. But for a long time, I think haven't been able necessarily to label that that is that those might be educator spaces. So when I was still in high school, I was leading, for example, the Junior Fire Fighters and our volunteer firefighter group. And then I was a tour guide on an organic farm and I was a tour guide in a water treatment plant. So those were all always groups and spaces where people really wanted to learn something and were really curious about stuff. And that was I found that super energizing. But I think the the pivotal point for me was when I first stepped into a CTL at the University of Wyoming, the Ellbogen Center. 

Derek Bruff 4:01
A center for teaching learning. 

Beate Brunow 4:03
And was part of those conversations in there and just listened in on on workshops. And that really solidified for me that those are the conversations I wanted to have and those are the spaces I wanted to be in and really kind of solidified that I wanted to be an educator in higher ed. 

Derek Bruff 4:21
That's great. That's great. Yeah. So it wasn't stepping into a classroom per se, but but having those conversations with other educators that helped you see yourself that way. 

Beate Brunow 4:30
Mm hmm. Yeah, I think so. Yeah. 

Derek Bruff 4:32
Yeah, that's great. What about you, Shawn? 

Shawn Simonson 4:35
So when I started out, I had no idea that I was ever going to be a teacher. I went to college in a pre-vet program. I had this vision of being a large animal equine veterinarian and I went to Colorado State University, and the vet program was extremely competitive to get into. And so to find a way to make my vet school application look unique, I thought, Oh, I'll get a teaching certificate. Nobody else is going to have a teaching certificate, that will make me stand out. 

So I enrolled in the teaching certificate program and really all the way through the prep courses and all the practicum teaching stuff, It really didn't appeal to me that much. It really stayed as: I'm doing this really to to enhance my vet school application. But then I did my student teaching and I taught at an inner city school in Denver and had a great cooperating teacher who really gave me some strong feedback, gave me some great ideas. 

And really my best class was I taught basic biology and with kids with special needs and really figuring out how to help them understand the content and not worry too much about grades. And by the time I finished that experience, I thought who cares about vet school? I want to be a teacher. So I quit applying to vet school and applied for teaching jobs. And after I got my first teaching job and that had mixed success. But based on that experience, I stayed in teaching and eventually through a very winding road, ended up teaching in higher ed and have found my place. Teaching is the best part of being a professor, being in the classroom with students. 

Derek Bruff 6:38
I'm with you there. Yeah, well, good to get to know you both a little bit. Let's talk about evaluating teaching, and this is a big topic. We're not going to explore all of it today, but I think you both have some interesting perspectives and experiences in this area. You've both been involved in the development of some frameworks for describing the components of effective teaching... two different frameworks. Although when I looked at them, I saw some similarities for sure. 

What led for each of you? What led to the creation of these frameworks and why do you think they're important for the work that you're doing on your campuses? And I'll start with you, Shawn. 

Shawn Simonson 7:21
My story actually started out pretty personal in that I didn't feel that student course evaluations were providing an accurate picture of what was happening in my classroom. And so I started looking for other ways to document what was going on and tell my teaching story. And along the way, I got involved with other people through our Center for Teaching and Learning, and really this desire for a tool to assess teaching that was more robust and equitable than student evaluations, really, that became our driver because faculty were being harmed by the use of this biased student evaluation of teaching. But on the other side, exemplary work that was being done was not being recognized with that tool. And so that's really was what the impetus was. How do we do a better job of telling our teaching story and and if we want to use that, how do we use that to develop a growth mindset around our teaching? 

Derek Bruff 8:27
How about you, Beate? What led to the development of your the framework that you helped work on? 

Beate Brunow 8:33
Yeah, a very different route here at Penn State. So maybe a little bit for context. So at Penn State, this came really out of the faculty senate, which is a has a long history of being a very strong legislative body here at Penn State. Penn State, of course, a huge institution, more than 5600 full time faculty members at 23 campuses. So one university geographically dispersed, as we say. So it's a massive, massive institution. So this came here through the faculty Senate in 2021. Several Senate committees submitted an advisory report to the president, which was a faculty teaching assessment framework in which they recommended that that there be multiple points of data besides just student feedback to assess faculty teaching, that if we use continue to use student feedback, we really have to think about ways to decrease bias in composition and interpretation of student feedback. And the report set the structure in place or recommended the structure that the multiple data points should be self-reflection, peer review and student feedback if we continue to collect student feedback.

So the report got to the president and then the vice provost charged an implementation task force that I was then part of, and we were divided into three committees: self-reflection, peer review, student feedback, and started our work of looking at this original advisory report and thinking about how we would implement all of the recommendations from that report. That was really our charge as a committee is to go through this report and decide what is implementable. 

And very quickly I think all the committees realize we need to have a common thread, right? We cannot all work in isolation. And it was interesting that the original Senate committees didn't necessarily think about that. So that emerged very quickly in our conversation that we needed to have like some sort of standard or some sort of common thread against which we measure all these different sources of evidence. I think what makes this important to have these elements of effective teaching, as we call them, so this is our definition of effective teaching, and to have this this assessment framework is that it provides us a shared language about effective teaching. I think that is really the key piece for us. With all the different mental models that we have about what good teaching looks like, right? This is putting some language to it. 

And I hope that in our case that this framework to the way it is set up now, it makes it make it easier for us in the future to tweak things and to make changes so we don't get stuck with something for the next three decades again. 

Derek Bruff 11:33
Let's let's get concrete here for a minute. Shawn, can you tell us the main components of the framework that that you worked on? 

Shawn Simonson 11:42
Sure, we came up with four main components and we really took a literature based approach. And so we came up with course design in that instructors design course materials in alignment with course learning outcomes. We included scholarly teaching, and that's implementing evidence based practices, but also taking a scholarly approach to teaching. Being learner centered, using a an inclusive, learner centered approach. And then our fourth piece was reflective practice, practicing reflective teaching to drive continuous improvement of teaching. And then we agree that there are those those three lenses that Beata mentioned at Penn State of the student perspective, the peer perspective and the personal perspective. We build those in to those four components. 

Derek Bruff 12:36
Gotcha. And Beate, what about what about the framework at Penn State? What were the what were the big pieces there? 

Beate Brunow 12:42
Yeah, our elements of effective teaching also have four pieces. The first one is effective design. So that is everything that has to happen before you start teaching a course, right? You have a student centered design, there's alignment, it's scaffolded and transparent, clearly structured, accessible, relevant. All of these things kind of fall into effect of teaching effective design. So then we have effective instruction. So that is what happens as you interact with students in your learning environment, whatever that might be. So that takes cognitive and non-cognitive aspects of learning into consideration as well.

Then the third element that we have is inclusive and ethical pedagogy and really in our case felt like we we really wanted to call that out. That was a conversation that happened, right? The idea of... does inclusion an equity, should that not live everywhere? And yes, absolutely. But we also felt like this was a good moment for us as an institution to call that out and bring special attention to this element of effective teaching. And then the fourth one is very similar to what Shawn talked about, which is reflective and evolving practice, so the idea that professional development is important, that there is an evolution to one's teaching practice over time. 

Derek Bruff 14:05
Yeah, so, so it's natural, maybe this is just me, but I like to compare and contrast is a natural move here, right? So, so both of you have the elements of design, right? Kind of what goes into the planning and the preparation and alignment there. Both of you mentioned inclusive teaching practices, kind of framed a little bit differently, but but still very important there. The reflective piece, the scholarly piece. Yeah.

So I'm a little curious what and this is where you may see where the rabbit trails go, but what what was challenging about putting together these frameworks and was there anything that was maybe surprisingly easy? Because I think you took different took different approaches on your campus of kind of how to come up with these frameworks, even if you've landed in somewhat similar structures.

Beate Brunow 14:54
Yeah, I think, I have to say the elements of effective teaching, developing those were surprisingly simple in some ways in our case, right? We had these committees that were charged with developing the peer review, student feedback, self-reflection, thinking about implementation, and realized that they needed to have something that connects all of them. So it was it was clear to the committees very, very quickly that we needed these elements. And we formed a small group of folks who got together, drafted them, shared them back out with the committees, set them back out to some other experts across campus, some rounds of revisions. And yeah, we really landed on those fairly easily I have to say. That was not a long process of getting by in and I think in our case, honestly, we were all so focused on getting these three lenses, as Shawn calls them, like the getting the student feedback right, the peer review, the self-reflection. We were so focused on those big pieces that the the elements didn't get that much attention, honestly, as we were developing them. I think they're getting more attention now. But in the process and as we were presenting them to the Senate, I think they they seemed like a necessity probably to most people. 

Derek Bruff 16:16
Okay. Yeah. What about you, Shawn? 

Shawn Simonson 16:21
I would agree that developing our framework was actually pretty easy. But then we took a different approach as well because we we took a scholarly approach and then we looked at the literature and we kind of did a backwards design approach and figured out what goes on in the classroom and what we would want to see from someone who was an effective teacher. So we didn't try to gain consensus from the beginning because there was a lot of information that was available about what effective teaching is. And a thematic analysis made it pretty clear what we were looking for. But then we went through and we modified it based on feedback and experience applying the framework. We used faculty learning communities and focus groups to to refine what we started with and where we ended up now. But I think, so developing the framework itself was pretty easy.

But I think the challenging thing is sharing it. Because our approach made it a little more challenging because we didn't try to gain consensus from the beginning and so we were like I said, we're working with faculty learning communities and focus groups and mostly people who really wanted to participate and who had a positive perspective and wanted to explore their teaching with us. So we had a lot of people working with us who were very engaged, but we didn't have as rich of an experience working with those who are not as enthusiastic about teaching. And so as we've tried to, to share our message, so to speak, to get a wider audience, we're finding that because we didn't do this consensus building in the beginning, that's created a little bit of a challenge for us. 

Derek Bruff 18:08
Beate, It sounds like you had more kind of cooks in the kitchen in the development of your framework, but you also indicated that like that doesn't mean that it's smooth sailing, right? Are you anticipating some types of challenges as more, more faculty start to encounter this framework and have to do something with it? 

Beate Brunow 18:25
Yeah, I mean, I think that'll be interesting to see. It's it's really this semester is the first semester where we have changed to student feedback questions. And next semester many departments will tackle peer review. So it's kind of a slow rollout over time. 

I'm sure there will be changes and I would be excited if we would have, you know, really meaningful conversations about changes. And I think one thing that I'm... that makes me hopeful too, is that the vice provost here at Penn State has a standing advisory committee now around the assessment of teaching. 

But the idea that there is kind of the opportunity to make changes as we articulate the need for them as a faculty, right. If we find out that something really doesn't hit the mark or doesn't make sense or it's just not implementable as we had envisioned it, I'm hopeful that there is opportunity to have conversations and make changes along the way. 

Derek Bruff 19:31
And, Shawn, you mentioned how, you know, this kind of came out of a need for something more robust than just student evaluations. What what would you like to be what would you like to see done with your framework on your campus? What how how would it be used maybe to improve the evaluation process or to kind of open up those other two lenses? 

Shawn Simonson 19:54
Yeah. So I think something that Beate mentioned earlier about creating a common language around teaching has been very useful. I think that's something that we hope to be able to to do moving forward. But we've also been able to promote conversations around rewriting policies. When we started this process, the only required aspect or the required tool for assessing teaching was student course evaluations. And at this point the policy states that you can no longer use student course evaluations as your only tool. And you can't set a benchmark saying that you have to achieve a 4.5 or greater to meet expectations or whatever. And so I think that's that's huge progress. And it's so it's creating more of a conversation around, okay, so if we don't use student course evaluations as our sole tool, what are we going to use? 

And one of the things that I've really enjoyed seeing is that we're talking about teaching. We're not talking about roles, now it's teaching is seen as teaching. It doesn't matter if you are a tenured professor or a lecturer or an adjunct. We're talking about assessing teaching in the same way, because from the student perspective, they shouldn't be able to tell who's teaching the course, right? And so we should be assessing it in the same way. So creating this common definition, creating clear expectations around what effective teaching looks like. But then one of the things that we do for our students in an effective classroom is have clear criteria and standards. And so we should be doing that for ourselves, as what does effective teaching look like by having clear criteria and standards. 

And by having those clear criteria and standards, we can then do some formative assessment and help generate a growth or engender a growth mindset and help people say, okay, maybe I'm not doing real well in this idea of alignment and and here's some evidence that it could be better and now I have something that I can work on to improve, and then we can take that formative assessment and eventually build to a summative assessment that aligns with what we want to see in effective teaching. And so I really I really hope and I think it's happening, that we're getting a stronger culture that values and rewards teaching. 

Beate Brunow 22:44
I'm noticing that, that we're both using the language of hope, Shawn, which is interesting, but it's true. I think it is a hopeful time. 

Derek Bruff 22:55
Yeah, Well, and I think about some of the kind of cultural roadblocks I've seen in the past around kind of valuing and evaluating teaching. You know, the idea that somehow like teaching is kind of a a fixed skill set, right? Some people have the... some people are born to be teachers, some people aren't, or even a kind of more subtle version of that, which is, you know, I've been teaching for five or six years, I've kind of figured out what I'm doing and now I'm a good teacher and there's nothing else to do to kind of change and grow over time. And I've seen that a lot, right, where folks are kind of good enough for the current evaluation scheme, but they don't think of teaching as something a skill, a profession that they continue to develop over their career, when in fact it could very well be that, right, and to some degree needs to be that right. I mean, the conditions under which we teach aren't the same ten years ago than they are now, right? The students are different. The topics are different, the technologies are different. There is a need for kind of continuing to invest in that skill set. 

I am a little, and maybe Sean see more of this, but I am a little shocked at how easy it was to come to consensus around this because I think if you put a bunch of teaching center people in a room together and gave them half an hour to knock out a pretty good framework for assessing teaching, they'd land at something very similar. Right? Like we've we've done the reading. We know these components. We talk about them all the time. But faculty have a lot of different perspectives on what good teaching is and what it's supposed to be. And so I think there may... are there particular parts of this framework where you feel like some faculty are starting to push back a little bit more? 

Beate Brunow 24:39
I think that's a that's a really good question. I think ours was written in a very broad way, right? Like I said, we have 23 campus locations. We have so many instructional contexts. We have instructors who may not have full control over design or content of their courses, right? I mean, really the the context varies so dramatically across the institution. So I think the the pros and cons, right here, is that it's a it's a pretty broad framework. So what we really did is we wrote like an introductory sentence for each of these four elements and then said and here are some examples. So these are not criteria that we have laid out under what is effective design. We just say when you think about effective design and you assess effective design here are examples of what you might look for. So I think that is what is possible in our context, because it is such a such a large institution with so many different contexts of instruction. 

Shawn Simonson 25:46
And in, as you said, Derek, there's no one perspective or one tool that tells the complete story. You know, teaching is a very complex activity. Learning is very complex in itself. And and so there's no single thing that's going to demonstrate effectiveness. So we can't just use one tool. 

Derek Bruff 26:09
Yeah, well, and I think it says a lot. I don't... I'm... It's my podcast, I can say what I want to I guess. I think for the department that is only looking at student course evaluations and really only looking at them in that crude way that you describe, Shawn, like we're going to look at one or two questions and we're going to look at your overall rating of the instructor and it has to be a certain average or higher, right? That's that's that's reductionist, right. Like that's that's a very crude way to make sense of a very complex professional endeavor. And we don't take that approach... even in research areas, right, where we're where we have found ways to kind of come up with lots of metrics, but at least we have lots of metrics and not just like the one metric, right? And that's, you know, I think we can do better. And I think though, it speaks to how we value these professional endeavors, right? If if research is job number one, then we're going to take some time and effort into evaluating research success really carefully. If teaching is job number one, then we're going to pay a lot of attention to how we how we evaluate that, too. And I think and, you know, it is a little telling that some departments in colleges don't do much there. And I'm I am also hopeful that that's changing. And it sounds like on your campuses that you're starting to see some changes.

Beate, you said you've rolled out student course evaluations. Are there are there particular changes to those evaluations this year that have come out of this work with the frameworks? 

Beate Brunow 27:45
Yeah. So the the original Senate report that came out back in 2021 had a list of new questions that they recommended for the student survey and we had to tweak some of them because we needed them to align with the elements of effective teaching. For example, there was no question on inclusion. So we added a question around that. 

An interesting experiment, I think we'll we'll have to see how how that works out is that the the student survey now has, I want to say four questions that go to the faculty member and the administrative unit head. And then it has two additional questions that only go to the faculty member. But those are open-ended questions. So we'll see how much feedback there is. But the faculty Senate recommendation was that that this assessment framework has formative and summative pieces to it. And some also they wanted to shift away from assessment of teaching only being a summative thing that we do and kind of integrate some formative pieces. So having two questions that only go to instructor at the end of the semester was one way to do it. And then we also now in the spring, were rolling out a mid-semester student survey that will only go to the instructor as well. So those are kind of formative pieces in the student feedback realm that that we're introducing. So those are some of the big changes there. Yeah. 

Derek Bruff 29:18
And so that's a mid-semester feedback as kind of standard process as opposed to just something instructors might do for themselves?

Beate Brunow 29:25
For every course, every student will automatically receive the prompt to complete mid-semester feedback for all of their courses. Yeah, we'll see how that goes. I mean, I think that's one of the big experiments, right? Is how, how significant is the survey fatigue? How how much feedback are students willing to give? But the student government at Penn State was was invested in this. Actually, they came to our committee and said, we want mid-semester feedback. We want to give mid-semester feedback that that was something that students were passionate about, which is really interesting. 

Derek Bruff 29:59
Shawn What changes are you hoping to see on your campus? Either either through the student piece or the other two lenses? 

Shawn Simonson 30:10
Well, I think going back to building a culture that values and rewards teaching and incentivizing good teaching. You know, we have all kinds of awards for scholarly work and we're building more for service and we're actually starting to build more for teaching now. And so identifying those good teachers and giving them some sort of incentive to continue, I think that that's something that we're we're hoping to to develop. But just making it it more of the forefront. You know, you mentioned this a couple of minutes ago about if we really value teaching, then we really need to to make it evident that we do as opposed to scholarly pursuits or or whatever. And there was a poll that came out, a research study that came out based on a poll, I think about a month ago that indicated that the public thinks that our primary job at the university level is to teach and that students will learn. And I don't know that every institution has that focus. And so so I would really like to see that be more visible to students to faculty, to those who teach, to the community. 

And thinking of teaching as is something, I think, Derek, you said that it's it... there are people who are more talented teachers to begin with, but we see that in everything, but it's something that we can develop. It's a craft that we can spend time thinking about and moving forward and improving. And yeah, there are going to be people who that's not their priority. But if you're interacting with students, we should be having some of a growth mindset in that regard. And so I think that's really what I'd like to see is really this this focus on quality or effective teaching and really putting the learning, really making learning really what we do. 

Derek Bruff 32:23
I find that particularly at universities that have a strong research focus, faculty do kind of see their departments and their disciplines as they're they're, they're kind of primary world that they live in. And so I think one of the things that I'm really excited about is the thought of of this these kinds of frameworks being debated and discussed and implemented in departments for that peer review piece. Right? Because-- I may have shared this story on the podcast before-- I was consulting with a young math faculty member many years ago on my campus and she had been observed by one of the more senior faculty in the department as part of a standard kind of teaching observation and feedback piece. And he had given her some advice that probably made a lot of sense to him, but didn't make a lick of sense to her and her kind of her identity and who she was in the classroom and her experience level. And I remember her telling me, I wish he would I wish he would help less. And so and I felt like I don't you know, I don't know what could have solved that problem, but I don't know that that department had had a robust conversation about what effective teaching is and what it looks like over time and for different types of instructors. And so I can imagine these frameworks helping to have those conversations.

Are you starting to see any of those conversations emerge on your campus among among faculty in the department who are who are maybe trying to do peer review in a in a more thoughtful way? 

Beate Brunow 33:52
Yeah. At Penn State departments have been charged with kind of updating their peer review guidelines this coming spring to align with this this overall framework. So I think academic units will have to have conversations about this. I think a lot of units have robust peer review guidelines in place, and that might be a good chance to just revisit and remind people what this is all about. But I think for others it'll be an interesting conversation to really think about what is it that we're looking for and how do we know? So I'm looking forward to spring conversations. I think that'll... that's coming up for us. Yeah. 

Shawn Simonson 34:38
And I would say we are seeing more conversations about it, but then at the same time we're requiring changes in our promotion and tenure policies across campus. And so this is just, I think, part of that conversation. But you've both have used some terms and I think it's part of the conversations we're having about this peer review or peer evaluation versus peer observation. Some... in a lot of departments, they see that as the same thing, but it doesn't have to be. A peer observation, yeah, just sitting in a classroom or doing a virtual observation is one way of giving some feedback. But there's so much more. And that's something that we've started talking about through our framework is, well, what would happen if we actually did external review of teaching, just like we do external review of our scholarship? And if we had a framework that people could look at and say, okay, this is how we're assessing it, this is what we're looking at, might we get some really robust feedback and something that's not biased by a senior faculty member's perspective on a classroom? And so I think that's an intriguing thing to look at. And and I know that both of you have heard this, but in many instances in that people will say, But my situation's unique or our institution's unique or... but I hate to say this, but it's really not. 

Derek Bruff 36:18
It's not. 

Shawn Simonson 36:19
The more classrooms I visit, the more teachers I talk to, the more people I talk about assessing teaching, yeah, there are some unique things and we call those situational factors too. But as I keep saying, teaching is teaching. There are things that make an effective teacher and it doesn't matter what your context is that exists. 

Beate Brunow 36:44
It's an interesting question, though. I think I feel a little conflicted sometimes when we talk about valuing teaching and at the same time we talk about teaching assessment. I don't know that it's that they always easily go together, right? In my mind I also think about, well, when I have to assess colleague's teaching right through peer review or on a P&T committee, that's also service work. And I might not think always about valuing teaching. I might think a lot about the time I have to invest into doing the kind of work. So I think there are... I think these frameworks, like we said, right, they can they can bring shared language, they can get conversations started. I don't know that they're going to change culture necessarily around the value, how much we value teaching.

And I want to be so negative right now, but to me that these two things, it depends for what purpose we're using these things, right? So for us, the purpose is very clearly assessment of teaching. Hopefully it'll and it'll signal right, that we value teaching. But yeah, they don't always go together that that neatly I think. 

Derek Bruff 38:07
Very true. Right. And I'm also hearing from both of you that sometimes a policy change can have some kind of trickle down effects that may or may not be significant, but might be, right? So if if you know, if student evaluations are not can't be the only source of evidence, That's going to motivate some change at different levels. And if each department has to kind of reconcile their peer peer evaluation process with this new framework, that's that, again, provides an opportunity for some some productive conversations and some useful change. And so I do think about, you know, how do we how do we fix this system so that it does work better? And I think, you know, there's a kind of bottom up and top down approach. And and I think it works best when it's in tandem, right? So, for instance, at Penn State, the fact that this came out of the faculty senate, I think gives it a lot of extra power that wouldn't happen if it were just some vice provost somewhere trying to wave a wand, right? 

Beate Brunow 39:09
Yeah, absolutely. I agree. 

Derek Bruff 39:11
Well, thank you both. This has been great. And this is work that continues. I know, but affecting this kind of change for the for the positive is hard and complicated. And thanks for sharing some of your experiences and trying to do that on your campuses. I really appreciate it. 

Beate Brunow 39:32
Yeah. Thanks for having us. 

Shawn Simonson 39:34
Yeah, thanks for getting the word out that this is going on. 

Derek Bruff 39:38
Happily, very happily. 

That was Beate Brunow, associate director of the Schreyer Institute for Teaching at Penn State, and Shawn Simonson, professor of kinesiology at Boise State University. In the show notes, you’ll find links to the teaching effectiveness frameworks developed on both of those campuses, as well as that Beth McMurtrie article I mentioned at the top of the show. 

I shared a lot of my thoughts on this topic during the conversation, so I won’t say anything more here, aside from inviting you, dear listener, to reach out with your thoughts about better ways to value and evaluate teaching!

Intentional Teaching is sponsored by UPCEA, the online and professional education association. In the show notes, you’ll find a link to the UPCEA website, where you can find out about their research, networking opportunities, and professional development offerings.

This episode of Intentional Teaching was produced and edited by me, Derek Bruff. See the show notes for links to my website, the Intentional Teaching newsletter, and my Patreon, where you can help support the show for just a few bucks a month. If you’ve found this or any episode of Intentional Teaching useful, would you consider sharing it with a colleague? That would mean a lot.

As always, thanks for listening.