Study Hall with Flower Darby, Josh Eyler, and Regan Gurung

Study Hall is back! This is the format where I talk to three fantastic guests about three recent studies on generative AI in higher education. 

Guests for this edition of Study Hall are: Flower Darby, incoming director of the Center for Teaching and Learning at Estrella Mountain Community College; Josh Eyler, senior director of the Center for Excellence in Teaching and Learning and assistant professor of teacher education at the University of Mississippi; and Regan Gurung,  professor of psychology at Oregon State University.

Episode Resources

STUDY #1

Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Nature Scientific Reports, 15, 17458. https://doi.org/10.1038/s41598-025-97652-6

STUDY #2

Lodge, J. M., and Loble, L., (2026). Artificial intelligence, cognitive offloading and implications for education, University of Technology Sydney, https://www.uts.edu.au/news/2026/03/experts-warn-unstructured-ai-use-in-schools-risks-cognitive-atrophy/contentassets/ai-cognitive-offloading-and-implications-for-education.pdf 

STUDY #3

Wadinambiarachchi, S., Kelly, R., Pareek, S., Zhou, Q., & Velloso, E. (2024.) The effects of generative AI on design fixation and divergent thinking. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 380, 1–18. https://doi.org/10.1145/3613904.3642919 

• Flower Darby's website: https://flowerdarby.com/
• Josh Eyler's website: https://josheyler.wordpress.com/
• Regan Gurung's website: https://regangurung.com/
• Study Hall 2025 with Lance Eaton, Michelle D. Miller, and David Nelson

Transcript

Derek Bruff 0:05

Welcome to Intentional Teaching, a podcast aimed at educators to help them develop foundational teaching skills and explore new ideas in teaching. I'm your host, Derek Bruff. Today on the podcast, I am excited to share another study hall episode. This is a format for the show I tried out last year, has some success, and I've been eager to bring it back to the podcast. I have three fantastic guests on to discuss recent studies about teaching and learning in higher ed. All three of my guests are experienced educational developers and accomplished scholars, and all three of the studies we discuss today focus on the potential impact of generative AI on student learning, a common topic on the show these days. Each panelist will introduce one study, share their thoughts on the research, and then we'll all jump in to discuss the study.

Derek Bruff 0:56

In this edition of Study Hall, my guests are Flower Darby, incoming director of the Center for Teaching and Learning at Estrella Mountain Community College, and author, most recently, of The Joyful Online Teacher, Finding Our Fizz in Asynchronous Classes. Our second panelist is Josh Eyler, Senior Director of the Center for Excellence in Teaching and Learning, and Assistant Professor of Teacher Education at the University of Mississippi. And his most recent book is Failing Our Future: How Grades Harm Students and What We Can Do About It. And our third panelist is Regan Gurung, a professor of psychology at Oregon State University, and author of, most recently, among many books, Teach Like a Champ, the Psychology-based Guide to Effective and Efficient Teaching, co-authored with Elizabeth Yost Hammer.

Derek Bruff 1:45

Each of the three studies we discuss on the show today will be briefly summarized by one of our panelists, so don't feel like you need to read the article to follow our conversation. Our panelists mentioned the authors and titles of the studies, and you can find full citations in the episode notes if you want to read more.

Derek Bruff 2:01

Flower Josh Regan, I'm very excited to have you on Intentional Teaching today to uh to the second edition of our study hall format. Um, this is a fun format, and I think we're gonna dig into some really interesting studies today. I'm very, very glad to be doing that with all three of you. Thank you for being here and to lending your perspectives on all of this.

Flower Darby 2:20

Thanks, Derek. Thank you for having us. Super to be here.

Derek Bruff 2:24

We'll jump in with our first study. And uh, Regan, I think this one is yours. Do you want to give us a little orientation to our first study?

Regan Gurung 2:31

Absolutely, absolutely. So I have the uh the fun task of uh introducing the Kestin et al. 2025 uh paper. This was in published in Nature, which right away goes, hey, it's nature, that must be something. Uh and the title, I I do want to read the title because you just read the title and you go, man, this is this must be something. Uh AI Tutoring Outperforms in-class active learning. Uh and it's a randomized controlled trial introducing a novel research-based design. And even with the stuff before the colony, you go, wow, AI tutoring outperforms in-class active learning. Okay, where's my job? Right. Uh so I duh I dove into this very eagerly. And what Kesten et al. did, uh, they are uh Keston is a Harvard uh faculty member, and they took they went to a physics class and they did something pretty neat uh where they had students, uh they randomly assigned students to either uh take part in a in-class activity on a certain topic or an AI-driven activity for that same topic. And what's interesting about this is that they they use the cross-lag design, which both is ethically sound and it really keeps uh your your your sample consistent and it as their own controls. And what that means here is that one week, and this is I should say they did it at the end of the semester, on one class period near the end of the semester, they had students either take part in the in-class or they were given an AI tutor that they could do online at home. The very next week they flip-flopped. So the students who had an in-class session got the AI uh online. Those who had uh AI online in the previous week did the in-class session. They controlled for a whole load of things, which is absolutely wonderful to the social psycho uh social scientists and me. They controlled for previous knowledge using the force concept inventory, they controlled for the instructors teaching the course, they measured time um and so on and so forth. And their main dependent variables, right, this gets to it, right? How are they measuring learning is they not only had a pre- and post-measure of learning, so they tested them before the session, they tested them at the end of the session, but they also measured uh their motivation during the session and their engagement during the session. And uh what they found was that those students uh scored uh seemed to learn more when they uh uh took part in the AI session than when they were in the class session. And this was constant over both uh lessons. As I said, there were two different times they did this. And when it came to the more subjective measures, they found significant differences in enjoyment and motivation and wait for it. The students in the AI condition enjoyed it more and reported more motivation, but there was no difference in enjoyment and and growth mindset. So we have these two class sessions, AI or in-class, and we find that students report learning more as uh a pre and post-test in the AI sessions. So there's more there, but I figured I'd put that out there. Yeah. Right there, there's a lot to chew on.

Derek Bruff 5:58

Sure. Well, and and what are what are what are what are some elements of that that stand out to you as most interesting?

Regan Gurung 6:03

Well, I think first off, there was uh just the absolute level of control. I mean, they have uh paragraphs and paragraphs, all the things they control for. Uh and that's often a very big issue in studies like this. Is that you know there are confounding variables, or for the stats geeks amongst us, there are mediators and moderators out there that they just don't take into account. And here they control for a lot. What I was most impressed by, what I was most impressed by is their in-class session was very explicitly designed as an active learning in-class session. So they designed both the AI Tutor, they they wrote the prompts to be as active as possible. But to be fair, they also made sure the in-class session was as active as possible. So this is not just some business as usual lecturing mode. No, even the in-class session, even the in-class session was very active and designed to be active. Because that was the first thing I thought of. You know, you if I go and give a very boring in-class lecture, sure AI is going to be bad, right? But they controlled for that, which was impressive. And then what really appealed to me was the fact that they leaned on the things I see often when I teach, which is in a large class, and here the sample size, the class was close to 300. The the sample size was again pretty large. We're not talking a class of 20 or 30, it's like over 100. In any class, there are some students who want more, and there are some students who want less. And it is very tough to meet all those students in a classroom setting in a group. And they designed the AI tutor to be more individualized, to be give students more timing, and so on and so forth. So these were just some of the things that really jumped out at me. The last thing I'll add is looking at the timing is really interesting. The classroom sessions were set at 60 minutes, but the AI, you know, which is a class session, the AI was as long as students took. And what you see in the data, because they measured the time spent with the AI, there were a number of students who spent less time on the material. Very few spent more than the 60 minutes. And the time spent on task did not correlate to the learning, which I thought is super interesting, right? So many of them are using the AI for less time, but still scoring higher. Sure. So all things for us to consider. I'm gonna leave my issue with it for later. Okay. But uh, a lot there to intrigue and uh make you think.

Derek Bruff 8:43

Yeah. Flower, what thoughts do you have about this this article?

Flower Darby 8:47

It does sound I'm I'm very intrigued by the um by the teaser about your issue, Lady. But uh, you know, I'm I'm actually sort of fixated on your opening question about, well, then what becomes of my job? Um I can imagine that thought uh causing a lot of anxiety in in the hearts and minds of professors and instructors who truly um care about student learning and work hard to provide those active learning um experiences. However, I also picked up on your comment that it's impossible to please every student in in an active learning classroom and for a variety of reasons. It could be introversion, neurodivergence, there could be lots of reasons, uh anxiety that students are not ready for all that interaction. So I am intrigued by that sort of ability to personalize. Those are some initial thoughts just now.

Derek Bruff 9:38

Yeah. Josh, what about you? What do you want to add to the table here?

Josh Eyler 9:43

Um I'm trying not to betray my own scholarly orientation here. But um I I do have I have lots of questions. I I I wonder about a lot of things. Um, and some of the things I wonder about include what was actually happening in the active learning sessions. And so the study itself, I looked multiple times to try and find it. It says first the instructor introduces an activity, then students work through the activity in self-selected groups with support and guidance. And finally, the instructor provides targeted feedback and paraphrasing a little bit there. But um a big part of me wants to know what the activities actually looked like. Uh, because I think that has bearing on this particular study, right? If um that some activities may have been more beneficial than others, we don't really know. Uh, we don't know how much time they spent on it. We don't know what the feedback looked like. And we also know that these were self-selected groups, and there's kind of a mix in the literature about uh about the effectiveness of that. So I wonder a lot about that. If we are measuring the two against each other, I'd like to know a little bit more. I wonder what was on the pre and the post-test. And we know a little bit more about the pre than the post, but the study's pretty silent on the post-test. And that matters to me because what I want to come away from this study with is knowing whether uh the performance on the post-test really was measuring long-term learning application, all the great things we know about learning, or performance on uh a set kind of assessment. So I want to know a little bit more about that. And outside of the contours of this really well-designed study, I completely agree with Regan that this is uh just a classic, um, uh, a classic model thing for how to do this work. I also wonder what would happen if it wasn't just uh a kind of a one-time thing, if it if you stretched it out uh longer over the course of the semester, whether you would find the same things, particularly with the dispositions like uh enthusiasm, that that sort of thing. So those are just questions that I have as as I went through it.

Regan Gurung 11:56

Yeah. Regan, do you want to? Oh, absolutely, absolutely. Because uh, so flower, uh no surprise, my opening job comment I can answer by saying my even after reading this and digesting it, my job is pretty safe. All our jobs are pretty safe. And for one of the reasons is actually, uh Josh, what you brought up. Uh and then there's one of the reasons, and I'll add another. One of the reasons is this these were two class periods. These were two class periods. We have no idea what would happen if uh you did this for more than two class periods. I could not imagine, I mean, the real test, and that's why I made a deal about talking about the title. That title, AI tutoring outperforms in-class active learning in two class sessions out of an entire semester, right? That's the asterisk. That's the asterisk. So so number one, and the reason I mentioned that is because by pure dishabituation, you're gonna get some good effects, right? Here's um uh any we change anything, you know, and and I'm gonna be facetious when I say if I move from a black marker to a red marker, people are gonna pay a little bit more attention. Now, of course, I'm talking there, but you get the idea. Right. There's a novelty effect, novelty happening here. The novelty effect, the dish habituation immediately is doing something. I can easily see that the dishabituation and novelty holding for two weeks. Will it hold for longer? We don't know. The second thing is, and Joshua nailed it, because that's what I always think about, and I'll be quite honest, this is an issue with most studies on pedagogy and instructional method comparisons, which is we don't know long-term learning. We don't know what the difference is beyond the end of the term. And in fact, to truly you know jump on this study, I would have loved to see if end-of-term grades different in any way. We don't get that. Now, of course, they've counterbalanced, so some of the effect is gone, but they can compare it to a class where they didn't do the active learning at all. I mean, the AI at all. Was there a difference? Because if indeed there's something to AI tutoring beyond the novelty and what we talked about, uh, we should see some changes, maybe see some changes. And again, I know learning is complex, but I would have loved to see what their differences between those groups, and especially between the people who had AI first and AI second. You know, uh, there are all those little nuances, but I think the big level thing here is uh we don't know long-term learning, we don't know if there's a novelty effect. Uh, I will also say, to be fair, uh, you know, Josh, with some of your stuff, I did dig into that a little bit more. Uh, and the even though I had concerns with the post-test, I had to tell myself, but it was consistent across both groups. So we've got to give them that, right? Yeah. Uh but that said, but that said, really, that big thing is longer term and uh sort of flash in the pan to some extent. So wonderful, but let's make it four sessions. Let's, you know, and the true test, of course, and this is what we truly need to watch out for because you see it happening in some places, is those places that are trying to offer a completely AI-based course, right? Uh, with none of that. I mean, here, I I I made a point of saying this happened late in the term. Students have already built rapport with each other and with the instructors. That rapport automatically is going to do something for learning that is gonna make a difference. And that, of course, wasn't controlled for.

Derek Bruff 15:30

Yeah. So, Flower, can you see more potential in AI for the kind of filling in the gaps, maybe, that sometimes happen uh in a in a large class session where you can't quite differentiate for every student?

Flower Darby 15:47

Well, this has been a very interesting and robust discussion that has given me a lot more to think about. Maybe a takeaway that I have in this particular moment that I think our students already are accessing AI in ways that can enhance their learning in the way that they need in that moment in time. So maybe AI tutoring is best reserved for outside of class.

Derek Bruff 16:11

Yeah. I I'm also struck by um, well, one, that this wasn't a vanilla AI chatbot. This was this was uh an AI experience that was as well designed as as many that I've seen in terms of how the AI interface was structured and prompted to engage students, right? Um uh it was it was kind of active tutoring in a useful way. And our students don't always use AI that way, certainly, as we'll see, I think, in the next study. Um uh but also the the kind of like I was interested by the kind of affective component of the students who liked studying by themselves with their AI buddy more than they did the kind of social interactions in the classroom. And I don't want to over-generalize about Harvard physics students or anything, but like I can imagine, you know, there's real value in social and collaborative learning that the AI experience won't replicate. And so maybe they learn physics in slightly different ways, but there's a whole host of other learning and cognitive and social skills that wouldn't come through that AI piece. And so um, I did appreciate the author said an AI tutor should not replace in-person teaching. Rather, it should be used to bring all students up to a level where they can achieve the max benefit from their time in class. The authors don't want us out of jobs either.

Josh Eyler 17:30

Uh could I just add one quick thing? Um Yeah. Uh, you know, I'll due credit to the research team. Uh I also wonder about the scalability because they designed a really great AI tutor. I mean, an amazing one. Uh, but if you're thinking about how these findings would scale, uh, the yeah, uh the notion that you would have that caliber and quality of AI tutor uh out there serving lots and lots of students in lots and lots of context, that uh that's a little suspicious to me. I don't, you know, that that would take a lot of careful planning.

Derek Bruff 18:06

This one was actually topic-specific. It wasn't just a Harvard physics tutor, but it was tuned around these particular topics. So yeah, that there, there's some lift there.

Regan Gurung 18:13

And and I think there's a little line in there that said, we took months to program. Right? And I love that. It's embedded right there. It took us months for these two topics. Two topics, months. Yeah. Yeah.

Flower Darby 18:30

It reminded me of conference sessions that I go to that I find just frustrating when somebody presents on this fabulous program that they piloted with all this funding and all this resource personnel support. And then I just get frustrated by those kind of um, you know, report outs. So this is very interesting to me as well. Good good point.

Regan Gurung 18:49

And and Derek, I just want to say, because I love your point about the interest of your peers, uh, my lab published a study where we use the small little AI thing and we compare it against uh working with a with a partner versus working with the partner and AI. Uh and sure enough, working with the partner without AI trumped uh those students at higher interest in the topic than when assisted by an individualized programmed AI. So that's why I think there's still hope. And now I didn't spend months making that, but you know.

unknown 19:20

Right.

Derek Bruff 19:20

Right. Well, let's let's move on to our second paper, um, which uh uh takes a I think a higher level view of about uh some of the research around AI and learning. And Josh, this is yours. So can you give us an introduction to paper number two?

Josh Eyler 19:35

Sure, I'm happy to. Um so uh paper two is called Artificial Intelligence, Cognitive Offloading and Implications for Education, published in March 2026, so not very long ago at all. And it's uh it was put together by two researchers at the Centre for Social Justice and Inclusion at University of Technology Sydney in Australia. And their names are Jason Lodge and Leslie Noble, uh Lobel. Um, and I I'm really excited about this paper, honestly, both as a researcher and as a director of a teaching center, because it's not a formal meta-analysis, but it is a meta-analytic-like report that has that has surveyed lots and lots of uh recent research on AI and learning and comes with uh comes away with that, uh, from that, with I think really useful, actionable takeaways that we can use in our classrooms and and as we work with faculty. So I really like that. Um I will say that it is a it is pretty balanced about the role of AI in education. Um it has some suggestions for how to use it effectively, for sure. One of the key quotes for me, though, uh is this one. Um the true educational risk of AI is not simply that students will use it to cheat on an essay. The far more profound risk is that AI may fundamentally interfere with the cognitive processes of knowledge construction and verification, the very processes that build the long-term memory stores and subsequent skills upon which the majority of critical thinking depends. And for me, that's kind of the lever that moves uh the research that's evident in this report, that they are most concerned about the long-term effect on uh of AI on student learning. And they really went in uh to try and uh find some answers to those questions. Um, so I really I want to talk about two uh aspects of this report. The first is um their discussion of a much talked about uh uh I think um subject, uh, and that is um cognitive offloading. Uh they really drill down and they say, okay, here's the deal. There is good offloading, offloading that we do every day. Um Uh it um you know, things like using a GPS uh to get around that does not affect your uh the your ability to think critically over the long haul. You the you know, the very complex aspects of human learning. Uh it doesn't have any effect on that, and in fact, can you know free up time and all the other things that people have been talking about with positive cognitive offloading. But on the other hand, there is detrimental cognitive offloading, which they cleverly call outsourcing, which I think captures metaphorically the labor uh conversation with AI. And it is the cognitive outsourcing that they are seeing evidence to suggest is detrimental to long-term student learning. That you know, Regan was saying learning is hard, learning is hard. And that's true, that is just the baseline. And so cognitive outsourcing means if you are giving over to AI the more difficult, challenging, complex parts of the learning process, it can be hard, not impossible, but difficult to develop those, uh, to develop those skills uh as as far as you might otherwise, right? So I think that is a really powerful conversation point for where the research uh is at this moment. The second thing I'd like to talk about with this report, um, I find really interesting, it connects exactly to the paper that Regan was just covering, and that is toward the end of the report, they uh they talk about something they call the performance paradox. And it is specifically about the research on AI. So not just the learning mechanisms, but the research that has currently been done. I think this is really important, and I think it's important because in the earlier days of AI, when we were having conversations, is it valuable, is it beneficial, is it not? Uh, people on either side of that divide could find a paper that would support their views. And to some extent, I think that's still a little bit true, but it it was much more nascent, and uh we were having different, it was difficult to have a research-based conversation because of the state of the research at the time. So, what they have done is they've gone back through those early studies and they've identified this thing they call the performance paradox, and it shows that two things are true at the same time, and thus it is a paradox. The first thing that's true is that in the short term, students who are using AI are showing more benefits on immediate performance tasks. So, whether it be the kind of uh assessment that Regan's uh papers were talking about, or some other kind of cognitive performance, students who have used AI tools are showing more gains on that. But it is also true that for long-term, more complex learning, the students who are not using AI are showing more gains in those areas, right? And I think this is so important because we can finally have a conversation that tries to reconcile those two parts of the paradox. If both are true, how do we handle this as educators uh and as members of a university? So there's lots more I could say, but I highlight those.

Derek Bruff 25:24

Yeah, and I'll say they make suggestions actually in the paper, which we may get to, but I want to hear, I want to hear what Flower has to say about this one.

Flower Darby 25:32

Thank you, Derek, and thank you, Josh, for that really elegant um sort of distillation. I I love your key takeaway that this paper, this report drills down to really begin to provide us evidence and talking points and even phrases, right, like performance paradox to help us bring the required complexity and the nuance to our conversation. I've seen so much in recent media headlines, uh, listservs, blogs, right, about the sort of the binary, the two camps of no AI versus all AI. And I think what we're talking about now is is a way to help us recognize that there are strengths and uh challenges to using these tools. And when it comes to cognitive processes, um, I like the nuance and the the fuller picture that this report is providing.

Regan Gurung 26:28

Yeah, you know, uh this is just uh this paper is such a great reminder of something that all of us who talk about teaching and learning say all the time, which is it's contextual. It's contextual, right? I mean, uh and just like the previous the paper I talked about, but it's it comes back here. And Josh, I love the dovetailing that you did there, right? It's it's contextual. There'll be some times when AI will be good, and there'll be some times when it'll be bad. And for me, something that jumped out was sort of buried a little further in this in this document. But uh you touched on it, but I'm gonna take it one step further. I think that distinction really taking cognitive load theory, which was you know, Sweller 1980 or something, but really advancing it and bringing it into the modern age. Uh and I think talking about it as the intrinsic load and the extraneous load, right? That non-essential, that word, that non-essential load uh I think is important because that intrinsic part is the, as they call it, the unavoidable. So I think that distinction is nice when we think about when do we use AI, right? Is it what are the intrinsic versus the uh uh extraneous goals? And then I like this reminder that we need to take a goal-driven approach, which is focusing much more on where the learner's activities, motivations, and goals are, not just the design. And I say this because I read so many papers where my biggest criticism is you've thrown together this new AI tutor with bells and whistles, but you have no theoretical reason for why it should work. You know, and the test is we've got this cool new thing, try it. You alluded to it when you're like, oh, here's this thing somebody's quick put together quickly, right? But where's the theory? Where's the what are the processes by why it should work? And I think this little reminder about the two-load model and the goal-driven approach made me think about that a lot more and say, all right, let's let's look at maybe this provides us with a better context for when we can use these things. So love that part.

Derek Bruff 28:35

One other finding that they saw in multiple studies was um they write the default unstructured use of AI by students appears to trend toward this detrimental outsourcing path. Right. That they they articulate that in fact there there may be, there seem to be ways to use AI in learning that are more constructive and avoid some of the the kind of cognitive offloading uh downsides, but those are not ways that students fall into. That that the kind of more typical ways for students use AI tools is this kind of appearance of fluency. They they they get the answers quickly, it does the work for them, right? It's kind of it's taking away all of those uh learning opportunities to build their skills and their understanding. Um, and that's kind of the default way that many students use these tools. Um, and I'm wondering if that resonated with you all. And and what does that mean for us as teachers who might want to do something useful here?

Flower Darby 29:35

I'd love to uh respond just briefly, if I could, Derek. Um, even just uh in the executive summary, I'm doing that thing that undergrads are accused of doing, which is to read just the first little bit of the article until you find a quote. Here it is, here's my quote. I don't have a page number. Um The report finds that pedagogically structured interventions, such as explicit teaching, load reduction, instruction, and integrated metacognitive prompts, can successfully foster the self-regulated learning, critical thinking, and the deep engagement required for learning. So I also noticed the call for more structure and scaffold, which I think aligns well with many recommendations we're seeing from teaching and learning experts to provide explicit guidance for students on what the expectations are for using AI in productive and helpful ways.

Regan Gurung 30:24

But I think Regan, did you have something? Well, I was I was just gonna say, you know, uh, Derek, your comment and and what we've been talking about really made me and and actually flower when you said the, you know, what should we be concerned about jobs, which is always a good question. And, you know, my my first article didn't didn't worry me. Uh and this doesn't really either, because the anal but because the analogy I make is uh, and actually just, you know, we would uh some of us in the department were talking about this just yesterday. You know, are some of these new papers more threatening than others? And the way I look at it is, and I tend to be an optimist, is the parallel is between, you know, the days when faculty would just lecture. And they lectured because there weren't books printed that were available to people, right? Now, and I mean, and that's just the actual You're talking like 400 years ago, lectures. But I think that's why I love this analogy because they lectured because all the students, yeah, it's not that there wasn't printing, but only the richest folks could afford the printing, right? So that's where the lecture really came from with this. I'm going to lecture and deliver the book because you can't buy it. Well, yes, I think all of us here probably know that some people still behave like books aren't available, right? Or students aren't reading them, therefore we still lecture. We don't do that. But the way we look at how do we change our teaching because a student does have a book now, is the same the way I approach it, which is how do we change our teaching because of the affordances of AI? That's the way I think of it. And that is always gonna be a moving target. But it also does mean that no matter how many years we've taught, or what our rank, or what how many, uh, how much experience we have, we've got to go. What can AI do that I don't need to do? And therefore, what more can I do? And I think that's the question that excites me. And even in papers like this, it's great. I'm like, okay, yeah, that and this here's here's a lot of fodder for how I can do better because it's showing the times that it AI can do a decent job.

Josh Eyler 32:30

So I also really like Derek that this conversation that the four of us are having is really rooted in outstanding research. That if we are going to make headway on this conversation, we we have to really, you know, dive in and deal with the best of the best. So the first paper was just a really well-designed study. This to me, this one to me uh does what great scholarship does. And so this is a hot button issue. Let's see what we actually know about these questions so that we can inform that conversation.

Derek Bruff 33:02

I kept finding moments when reading this paper where I realized I had talked about this topic in one of my many presentations to faculty about AI, but I was making up metaphors to make it make sense. And they're they're they're they're grounding the same notions I was trying to communicate in the research, in the scholarship, in the it, you know, in what we know about cognitive load and such. And so it was, I realized I'm gonna have to go back to a lot of my slides and now add in more references because I feel like it's grounding stuff that maybe, um, like you were saying, flower, you know, there's a lot of, I think, instructor responses to AI that are based on our intuition, and some of them are working reasonably well. And I think this paper helps explain why some of those things are working, right? When I advise faculty, hey, if you're gonna have students use AI on an assignment, have them document their interactions with AI and reflect on those interactions and see where they were helpful and where they weren't helpful. I'm like, okay, I'm I'm giving the kind of scaffolding that they're talking about in this paper, trying to root students away from the unhelpful use to something that might preserve some of that metacognitive awareness. And so, not that that's gonna kind of solve the problem, but it helps me see kind of why some of these moves may be more helpful than others.

Flower Darby 34:16

Absolutely. Yes. It's always nice when you find the really robust uh publications that provide that support for what we just kind of are figuring out, right?

Derek Bruff 34:27

Yeah. So, Josh, are are you uh you you're you're excited about the research. Are you are you hopeful that we'll as a higher ed community continue to figure out how to respond to this challenge?

Josh Eyler 34:40

Isn't that the that's the that's always the game, isn't it?

Derek Bruff 34:44

Um Regan said he's an optimist. I don't know if you're an optimist here.

Josh Eyler 34:47

Um well, it's funny. I'm always an optimist uh when it comes to um the what we can do for our students. And so I think that what's uh the the nature of this conversation is really how do we best help them? How do we best support them in their learning? And uh I think this is a big one that we need to that we need to figure out, and we won't do that by uh by yelling at each other or running away from the issue.

Derek Bruff 35:14

Yeah, yeah, I like that. Um yeah, I was just reviewing, we're putting out a call for proposal or call for applications for our faculty AI program at UVA, and we were trying to shape next year's conversations around four big questions. And one of them we wrote as how might instructors design AI aware assignments and activities to minimize cognitive offloading and maximize learning. And I thought, oh, oh, this paper, right? Like this is gonna help us figure that out. So um I'm excited about that. And and I second your point of not yelling at each other and actually putting our heads together and collaborating.

unknown 35:49

Right.

Derek Bruff 35:50

So with that, I think I'll move to our third paper. Uh, Flower, you get to take lead on this one.

Flower Darby 35:54

Absolutely. I'm here today to talk about a 2024 paper uh that uh represents conference proceedings from the conference on human factors in computing. Lead researcher Varinambi Arachi and colleagues looked at the effect of generative AI on creative design ideation, specifically visual design ideation, and they wanted to see if using generative AI enhances human creativity or hinders human creativity. Now, this was in contrast to other our other two papers. This is a very small-scale study involving 60 students at the I don't know exactly which university, but um university students, um undergraduate and graduate, and it's a between participant study. I uh yeah, I found this study very interesting when I took off my academic hat and put on my creative hat. So, as you may or may not know, I was a jazz dance choreographer for 25 years, and when I thought about reading this paper, I thought about okay, how did I choreograph a routine? How did I put together a dance piece? And I found myself asking the same question from the perspective of my now my 19-year-old, who some people find it ironic, is a florist. We did not do that intention, pick by intention. Pixie is a florist, and I asked them recently, how do you come up with your floral designs? And the answer was the same in both cases. Whether I was creating a dance or they were creating a design, you close your eyes and you could see it. And essentially, I'm putting the cart before the horse here. That is kind of the finding of this particular story, or sorry, this particular article. The researchers do acknowledge the various limitations of it being a smaller size and the technology being nascent and such. But takeaway, they found that using generative AI hindered human creativity in four kinds of ways that are established measured standards in the design literature. I don't pretend to be an expert on design literature, but there is this concept called design fixation. And they provide a super helpful scenario right at the beginning where they say, imagine that there's a group of people in a room coming up with ideas to address public transport problems, and the first idea thrown out there is for an electric bus, and then people get fixated on that one concept, and all the other ideas have to do with electric motor in some way or another until an intern comes into the room a little bit late and says, Well, what about a bicycle? So this design fixation is about getting kind of stuck on one idea. The literature shows that it happens consciously and unconsciously, and there's a lot of investigation in how we overcome that uh that kind of challenge. The researchers of this particular study did not find that generative AI helped to overcome design fixation. They found that it increased in this small-scale study, and they uh also found that it limited fluency, variety, and originality. Just briefly about the design, and then I'll pause and see what reactions folks have. The participants were sorted into three conditions. In all conditions, they received a written design brief. They had to sketch out on paper with pencils and pens ideas for a chatbot avatar, a visual representation, sketches of a chatbot avatar. Each group was given the same design brief and one example, one visual example. In the first condition, that group of people did not have access to any other source inspiration. Second condition, they could use Google image search, and in the third, they could use an AI image generator, specifically Mid Journey V4. And the upshot of the experiment was that uh the people in that third condition actually had less creativity and um less uh they they were more likely, they got more fixated on some of the concepts that the AI images uh had created. So interesting study to me in terms of the conversation that is currently going on about how AI can augment human creativity. And I again, this question has been dancing around in the back of my mind, literally, as a professional dancer, like, all right, what is what does AI mean for the performance arts, right? Uh I have these conversations with musicians. Um, so it was interesting findings, and I can share one final takeaway after I get some some thoughts from all of you here.

Regan Gurung 40:23

Regan? Yeah, let me dive in because uh couple of couple of different things here. You know, this is a great example for me of when do we, and listeners or the four of us here, and especially those listening to this, we should ask ourselves when do we reach to AI and why, right? Uh because here, and I'll give you an example. I'm not a designer or an artist, and I have very little artist still skills, but something I do take some pride in and enjoy is coming up with titles. And uh and you know, every once in a while I'll try AI for a title, and I'll admit it gives me some things I haven't thought of, but most of the time it and I can and and because I'm being very cognitively aware of it, when it takes me down routes that I don't want to go, I just stop. And now I actually I just don't even do that anymore because I find I can be more creative, not even being tempted by it. And that's what I saw here, right? Was that's what really got got me about this study is that it's sort of, you know, and I think we talked about our own biases before. It really gave some evidence to that bias of if something is there, it's you're gonna jump on that train and take it. And in contrast to my case, where I pulled back and go, no, I'm not gonna use that for titles or for whatever. In this particular case, the participants, of course, just went with it. So, you know, uh that for me was a big thing where it's the why do you want to use it? And of course, they were in a study. And I'll yoke that to my other big thing about it here, which was what were the incentives, right? I mean, in stark contrast to the first paper we discussed where it was an actual class. And to be fair, they were standalone lessons, but still it was an actual ongoing class. These are people responding to flyers around campus. They're, you know, they they want to just get done with their study. And if AI is gonna give them some stuff, well, let's do it. So I think that's why that motivation and incentives that Josh, you brought up in that second paper, too. Here's where we go. What's your motivation for using it? Why will you use it? And I think here we see an example of the downside of not being motivated to actually, you know, go into those desirable difficulties that you will like. Yeah.

Derek Bruff 42:39

Yeah.

Josh Eyler 42:39

Josh, what would you add to this? Yeah, that's a really nice point, Regan. Um, so this is such an interesting paper because on in one way, it's so different from the other two. Uh, and another way, there's a kind of bridge uh between some of them. So it's different in the sense that the researchers were looking at something, uh, you know, a phenomenon that already exists, uh design fixation. And they were really just they were asking to see the how does AI fit into this? Does it uh does it also lead to this or or not? And so you intuitively, I think, you see uh that it doesn't. Um but I think the bridge uh between this and the others is really the question of why does why doesn't AI help with this? Why does AI also lead to design fixation? And there, I think, uh is uh the the most interesting question. Um my wife teaches, uh she's an art professor and she teaches drawing and painting. And the the teaching of creativity is not about necessarily the kinds of lines that you're making. The teaching of creativity is what are you interested in? What are your questions? What are what are you thinking about right now? And how might we how might we take that interesting idea and put it into You know, the form of a drawing or a painting or a sculpture or something like that. And that, you know, building on what we were talking about with the other two studies, that to me is provides a kind of answer for why is why does AI also lead to design fixation? Because it is not great at helping people to do that really complex creative thinking that then translates into what we all see visually, right? And so I I think it kind of reinforces some of the other things that we've been discussing over the last few minutes.

Derek Bruff 44:37

Yeah, and I I'll jump in here because I uh um I have lots of things I could say, but I I'm thinking about I know in at this point in my life, my career, I know a lot about learning. And when I need to learn something new, I have a lot of strategies and I understand how it works. And like Regan was saying, like I can figure out when AI might be helpful to my process and when it's going to undercut that process. I know less about being creative. I know a little bit, right? But I I don't I don't I don't know nearly as much about how to be creative. And so I I feel like I might be more likely to fall into these unproductive uses of AI. And then I think about our students who typically know a lot less about learning than we do in this room. And so I think that that gives us a a thing to do as teachers is to try to help them understand how learning works so that they can they can react to these tools in more useful ways, which may mean avoiding the tool entirely. Flower, what you said you had another takeaway here.

Flower Darby 45:35

Yeah, and I just want to comment on your last comment there about it gives us a thing to do as teachers, right? I feel extremely affirmed after today's conversation that we still have jobs and our experience and expertise uh is definitely needed. And that is that leads to one of my two takeaways. Um the first, so I'll I'll start with that one, and that is that these uh researchers also conclude that providing more guidance on how to use, specifically prompting, that's what their takeaway was from the qualitative aspect of this mixed method study, is that the effectiveness of the prompting and then the follow-up and the iterative prompting, that's what the lack of effectiveness, that's what kind of led to more and more design fixation and fixation replacement, as they called it, where we sort of really get glummed onto the one idea. So again, I'm gonna tie that to the comments that we've made here about providing that effective structuring, metacognitive prompting, and reflecting, and and that, right, that's what we can do well. But the other takeaway I thought was fascinating, again, just thinking from a creative aspect, um, they they concluded that in this particular context, with all the limitations that they acknowledge, the 20 minutes that they uh allotted for this study were better spent with paper and pencil and no digital uh input whatsoever. And that really reminded me that we're seeing this recurring theme and recommendation again coming out in the research and the more sort of popular discourse. We're inviting or requiring or asking students to get their own ideas out there first, and then if they're gonna use AI, then see what they can refine as opposed to earlier recommendations, which I have made, right? You know, ask AI to create an outline and then you go in and edit it. Well, I think I think the direction many of us are beginning to go is along the lines of get out your paper and pencil and work for 20 minutes with your own brain, and then you can see where AI may or may not be able to help you. So I I thought that was, I love again the divergence of a really creative sort of aspect and topic, and acknowledging that it's a very different study, but uh I really found this intriguing as well.

Regan Gurung 47:45

Yeah, you know, just one thing to add there. When you talk about, you know, creativity and the piece of paper, uh, this also reminds me about, you know, yeah, we're talking about AI, but when you think about where you get creativity from, I've heard some people say, oh, I that that's why I scroll social media for ideas, right? And I think in this context, it's not just AI and a piece of paper, but I want to I I love the research that shows as long as we create space to think, even being bored, even sitting outside under a tree without paper and pencil, that's been shown to uh lead to higher levels of creativity. So especially when we talk about creativity, you know, I want to see a control condition where it's them sitting down under a tree doing nothing, you know. I mean, the research does show that at least the link increases creativity.

Flower Darby 48:32

So yeah, sorry to jump in, I just can't help myself. I with the metacognitive reflection that again, all of us are adept at doing, I I can look back and realize that many of my best insights and life direction changes, for example, I arrived at while, for example, teaching Pilates. It doesn't require, right? It doesn't require a lot of focus. It's similar to sitting under a tree and I feel the same thing about um other activities which allow the mind to wander. But um yeah, the the the things come to you when the mind isn't actively focused on something else, right? We know that.

Josh Eyler 49:07

Yeah, and Flower, I really liked um what you said just a few minutes ago about how this study and some of the emerging research is really starting to underscore the directionality of student thinking um and how we might want to use uh AI and what parts of student thinking the thinking process. Um the you know, actors have this mantra, you have to learn your lines in order to forget them, and so that uh so that the performance seems natural. And you know, it's always seemed to me that that's uh a nice metaphor for creativity, that uh it requires lots of reading and thinking and dealing with amazing ideas, but not being presented necessarily with the product and and tweaking it then, right? That the creativity comes from in those moments of quiet that Regan and Flower are talking about, having done all of that thinking and preparation work, that then we uh then you connect the dots in a way they've never been connected before.

Regan Gurung 50:08

Absolutely.

Derek Bruff 50:09

Yeah. Well, I think that's a good place to leave it. Thank you uh all three for joining me in this discussion. There's lots more we could say about all of these studies and and explore their their ramifications and applications, but I think we've we've uh we've done a good job um pulling back the cover a little bit on these studies and pulling out some of the most interesting parts. So um thank you for joining me in that process and thank you for uh uh bringing your insight and your perspectives to our listeners here today. Um I really appreciated this.

Regan Gurung 50:38

Thank you. Thank you.

Derek Bruff 50:40

Thanks.

Regan Gurung 50:40

All right, take care.

Derek Bruff 50:43

Thanks so much to our panelists for the second edition of Study Hall. Flower Darby is incoming director of the Center for Teaching and Learning at Estrella Mountain Community College. Congrats, Flower, on the new job. Josh Eyler is Senior Director of the Center for Excellence in Teaching and Learning, an Assistant Professor of Teacher Education at the University of Mississippi, and my old boss. And Regan Gurung is Professor of Psychology at Oregon State University, and actually a former podcast guest. He was on way back in the first year of intentional teaching. See the show notes for links to more information about each of our panelists. They are all worth following online.

Derek Bruff 51:21

Now it's your turn. What are your thoughts on the studies we discussed today? What do you take away from our discussion that you can use in your teaching? You can contact me by email at derek at derekbruff.org or click the link in the show notes to send me a text message with your thoughts. Be sure to include your name if you use the text message option.

Derek Bruff 51:40

Intentional Teaching is sponsored by UPCEA, the Online and Professional Education Association. In the show notes, you'll find a link to the UPCEA website where you can find out about their research, networking opportunities, and professional development offerings. This episode of Intentional Teaching was produced and edited by me, Derek Bruff. See the show notes for links to my website and socials, and to the Intentional Teaching newsletter, which goes out most weeks on Thursday or Friday. If you found this or any episode of Intentional Teaching useful, would you consider sharing it with a colleague? That would mean a lot. As always, thanks for listening.