Wednesday, February 10, 2010

Same class, two sections, two completely different sets of evaluations

I taught two land-based sections of ethics last semester, one right after the other, using the same syllabus and lessons plans, and the course evaluations could hardly be more radically different. The whole thing is a good case study in interpreting student evaluations.

The chart above is just one measure of the difference. In the 10 AM section, a big majority (70%) of the students strongly agree that I was an effective teacher. For the 11AM section, few strongly agree that I was effective, and some even disagree with the idea that I was effective at all. (Note, there is a strong bias toward positive evaluations on this question. You simply won't see evaluations that are the exact inverse of my 10AM question unless the teacher was, like, drunk in class.)

The written comments paint a bigger picture of the 11AM section. I got twenty seven comments, and only four were positive. This was actually the first page I saw when I opened the envelope, and the whole thing hit me like a sack of bricks. They hated me, they hated the textbook, they hated the assignments, they hated the syllabus. Almost everyone used the word "confusing." It gave me one of those "Have I really chosen the right career?" moments. The 10AM section, on the other hand, only had four comments, all of which were blandly positive.

A typical response from teachers here would be that course evaluations really don't tell you anything, and this is evidence for that. Responses are all over the map, and they only reflect the idiosyncratic responses of students, who don't really know what is good for them anyway. But this would be a big mistake.

Thing is, I knew the 11AM section wasn't going well while it was happening. The students were so unengaged that I had the cameraman move to the 10AM section so we'd get better video for distance learning. I didn't realize how bad it had gotten, I think, because the good earlier class put a halo over the later class. But I knew there was a difference. The negative evaluations for the second section reflected a real difference in student experience and student learning.

Here are the real lessons I take home. The first is something every customer service representative knows: unhappy customers give a lot more feedback than happy ones. I don't have two pages of effusive comments about how great I am from the 10AM section, even though they all checked that I was a good teacher.

The second is that you can't design a course that will appeal to all students. Is the design of my ethics course sound? It is for some students. I suspect that this course, in particular, works for students who were better prepared for college. One positive review from the 10AM section said "Its challenging in a very good way." The wave of negative reviews from the section section all said "confusing" and "too advanced for an introductory class."

Ideally, you would respond to this situation by tailoring the class to fit the students, so that if you see that a section is floundering, change the syllabus. But this is a lot harder to do when you have, say, 100 students in three sections of ethics, plus another class or two. To keep your own workload manageable, you need to keep all the sections doing the same material.

Another observation is that classes tend to consolidate around an opinion of you. If any one of the dissatisfied students in the 11AM section were moved to 10AM, they would have softened their critique, because they would have seen other students responding positively, rather than having their negative opinions reinforced.

There are definitely things I'm going to change about the class. The negative comments are far from useless. They also hurt a lot, too. I had to buy some junk food after reading these. I also needed to write this post to put things in perspective.


mcmc said...

Ow. Maybe your 11:00 students were hungry. You should have brought junk food for them, the little monsters.

Josh K-sky said...

That bites. I'm sorry you had to get that response. Bravo to you for trying to extract some useful information from it. But it also sucks.

LizardBreath said...

Same room? I've taken classes in rooms where I couldn't stay awake because of ventilation or heating issues. If it's a different room, I'd think about whether there might be something physical that screwed up the later class.

Rob Helpy-Chalk said...

Same room. I'm going to have to do some major re-evaluating. I've been very reluctant to change the textbook, because its a beautiful piece of work with a unique approach, but LCCC student consistently find it confusing. I might be able to find more popular sources that cover a similar range.

Ariel said...

I wonder if the types of students you had correlated to the time they choose for the class. Students more prepared for school are willing to sign up for an earlier class.

Rob Helpy-Chalk said...

Ariel: I've definitely noticed that difference between morning and afternoon classes, but never between two morning classes.

joel hanes said...

I'm with mcmc.

The 11:00 students skipped breakfast and were deep in low blood sugar ennui.
Orange juice and some kinda bread thing could have helped.

weserei said...

As a counterpoint mcmc and joel hanes--wouldn't the 10:00 crowd be more likely to have missed breakfast, on average? Some people do naturally get up at 6 AM, and some people find it difficult to be out of bed before noon, but most college students that I have known fall between the two extremes--in the nine to ten range. Still, bringing your students juice and scones couldn't hurt.

On a related note, it's possible that it's you that's tired at 11:00. Teaching is hard work even if you love doing it. At my alma mater, most advanced classes were in a 140-minute format, and most professors were visibly flagging by about minute 90. How about having a little bit of coffee and/or juice between sessions?

How does course registration work at your institution? Do students pick their own sections or are they assigned them? It's possible that the office of the registrar is sorting students by seniority or GPA or some such. Worth checking on.

I think you are right to be suspicious of those who tell you to simply ignore critical comments. Some people give the impression of thinking that there is no such thing as good or bad teaching.