How much of your students’ grades in your courses is subject to random chance?
Is your first reaction a defensive one?
“Why, NONE, of COURSE!”
We all like to think we are assessing our students fairly, and that the grades they get are some sort of true, objective reflection of their mastery of our course content.
Let’s start with exams. Of necessity, you can’t ask questions about every single thing you covered(*) in class, assignments, and readings. It’s just not practical.
(*It’s a handy fantasy to think that just because you ‘covered’ it, they ‘learned’ it.
It just ain’t so.)
Let’s imagine how it would look if you did create a comprehensive test.
This is a super simplified representation of the course content in my course. I have 8 topics, and each square represents the stuff I covered.
If I create a truly comprehensive exam, then I need to ask things on my exam that touch on every single Bit of content. Again, in my super simplified representation, each of my 8 topics has 25 Bits to it, and so my (let’s say Multiple Choice) exam should have 200 questions, each one addressing a different ‘Bit‘.
That’s completely untenable.
So your exam, of necessity, picks on a small percent of the course content. Even in a 3-hour exam (which, quite frankly, I think is both ridiculous and cruel), you can only touch on a small portion of the course content. Let’s guess you manage to ask questions that touch on a third of the course content.
Each ‘Bit‘ from the content appears in your exam. In the above image, each ‘Bit‘ that appears in the exam has been replaced with a white square in the original course content. Can you see how much of your content didn’t make it into the exam?
- How do you decide which third to include in the exam and which 2/3 to leave out?
- Does your textbook question bank have any influence on which questions you choose?
- Do you let an automated test bank choose random questions for you?
- How much do you trust it?
- Have you ever even thought about that?
- Do you know whether or not the author of your textbook did any statistical analysis on the reasonableness or predictive qualities of their questions? (It might surprise you to know that many authors simply make up questions without ever testing them on real students.)
- Do you ask questions that you believe ‘cover’ multiple Bits?
- Have you ever verified this?
On top of that, also keep this in mind when you look at the “Bits” of your course content. Your course consists of:
- 39 Lecture hours
- Is everything you say in class relevant course content?
- If not, what isn’t, AND how do students know the difference?
- 13 Tutorial Hours
- Are you doing your own tutorials?
- If not, what assumptions do you make about what’s going on in the tutorial?
- How knowledgeable are your teaching assistants w.r.t. the course content?
- How well do your teaching assistants match YOUR teaching style?
- Are all your teaching assistants equally prepared and dedicated to helping your students master the course material?
- Do you include material from the tutorial in your exam?
- 450 pg. Text Book
- How much of the textbook is relevant course content?
- Is it only certain chapters, or is it certain parts of some (or all) chapters?
- Do your students know EXACTLY what parts you consider to be course content and what parts are not?
That’s only PROBLEM 1