Let's say you have a final-exam assessment that is a multiple-choice test, with 25 questions, each of which has 4 options, and requires a 60% score (15 questions correct) to pass. As one example, consider the uniform CUNY Elementary Algebra Final Exam (link).
How robust is this as an assessment of mastery in the discipline? As a simple model, let's say that any student definitely knows how to answer N types of questions, but is randomly guessing (uniform distribution over 4 options) for the other questions. Obviously this abstracts out the possibility that some students know parts of certain questions and can eliminate certain choices or guess based on the overall "shape" of the question, but it's a reasonable first-degree model. Then the chance to pass the exam for different levels of knowledge is as follows:
Obviously, if a student really "knows" how to answer 15 or more questions, then they will certainly pass this test (omitted from the table). But even if they only know half of the material in the course, then they will probably pass the test (12 questions known: 67% likely to pass). Of students who only ever know about one-third of the basic algebra content, but retake the class 3 times, about half can be expected to pass based on the strength of random guessing on the rest of the test (9 questions known: 19% likely to pass; over 3 attempts chance to pass is 1-(1-0.19)^3 = 1-0.53 = 0.47).
Thursday, April 3, 2014
Interesting article about Dr. John Ionnidis at Stanford founding the "Meta-Research Innovation Centre" to monitor and combat weak and flawed statistical methods in science research papers, especially medicine. Good luck to him!