In my job which involves teaching lots of remedial classes at a community college (in CUNY), the students frequently have deep, yawning gaps in their basic math education. Many can't write clearly, they interchange digits and symbols, they don't know their multiplication tables, they can't long divide, they have trouble reading English sentence-puzzles, they've been taught bum-fungled "PEMDAS" mnemonics, they've been told that π = 22/7, etc., etc., etc.
So to a large part my job is to ask the question, police-detective style: "Where are the bodies buried?" For this particular crime that's been perpetrated on my students' brains, what exactly is causing the problem, what is the worst thing we can find about their conceptual understanding? Doing the easy introductory problems that immediately come to mind doesn't do dick. What I need to do, in our limited class time, is to dredge the the murky riverbed and pull out all the crap, broken, tricky, misunderstandings that are buried down there.
Another way of putting this is that the in-class exercises we use have to cast a wide net, and be constructed to not just do a single thing, but to demonstrate at least 2, 3, or 4 issues at once. (Again, if you had unlimited time and attention span to do hundreds of problems, this might be an issue, but have to maximize our punch in the class session.) I'm constantly revising my in-class exercises semester after semester as I realize some tricky detail that was pitched at my students along the way. I need to make sure that every tricky corner-case detail gets put in front of students so, if its a problem, they can run into it and I get a chance to help them while we have time together.
This is a place where the poorly-made MOOCs and online basic math classes (like Khan Academy) really do a laughably atrocious job. Generally if you're a science-oriented person who can do math easily, and never taught live, then you're not aware of all the dozens of pitfalls that people can possibly run into during otherwise basic math procedures. So if someone like that just throws out the first math problem they can think of, it's going to be a trivial case that doesn't serve to dredge up all bodies lurking around the periphery. And you'll never know it through any digital feedback, and you'll never get a chance to improve the situation, because you're simply not measuring performance on the tricky side-issues in the first place; it remains hidden and forever under-the-surface.
I'll plan to present some examples of exercise design and refinement in the future. For the moment, consider this article with other educators make the same critical observation about how bad the exercises at Khan Academy (and other poorly-thought-out MOOCs) are: