Sample Size of One

So in response to my review of the Udacity Statistics 101 course (earlier this week) about a hundred people all thought it was clever to say some iteration of, "your sample size is only one course, ha-ha".

The one thing I'll say on this point is that it took me several weeks of full-time work (spread over the summer) to watch, review, interact with problems sets, annotate, organize, draft, and edit my observations for the review. If someone wanted to fund a continuation of this project for other courses and institutions, then I'd be sincerely happy to do so. I would love for people to have a better capacity to identify good teaching and be able to access it, and also crap teaching and be able to avoid it. (Compare to: "Basic Teaching Motivation".)

Thanks to everyone who read and commented, particularly with their experiences in related courses; I think it was (and continues to be) a highly rewarding discussion.


  1. You may or may not know that the Udacity statistics course is being re-vamped, to address some issues including those raised by your previous post.

    I have always believed that a good editor is as important as a good author, and on-line courses are no exception to the rule.

  2. By Gagan Biyani, co-founder of Udemy.com

    I read your first article and thought it was really interesting and valuable, yet came to the wrong conclusion. This follow up was really frustrating, and compelled me to write a comment about it. Your original conclusion (that the poor nature of Sebastian Thrun's course means that online courses are all shit) is HIGHLY illogical as I think you know. But there are TWO - not just one - reasons why this is true. First, the sample size is low and so of course you don't have enough data to make such broad claims.

    Second, you don't acknowledge the difference between an implementation problem and a problem that is inherent to online courses. There's nothing wrong with stating that you had a bad experience with an online course, and honestly your specific concerns with this statistics course sound extremely justified.

    However, its highly illogical (and downright misleading) to say that your experience in any way indicates that there is a bubble in MOOC's or online courses in general. As a statistics and math professor, I'm sure you understand this.

    Take a look through your points again - all but one of them is an implementation problem (meaning that they are not related to the fact the course is online - but just that the course was poorly taught). Basically, the main conclusion that can be drawn from your points is that Sebastian Thrun should not be teaching a statistics course when he has almost no time to dedicate to it.

    I looked at your article and came to the exact opposite conclusion. Despite the shortcomings of the course, it sounds like online courses are a GREAT idea because encourage the kind of open dialogue about content that you were a part of. Simply put, you made one post on a blog about the course (and many others sent feedback to Thrun via e-mail), and voila! he's changing the course for the future. How is that not a great case for online courses vs. in-person (where courses rarely change year-to-year and the curriculum at many universities hasn't changed for decades).

    1. Goodness! Let's protect the potential for shareholder value!

      I teach a flipped statistics course (my second year flipping it), and have taught variations of my course for about seven years now. I read the article, and found it valuable for spelling out how sloppy and poorly planned implementation leads to a poor educational product.

      Evidently you've never taught at a university that actually cares about teaching, because in my university [a comprehensive university, so in the US it would count as a mid-sized university with just under 20,000 students], tenured and tenure-track faculty are called to the carpet if their teaching is sub-standard, so long as enough students complain to the Chair or Dean. Woe unto sessional instructors, as the trigger for the chair's intervention is much lower. Furthermore, all of my colleagues tweak their courses annually, and every five or so years tend to scrap them and try something new. Granted, this is all anecdote, but I submit that a continuous product improvement mindset (e.g., Kaizen) is held by thoughtful and effective instructors.

      While it is ostensibly laudable that Thrun is changing his course for the future, the fact that it is laudable just points to the low esteem held for teaching by those who teach or were educated at world-class research universities. Granted, while he is providing a comparable teaching experience, he at least is not charging an arm and a leg for it...though at the same time he is gathering data and prestige.