Paucity of P-Values

How Effect Size is Immensely More Important than P-Value; Small Effect Sizes for Educational Techniques Explainable by the Pygmalion Effect; Reporting Bias; Positive Outlier in Hostos Study

One of the very nice things about the recent conference papers was getting a chance to dig into statistical research in a field that I'm actually knowledgeable about. Of course, I've taught college statistics for approximately 7 years at this point. In the last year I started assigning a research project to report and interpret a medical article of the student's choice from JAMA -- which is a fair bit of work for me, since I'm not previously aware of the medical issues or terminology involved (and as an accidental side-effect of this assignment, I've gotten a rapid-immersion pickup of lots more medical information than I ever expected). So here's an opportunity to see how our statistics apply to actual math education issues.

In my statistics course, P-value statements are among the "crown jewels" of the course (assessment of reasonableness of a hypothesis such as "do average test scores increase with this technique?"), and frequently the last thing we do in the course. It takes the whole semester as prep-work, and then about a week of lectures on the subject. It's an interesting and clever piece of math which can frequently establish whether results in the population are, on average, improved. For example: A JAMA medical journal might say, "Infection was... increased with number of sexual partners (P < .001 for trend) and cigarettes smoked per day (P < .001 for trend)." [Gillision, 2012, "Prevalence of Oral HPV Infection in the United States, 2009-2010"] As our textbook would say, this indicates extremely strong evidence that the claim was true for the population in general (lower P-values being better; sort of a probability of being wrong in some sense).

The somewhat thunderbolt-realization I got from the math-education articles I've been reading is that suddenly, I kind of don't give a crap about the P-values. What I really care about is the effect size; how much did scores go up (if any)? We want to make a cost-benefit analysis on completely overhauling our classes; is it worthwhile? Granted that some increase exists -- is it useful, or negligibly small?

A textbook will usually discuss this briefly; "Statistical significance is not the same as practical significance", but until now I didn't realize how immensely critical that was. Several of the papers I'm looking at take some delight in spending several paragraphs explaining what a P-value is, and how it can establish overwhelming likelihood for (some, possibly negligible?) increase in average test scores. Others get sloppy about saying "these findings are very significant!", not specifying statistical significance -- which is to say, possibly not actually significant at all. P-values for some change are somewhat interesting, and in the JAMA article I think they're worth the 3-or-so words expended on them ("P < .001 for trend"), but not any more than that. Noting that most of our math instructional papers gloss over this without highlighting effect size, Wikipedia says this:

Reporting effect sizes is considered good practice when presenting empirical research findings in many fields. The reporting of effect sizes facilitates the interpretation of the substantive, as opposed to the statistical, significance of a research result. Effect sizes are particularly prominent in social and medical research. (Wikipedia, "Effect Size")

This perhaps brings to mind a joke:

"There are many kinds of intelligence: practical, emotional. And then there’s actual intelligence, which is what I’m talking about." (Jack Donaghy, 30 Rock)

So the truth is, most of the recent conference papers are demonstrating fairly small effect sizes from any of the various techniques tried; for those that showed any significance at all, it's something like a 2, 5, or 7% increase to final exam scores or overall passing rates (as one said, "about two-thirds of a letter grade"). Is that worth the effort entirely overhauling an educational program for? It seems to me like effect sizes of this amount are quite likely to be accounted for by one or more of the following well-known testing phenomena:

• Novelty Effect -- Subjects tend to perform better for a limited time after any kind of change in the environment.
• Hawthorne Effect -- Subjects in an experiment may increase performance because they know they're being studied.
• Pygmalion Effect -- If a teacher simply expects students to perform better, then students do in fact perform better.

Let's think about that last one a bit more. It seems pretty well-known that if researchers are invested in the outcome of their research in an educational setting, results tend to track their expectations/incentives (perhaps they will put more energy than usual into the new technique, etc.; something that won't scale to other instructors in general). In medicine, for example, that's analogous to the reason why you want double-blinded trials. Robert Rosenthal presented findings (in a meta-analysis of hundreds of experiments) that interpersonal expectancy by the teacher has a mean effect size on learning and ability of r = 0.26 -- that is, r^2 = 0.07, such that it alone explains 7% of the variation seen in outcomes (Rosenthal, Dec-1994, "Interpersonal Expectations: A 30-Year Perspective", in Current Directions in Psychological Science).

So now let's momentarily limit ourselves to considering the studies from this conference that that did not have the primary investigators specially teaching the experimental groups (my hypothesis being: those studies will show reduced effect sizes as regards to overall passing rates). There are 3 of the 10 studies where this can be established. First: BMCC (in Algebra) which had randomized instructor assignments -- control pass rate was 32.8% vs. treatment pass rate 36.4% (sample effect +3.6%; P = 0.3013, not statistically significant). Second: Laguardia (in Algebra) which only made extra outside tutors available (no change to in-class teaching) -- control pass rate 56.6% vs. treatment pass rate 58.9% (sample effect +2.3%; P = 0.471, not statistically significant). Third: Brooklyn College (in Precalculus), which scrupulously avoided having investigators teaching -- They report fail/withdraw rates and a somewhat questionable statistical procedure (assumes that control group counts as population, generating P = 0.0869), which I'll re-do myself here. Control pass (non-F/W) rate was 67.46%, treatment pass rate 77.55% (sample effect +10.09%; running a two-proportions z-test [per Weiss Sec 12.3] gives me P = 0.0749, only moderately-strong evidence -- not strong or very strong -- for the trend according to my book). A little more info on the last one: A two-proportion z-interval calculation for the improvement in population passing rate gives (95% C.I.: -2.42% to +22.60%), which is to say, the demonstrated effect size is less than the margin of error.



The other interesting lesson here is that uniformly, there's always "reasons why more research is called for" (or some-such). If a technique did not show improvement, then there's a paragraph explaining extenuating or confounding reasons for that, that could be fixed in a future round of research (but there's never a parallel explanation for why a significant result was accidental or one-time-only). Out of 10 research papers I'm looking at, no one ever said, "The results of this study show no evidence for this technique improving scores, and therefore we recommend not pursuing it in the future." (I guess that can be called "Reporting Bias".) Likewise, the introduction from the university goes on about how many "positive effects" of the various studies, but again, I'm not seeing effect sizes that are tremendously useful.

There is one outlier in all this. The study from Hostos claims a near-doubling of remedial class passing rates (from 24% to 43%) when online software is used for homework assignments (specifically, MathXL). It's a short report, and a bit unclear on the study setup, and whether these results are just for Arithmetic or also Algebra classes. (Note that the following report from City College on using Maple TA software for Precalculus classes showed showed no statistical difference in performance). I trail-blazed using MathXL myself at another college about 10 years ago, but didn't find as much improvement as I expected at the time (plus lots of technical complaints, can't-get-online excuses, not accessible to vision-impaired students, etc.) I'd like to see more clear information on exactly how this was achieved.

(Consider how this relates to recent troubling revelations that most published medical results cannot be reproduced.)

Math Attitudes

A Contrarian's Take on the Issues of Math Anxiety, Confidence, Motivation, and "Fun"

This discussion strikes pretty close to the heart of this blog. From the recent math conference, here are some comments from speakers, reports, and power-point slides on the issue of math attitudes:

"Attitude is key; we must convince students that math can be fun and easy."

"The important thing [about this technique] is that your students are happy."

"There is no other way to make students mathematicians than to make them happy."

"This technique did not demonstrate an increase in test scores, but it did increase student confidence."

"In particular, [this teaching approach] stresses the principal importance of elimination of mathematics anxiety -- the main barrier to success in mathematics."

Okay; so as always, I'll be the contrarian and come down on the other side of this issue. While others talk about math being beautiful or elegant or fun or (classically) musical, my gut-sense of it is more of a horrible brutal war that you commit to out of dire necessity. Personally, I could find math important, or insightful, or transformative, or useful, or even an addiction, but rarely has it ever been "fun" to me. See the inaugural post for this blog -- The MadMath Manifesto. I don't see any evidence that confidence is a signal of ability (in fact, some of my most confident students are the most helpless -- see also Socrates), nor do I think that "fun" is either a requirement or even something very informative to talk about (which reiterates stuff I've said vehemently on my gaming blog). Intriguingly, neither does at least one specific research article from this conference:

On the first day of class in Spring 2011, all Precalculus students were given a customized 20 minute Diagnostic Test that check basic Pre-algebra concepts and a Motivation Test that measured student motivation as a prediction of success... However, as Figure 4 shows, there is no correlation between the Diagnostic Test and the Motivation Test (n = 159, r = -0.07). This counter-intuitive outcome indicates that student motivation is not a factor influencing the outcome of the Diagnostic Test. The math scores are low whether or not the students are motivated... it is reasonable to conclude that students are motivated, but genuinely do not know or cannot recall the prerequisites at the beginning of the semester. (Kingan, Clement, and Hu, 2012; "The Gap Project: Closing Gaps in Gateway Mathematics Courses")

Now, I don't find this result "counter-intuitive" in any way (again; in some sense it's the whole point of this blog since day one). To me, I think if anything, the perceptual problem for our remedial students is not that they are insufficiently confident and upbeat; I worry that they are not sufficiently aware of what grave peril that they're actually in (like, a 10% chance of getting a degree within the next 3 years; not something that is communicated to them in an honest fashion), and the amount of time and effort they need to devote to their math studies if this is truly a priority for them.


Likewise, the Dunning-Kruger Effect seems critically important -- the observation that tremendously unskilled people will radically overrate their ability at a certain task. Note in the analysis above, the value r = -0.07; if anything, in this sample high motivation is somewhat correlated with reduced math ability (not that the effect is very strong or significant). So to me, remedial students' "fear" of math is largely an honest and accurate assessment of their very weak skills. If we have an alternative teaching method that raises confidence, but leaves math ability unchanged, then is that not actively harming them -- changing their self-assessment from accurate to inaccurate (exacerbating Dunning-Kruger)?

I suspect that "math-anxiety" is a convenient whipping-boy that frequently lets students, teachers, and administrators avoid talking about actual, concrete math and quantitative reasoning deficiencies (which is frequently sad and hard). Consider another article by Clark: "Antagonism between Achievement and Enjoyment in ATI Studies", which found that when given the chance, both weak and strong learners chose the learning style which produced the least amount of learning (reversed for each type) as preferred -- i.e., the more enjoyable, the least work, the most "fun".

Finally, consider findings by Robert Rosenthal (Harvard researcher, famous for investigations on the Pygmalion Effect): 

"The most surprising finding in our research," says Rosenthal, "has to do with what we called the 'psychological hazards' of unexpected intellectual growth." When so-called "lower track" students in the control group at Oak School (students who were not expected to shine) began to show marked improvement and growth, their teacher evaluations on such things as "personal adjustment," "happiness," "affectionate" declined. ("Pygmalion in the Classroom")

Rosenthal sees this as society (teachers) punishing those who break its expectations, but I think an equally defensible interpretation is that actual learning growth is related to struggle and challenge and resistance, and inversely related to happiness.

Perhaps the real over-arching struggle in all of our different teaching techniques is the fight to get students to actually spend the required amount of time studying and practicing their math -- whether we try to do that in-class, out-of-class, with online software, outsourced video lectures, extra math tutors/coaches, or whatever. Ultimately (as I say on the first day in all my classes) it's patience and focus that are both the requirement and the end-goal of remedial math classes -- and, much like the Samurai-mindset, excitability-fun-confidence are likely to be either uncorrelated or negative indicators for those traits.

Math Conference

Initial Thoughts on the CUNY 2012 Mathematics Conference

About two weeks ago, I had the good fortune of attending the CUNY 2012 Mathematics Conference on Effective Instructional Strategies. To a large degree, this is a product of CUNY's Improving Undergraduate Mathematics Learning (IML) program, which in 2009 vetted and funded 10 research programs at various of the school's colleges. Many of the final papers were presented at this conference, and I'll plan to spend several posts presenting some of my thoughts on them -- you can see the final reports here.

As noted in many places, several of the reports, and on this blog previously, at least half our (my) job is teaching non-credit remedial Arithmetic and Algebra classes for students who can't initially pass a placement test for those subjects, and so many of the research projects were looking for ways to improve that teaching. (Again: Nationwide, about 60% of CC students need such remediation, and only about 30% complete it after some number of years.) Interestingly, within the hour of my sitting down to write this, a top AP headline crossed the news wire on exactly this issue: "Experts: Remedial College Classes Need Fixing".

The strategies looked at in the different CUNY research projects were fairly wide-ranging, and usually required total overhauls of the classes in some way. In general, they seemed to usually be one of: (1) group work/project-based exploratory learning, (2) online/software-based homework and exercise applications, and (3) "inverted" classrooms where video lectures were watched before class, and discussion and exercise drills run in-class.

Compare to my previous blog post. On the one hand, the American Educator article by Clark, et. al., on "Fully Guided Instruction" had me thinking that the group-work-exploration craze was petering out, but apparently that's not quite the case. Consider also, say, David Klein's anti-reformist essay "A Brief History of American K-12 Mathematics Education in the 20th Century" (section "Public Resistance to the NCTM Standards"):

To understand the public backlash against the NCTM math programs of the 1990s, one needs to understand some of the mathematical shortcomings of these programs... Student discovery group work was the preferred mode of learning, sometimes exclusively, and the guidelines for discovery projects were at best inefficient and often aimless... Arithmetic and algebra were radically de-emphasized. Mathematical definitions and proofs for the higher grades were generally deficient, missing entirely, or even incorrect. Some of the elementary school programs did not even provide books for students, as they might interfere with student discovery. 

I would say that each of these identifiers from the 1990's programs appeared in at least one of the research programs from the IML.

At a certain point, one of the speakers referred to Clark's research that expert learners do well with discovery-based methods, and novices do better with fully-guided instruction -- that being the same Clark who inspired my previous blog post (see more here). I was nodding along with this line of reasoning as very important, and then the speaker concluded with the line, "And since most of our students are aged 20-25, they count as experts, and need self-directed methods", at which point I almost fell out of my chair. (If our students can't pass a basic arithmetic/algebra test, then I don't see how it's valid to conclude that they're experts.)

More thoughts later.

Fully Guided Instruction

Highly Recommended Reading by Clark, Kirschner, and Sweller in American Educator Magazine: "Putting Students on the Path to Learning: The Case for Fully Guided Instruction".

I suppose that the "math wars" of the 1990's aren't entirely over yet. For example, for several years I've gotten this one magazine every other month called the NEA Higher Education Advocate (which in this household is referred to as "the crappy teaching magazine"). Every edition has a central keynote article under the heading of "Thriving in Academe", which almost uniformly features a call to "reform" type instructional techniques such as group-work projects, exploratory/discovery-based learning, and the like. When it comes in every other month, I tend to say, "Ah, the Pravda has arrived."

Among the funny recurring jokes of the feature is that it always has a "Issues to Consider" section (FAQ, basically), which frequently fields the question of "Won't this take more time and effort/ Not allow us to cover as many topics?" And the answer is usually some flavor of "Oh, yes!" For example, from the Dec-2008 article on "CRISP" (there's usually someone peddling a new acronym/system in every issue):

Won’t being C.R.I.S.P. cause me to sacrifice coverage? 

Of course. The sciences are especially concerned with complete nomenclature. They are worried that if a Biology 101 student doesn’t learn every bone, muscle, and organ in the body, the student won’t be prepared for Biology 102, not to mention advanced study in related fields such as nursing and exercise and sports science. However, since studies indicate that students will “forget” (they actually put the information in their short-term memories only) 75 to 90 percent of the material in three months anyway, shouldn’t you worry more that students develop skills and fundamental concepts? If students truly comprehend, for instance, how the bones work in general, shouldn’t they be able to figure out how a specific bone functions or know where to look it up? 

So in contrast, the American Educator magazine is what I call the "good teaching magazine" and seems to have much higher-quality, more in-depth, and more interesting articles in each issue (it's published on a quarterly basis). The current "Fully Guided Instruction" article by Clark, et. al. was a bracing breath of fresh air, representing the opposite point of view, that attempting to have students "discover" principles on their own is a weaker technique than instructors cutting to the chase and simply telling them how things work in a straightforward manner (and modeling proper usage, and then overseeing practice). It cites seemingly strong research that the two techniques can be appropriate for different groups of students: in particular, strong students (with pre-existing deep background knowledge) work well with discovery-based learning, whereas weak students (those with deficiencies) do better with explicit direction. In fact:

Worse, a number of experiments found that less-skilled students who chose or were assigned to less-guided instruction received significantly lower scores on posttests than on pretest measures. For these relatively weak students, the failure to provide strong instructional support produced a measurable loss of learning.

So this seems particularly relevant to my work, over half of which is teaching remedial arithmetic and algebra at a large, urban community college (the stats for us, and nationwide, being about 60% of students taking remedial math, and only about 30% successfully completing it after 3 years).

And the other fascinating thing in the article was the description of a decades-long history of similar "discovery-based" reform efforts since at least the 1950's, each of which have been given a new name and similarly came up empty with research-based results for it. Highly recommended reading.

Necessary but Not Sufficient

Here's a random math/logic-book exercise I'd like to see:

The picture above shows a door with triple locks. Answer the following questions:

(a) Do you need the key to the top lock to enter this door? Yes.
(b) Will you be able to enter this door if you have only the key to the top lock? No.
(c) What's the technical term for this relationship? Necessary but not sufficient.


[Photo by LiGhtSynC under CC2.]

"Something Highly Unlikely"

On the Need to Establish Hypotheses Prior to Testing; Or, The Fact that It Is Overwhelmingly Likely that Something Highly Unlikely Will Happen in Any Experiment.

Lately I've gotten in the habit of doing several card-drawing demonstrations in my statistics classes (as concrete examples of sampling, estimating a population mean, hypothesis testing, interpretations, etc.) Here's one for a test question that I sometimes ask: "Is it acceptable to decide what type of test to conduct [left, right, or two-tailed] by examining the sample data?" Say that I bring in a deck of playing cards, shuffle, and deal out 6 cards. For example, when I just did this at my desk I got this:

Now, what follows in this paragraph would be an example faulty reasoning -- Note that I just got duplicate 4's in this draw, and of course, the probability of that happening is highly unlikely in a standard deck (specifically, a 6% chance to get two or more 4's *). Therefore, one might conclude that I doctored that deck with extra 4's.

What's wrong with that reasoning? Well, we didn't establish the hypotheses prior to testing, so it's unfair and biased to use this as data in support of that hypothesis. Or perhaps it's better to look at it this way: It's overwhelmingly likely that something highly unlikely will happen in any such experiment (if you look at the data post-facto and labor to draw out some weird numerology-like pattern). Specifically, the chance of getting some duplicated card value from a standard deck in this case (not necessarily 4's) is actually 65%. **

So let's try this again: It is fair to use the first draw as suggestive of a new hypothesis. Let's hypothesize: "He doctored this deck with extra 4's". So if we shuffle and draw 6 cards again, then we should expect to see one or more 4's. And when I ran this experiment just now the result was:

Which rather obviously destroys the hypothesis; in this case I didn't get any 4's at all. (I did get duplicate 8's, but again, normal probability says that you'll usually get duplicate somethings from a standard deck when drawing 6 cards, so it's not really surprising or interesting at all.) To be doing interesting science, you have to establish coherent hypotheses in advance, and be able to predict and replicate your results.

---

Calculation Footnotes:
* Drawing 6 cards: Chance to get zero 4's is: 48P6/52P6 = 0.603. Chance to get an initial 4 and then all non-4's is: 4×48P5/52P6 = 0.056; so chance to get a single 4 in some order is 6×0.056 = 0.336. Sum of these is 0.603+0.336 = 0.939. Therefore, the chance to get two or more 4's is P(not zero or one 4)  = 1 - 0.939 = 0.061 ~ 6%.

** Drawing 6 cards: Chance to get no duplicates is 52/52×48/51×44/50×40/49×36/48×32/47 = 0.345. Therefore, the chance to get at least one duplicate is P(not zero duplicates) = 1 - 0.345 = 0.655 ~ 65%. (And as a sanity check, the preceding should be approximately 1/13 this, i.e., 65%/13 ~ 5% which does check out.)

Visualizing the Power Function

After some recent discussion on the power function, it occurred to me that if I tried to visualize the function b^a, I really didn't know what it would look like. Here it is below:



As shown here, the base b is along the x-axis, and the power a is on the y-axis. The range displayed is between +/-2 on both axes, with the origin (0, 0) in the center. Positive values are shown in red, negative values in blue; intensity is scaled to the highest value in the top-right corner (i.e., 2^2 = 4). Black pixels represent either very small values (on the right half, for b>0) or else undefined values (on the left half, for b<0).

So a few things are apparent. In the 1st quadrant, going towards the top-right, you get larger positive values (when b>0 and a>0); near the y-axis in that quadrant you get diminishing values, namely 0 when b=0. But in the 4th quadrant the situation is reversed: you get diminishing values towards the bottom-right and arbitrarily large value near the y-axis (hence the intense bright region on the bottom, with vanishing b and negative a, generating values much larger than what you get in the top-right). On the left-hand side (b<0), the graph is mostly black, with only narrow bands of value where the power a is an integer (alternating red and blue, as the powers alternate positive or negative values).

One discovery regarding that left side: I didn't realize how contentious it was to possibly define rational exponents for a negative base! Apparently some textbooks go either way with that. For example, the textbooks at my school permit it, but they have to institute a clunky "assume root exists and exponent on b^(m/n) is reduced to lowest terms" definition, so as to avoid a contradiction like -2 = (-8)^(1/3) = (-8)^(2/6) = ((-8)^2)^(1/6) = (64)^(1/6) = +2. On the other hand, you have academic papers such as from Tirosh/Even, "To Define or Not to Define: The Case of (-8)^1/3" (Educational Studies in Mathematics, Vol. 33 No. 3, Sep. 1997) which point out this problem and others, and recommend leaving them undefined. I think I'm personally convinced by that. Hopefully we all agree that irrational exponents to negative bases are undefined, so the left-hand side of the graph above really does need to be black almost everywhere.

And then of course you've got the case of 0^0, which I'm likewise convinced (again contrary to the books at my school) should be defined to be 1. On the one hand, the horizontal axis a=0 will definitely have a value of 1 for all b with that possible single exception. While on the other hand, the vertical axis has a value of 0 coming down from the top, but you're going to have a discontinuity at the origin no matter what; either a value of 1 when a=0, or, failing that, an undefined value as soon as you take an arbitrarily small step below the origin (since given negative power -a, 0^(-a) = 1/(0^a) = 1/0 which is undefined). So I figure you might as well define 0^0 = 1 and the loss in continuity on the vertical axis is immeasurably small.

In summary: Certainly not a picture I could have intuited, and one with unexpectedly complicated structure and more regions of controversy regarding definitions than I expected (granted such a fundamental function as b^a).

Possibly there's some additional use in seeing a spreadsheet of numerical values from the same region, below:


Download if you want:
- Java source code (JAVA) to generate the image
- Open Document Spreadsheet (ODS) for the numbers
- Wikipedia 3D graph of similar region (positive base only)