2022-09-04

NY Regents: Trivial to Pass

Multiple choice with all C-answers

Ed Knight is a teacher in New York state. Writing at Medium, he points out the disturbing fact that the vaunted "New York Regents" exams required to graduate from high schools in the state have become completely trivial to pass. For example: In the Algebra Regents, you can ignore all of the (already simple) open-response questions, and just mindlessly mark "C" for all the multiple-choice questions, and you'll be given a passing grade.

Shame on NYSED and the Regents.

Really, the root of this problem is the insane scaling procedure that the NY Regents has been doing for years to fake up the test scores. Below is the most recent test's table for converting a "Raw Score to a reported "Scale Score". The scale score is 0-100, making recipients thinks it's a percentage, but it's not. For example: if you score a raw 27 out of the possible 86 points (that's 31% correct), this then gets converted to a reported Scale Score 65 -- i.e., a Performance Level of 3 out of 5, which is considered passing.

Think about that: for years, the NY Regents has considered a score of about 30% as passing for a basic (very simple!) algebra test. And yes, this was exacerbated because for the pandemic years (still ongoing), the policy was adjusted to accept even lower scores than that -- now as low as 20% (i.e., Raw 17, reported as a Scale 50). 

Scoring for Regents Exam in Algebra I: June 2022

Read more at Medium: Guessing C For Every Answer Is Now Enough To Pass The New York State Algebra Exam

2022-08-08

Curriculum in Califormia

Cover of California Math 6th-grade textbook
A nicely comprehensive article late last year outlines the plans for the next overhaul of school mathematics curriculum in California -- delaying any algebra until high school, cutting and compressing later classes to fit the reduced time, disposing of gifted & talented or accelerated programs, de-tracking, etc. As usual, the motivation for this to hopefully see higher pass rates from the easier courses, claim to better support inequalities among minorities, be better positioned for well-paying STEM college programs and careers, etc. Among the fonts of battle are particularly academic math professors vs. math-education faculty, who are generally on opposing sides of the issue. 

One thing that really stuck out to me was the case of one student, who's held out as being the one black student throughout her advanced math courses in school, and currently studying as a math major at UC Berkley. Here's how her story is presented: 

Mariah Rose, a third-year applied math major at UC Berkeley, said she didn’t have another Black classmate in any of her math classes until this semester.

“There’s one other Black student in my class right now, and that’s just crazy to me,” said Rose. “The number of Black and Brown people in math is so low.”

Rose, who is half Black and half Latino, said this is nothing new. She said she was the only Black female student in her advanced math classes during high school. And her successes in math make her an outlier in California’s public school system where Black and Latino students score lower on standardized tests...

Rose, the UC Berkeley math major, said she has mixed feelings. She agrees with the framework’s recommendation to delay more advanced math classes and avoid labeling students based on their math abilities at younger ages. But she isn’t sure if she would be where she is if she hadn’t been accelerated into a higher-level math class in 6th grade. 

“It was a game changer,” she said. “I don’t know if I would’ve pursued math if I hadn’t advanced so early.”

Read the full article at The San Fransisco Standard.

2022-08-01

Proofs and Applications

"Burden of Proof" on laptop

This is a quote that lives rent-free in my head, and comes up a lot in discussions I participate in.

From Stein/Barcellos, Calculus and Analytic Geometry, 5E, "To the Instructor", p. xxii (1992):

At the Tulane conference on "Lean and Lively Calculus" in 1986 we heard the engineers say, "Teach the concepts. We'll take care of the applications." Steve Whitaker, in the engineering department at Davis, advised us, "Emphasize proofs, because the ideas that go into the proofs are often the ideas that go into the applications." Oddly, mathematicians suggest that we emphasize applications, and the applied people suggest that we emphasize concepts. We have tried to strike a reasonable balance that gives the instructor flexibility to move in either direction.

2022-07-13

Willingham on Automaticity

Some solid thoughts from Daniel Willingham on the need for automaticity in basic mathematics skills (re: automatic-algebra.org) from his article "Is It True That Some People Just Can't Do Math?" (American Educator, Winter 2009-2010):

In its recent report, the National Mathematics Advisory Panel argued that learning mathematics requires three types of knowledge: factual, procedural, and conceptual. Let’s take a close look at each.

Factual knowledge refers to having ready in memory the answers to a relatively small set of problems of addition, subtraction, multiplication, and division. The answers must be well learned so that when a simple arithmetic problem is encountered (e.g., 2 + 2), the answer is not calculated but simply retrieved from memory. Moreover, retrieval must be automatic (i.e., rapid and virtually attention free). This automatic retrieval of basic math facts is critical to solving complex problems because complex problems have simpler problems embedded in them. For example, long division problems have simpler subtraction problems embedded in them. Students who automatically retrieve the answers to the simple subtraction problems keep their working memory (i.e., the mental “space” in which thought occurs) free to focus on the bigger long division problem. The less working memory a student must devote to the subtraction subproblems, the more likely that student is to solve the long division problem.

This interpretation of the importance of memorizing math facts is supported by several sources of evidence. First, it is clear that before they are learned to automaticity, calculating simple arithmetic facts does indeed require working memory. With enough practice, however, the answers can be pulled from memory (rather than calculated), thereby incurring virtually no cost to working memory. Second, students who do not have math facts committed to memory must instead calculate the answers, and calculation is more subject to error than memory retrieval. Third, knowledge of math facts is associated with better performance on more complex math tasks. Fourth, when children have difficulty learning arithmetic, it is often due, in part, to difficulty in learning or retrieving basic math facts. One would expect that interventions to improve automatic recall of math facts would also improve proficiency in more complex mathematics. Evidence on this point is positive but limited, perhaps because automatizing factual knowledge poses a more persistent problem than difficulties related to learning mathematics procedures
.

Get the whole article by Willingham (including citations for all the claims above) at the AFT website.

2022-03-08

Remedial Equality Check

Equals sign

I discovered something last semester that made me insert a new little thing in the first day of my basic-level (remedial, liberal arts) community college math courses. A surprising proportion of my students are very confused about what the equality (=) sign means. 

Now, this isn't a tremendously novel observation, e.g., see: Baroody, Arthur J., and Herbert P. Ginsburg, "The effects of instruction on children's understanding of the 'equals' sign." The Elementary School Journal 84.2 (1983): 199-212. But the new discovery for me was how extremely simple a question it takes to make this visible. 

All I've done is start asking, "True or false? (a) 6 = 6, (b) 3 = 5". So far, everyone, confidently answers "true" to the first. But it seems like fully half of my students answer the second one incorrectly.

Last semester I had at least one or two students who were so enormously challenged that we could repeat this every day all semester and they'd never get it right. "True or false? 0 = 100"; "True"; "No, it's false"; "I don't get it". Over and over again every day, no matter how many times it was explained.

It seems pretty amazing, but there it is. Lots of our incoming community-college student literally don't now what the equals relation means. My best guess is that they've either come to think of it as "and here's the next thing", or that maybe instructors have been double-checking solutions to equations and every single time it's always come out true. 

I suppose it would be a decent research project to interview such students and ask them to explain what they think the equals sign means. Consider trying that for an opening exercise and share what your results are?


2021-11-02

Backtracking Detracking

Tracking station

There's an interesting article from the Brookings Institution last month, on the perpetual debate over tracking in the U.S. school system -- separating classes at the same grade by skill level. 

Whenever this comes to mind I think about the giant debate that occurred at my high school system right around the time I graduated, which saw a new principal hired with a mandate to detrack all of the school's curriculum. I didn't actually experience that, but my sister, who's two years younger than myself, did. This all being 30 years ago as I write this. (I also had a more recent awareness of a friend's child in junior-high-school, who had special-needs students detracked in the same classroom, who would basically scream incoherently all day long and make any kind of learning impossible.)

As usual with education issues, the rocky shoals upon which all proud ships crash is the mathematics discipline. You can pretty easily get away with a mix of skill levels in arts and social disciplines -- read the same texts (or whatnot), and accept that you'll get different levels of interpretations, and it's possible to grade on a relative "best effort" basis. In the hard sciences (things that are based on math), things get harder -- maybe you can get across some core concepts, but people with math skills will be able to dig deeper and make predictions and verifications in ways that other people cannot. But with our mathematical queen, this is basically impossible -- if someone doesn't have the prerequisite ability to read, write, and think in our language, then absolutely nothing will make sense, and they won't be able to interface with it in any way, producing nothing but raw gobbledygook (as I've seen hundreds or thousands of times). A number of times on this blog I've called this the "brutal honesty" of mathematics. That said: it never stops a legion of arts & social-science people from dictating supposed solutions for the mathematics professors, as crazy as that sounds. 

So in the recent article, Tom Loveless of Brookings notes that the "tracking" argument goes back even farther than my 30-year experience:

Research on tracking extends over a century. Hundreds upon hundreds of studies have not settled the debate. The literature is usually described as “mixed,” but with a clear warning that tracking can exacerbate gaps between high and low achievers.[1] Research is more plentiful on tracking as a problem, as a source of inequality, rather than detracking as a solution. Reformers have been hampered by a lack of empirical evidence that abolishing tracking would reduce inequities. Evaluations of untracked schools tend to be based on a small number of schools or on samples that were not scientifically selected to support generalizable findings...

These case studies indicate that detracking may work under certain conditions, but they are less persuasive evidence that abolishing tracking in favor of classes with students heterogeneous in ability, all studying the same curriculum, will work everywhere or even in most schools. A study that forcefully raises that question was conducted by David N. Figlio and Marianne E. Page. They analyzed data from the National Education Longitudinal Survey of 1988 (NELS:88), which followed a random sample of several thousand students from eighth grade through high school and into post-secondary education and work. Using several methods of identifying whether schools were tracked or untracked, Figlio and Page uncovered neutral to positive effects of tracking. The most surprising finding of the analysis was that students from disadvantaged backgrounds appeared to benefit from tracking. Figlio and Page concluded, “We can find no evidence that detracking America’s schools, as is currently in vogue, will improve outcomes among disadvantaged students. This trend may instead harm the very students that detracking is intended to help”.

Ironically, the data that Figlio/Page analyzed was current to the year right before I graduated high school; but the article they published about it wasn't until 14 years later. I wonder if it would have made any difference at the time?

At least as interesting is what prompted the Brookings article at this time: back in May of this year the Washington Post had an article about a contentious push in the state of Virginia for detracking. After parental outcry, the state superintendent was forced to release a statement backtracking from the idea:

Under the VMPI plan, [parent] Fox said, “every student would be required to take the same math class through 10th grade of high school. There would be no classes for struggling students needing remedial help or for advanced students seeking accelerated math.”

When I called Virginia State Superintendent of Public Instruction James F. Lane to ask about this, he insisted that the state has no plans to eliminate tracking (separate classes for students at different levels) from kindergarten through 10th grade, even though the VMPI website strongly suggests that ending tracking is key to the suggested reforms...

Lane, the Virginia state superintendent, is an experienced administrator, having led three school districts. He seems to understand how politically poisonous it would be to tell parents that every child is going to be on the same math track through 10th grade...

Lane’s spokesman later told me “he does unequivocally denounce the idea that every student should be forced to take the exact same math courses at the same time without options for acceleration.”

Will this detracking debate go on ad infinitum?

Brookings: Does detracking promote educational equity?

Washington Post: Virginia allies with, then backs away from, controversial math anti-tracking movement

2021-10-18

Growth Mindset Theory: Failures to Replicate

Psychologist Carol Dweck's "growth mindset" theory has become a popular solution and intervention technique in (mostly American) schools of all ages. We might say that it's become the new version of the "self-esteem" movement seen in the 80's. While Dweck first developed the theory in the 90's, it's really taken hold of popular consciousness from the 2010's on.

Unfortunately, we should remember that psychology has an ongoing replication crisis in many of its landmark findings. Many of the "easy" ideas for transformative effects have not borne fruit over the years, and been later found to have tainted methods by core researchers. Sure enough, in recent years many or most of the large-scale, high-quality attempts at replicating the claims of growth mindset have failed to so. Here are a few examples:

Li, Y., & Bates, T. C., Ph.D. (2017). Does growth mindset improve children’s IQ, educational attainment or response to setbacks? Active-control interventions and data on children’s own mindsets. https://doi.org/10.31235/osf.io/tsdwy (Study done in China, students aged 9-13 years, N = 624)

No effect of the classic growth mindset manipulation was found for either moderate or more difficult material... children’s mindsets were unrelated to resilience to failure for either outcome measure... Finally, in 2 studies relating mindset to grades across a semester in school, the predicted association of growth mindset with improved grades was not supported. Neither was there any association of children’s mindsets with their grades at the start of the semester. Beliefs about the malleability of basic ability may not be related to resilience to failure or progress in school.

Bahník, Štěpán, and Marek A. Vranka (2017). Growth mindset is not associated with scholastic aptitude in a large sample of university applicants. Personality and Individual Differences 117: 139-143. https://doi.org/10.1016/j.paid.2017.05.046 (Study of university students taking an admissions test in the Czech Republic, N = 5653).

We found that results in the test were slightly negatively associated with growth mindset (r = −0.03). Mindset showed no relationship with the number of test administrations participants signed up for and it did not predict change in the test results. The results show that the strength of the association between academic achievement and mindset might be weaker than previously thought.

Foliano, F., Rolfe, H., Buzzeo, J., Runge, J., & Wilkinson, D. (2019). Changing mindsets: Effectiveness trial. National Institute of Economic and Social Research. Summary at PsychBrief. (Study in England, Year 6 students, N = 4584.)

The difference between the control group and the intervention group on all 3 primary outcomes [math, reading, GPS] was 0... The difference between the groups for all 4 secondary outcomes was also 0... This RCT was a highly powered test of the efficacy of growth mindset in a real-world environment across a wide range of schools in the England. The fact none of the primary or secondary outcomes were distinguishable from 0 raises serious questions as to the efficacy of growth mindset for Year 6 students... Given the evidence so far, it is unrealistic to expect growth mindset to have large and/or wide-scale impact.

Caitlin Brez, Eric M. Hampton, Linda Behrendt, Liz Brown & Josh Powers (2020) Failure to Replicate: Testing a Growth Mindset Intervention for College Student Success, Basic and Applied Social Psychology, 42:6, 460-468, DOI: 10.1080/01973533.2020.1806845 (U.S. study, university math & psychology students, N = 2607).

The pattern of findings is clear that the intervention had little impact on students’ academic success even among sub-samples of students who are traditionally assumed to benefit from this type of intervention (e.g., minority, low income, and first-generation students)... These findings support some of the emerging literature that demonstrates that growth mindset interventions may not be as effective as once thought... The proposition that a one-time intervention at the postsecondary level will result in long-term measurable student outcomes was not supported in the present study.


Now, a not-uncommon defense in a number of these cases in psychology is that the attempts to replicate didn't properly recreate the conditions or variables for a true test. The counter-argument here would be the observer-expectancy effect -- in some cases a primary researcher has even argued that only they have the necessary knowledge to ever do so. Indeed, Dweck has made the "not anyone can do a replication" argument (BuzzFeed News interview). In response, Nick Brown, who developed the GRIM (Granularity-Related Inconsistency of Means) test and found several errors in Dweck's seminal paper, said this:

The question I have is: If your effect is so fragile that it can only be reproduced [under strictly controlled conditions], then why do you think it can be reproduced by schoolteachers?

Finally, psychologist Russell Warne wrote on his blog:

I discovered the one characteristic that the studies that support mindset theory share and that all the studies that contradict the theory lack: Carol Dweck... So, there you go! Growth mindsets can improve academic performance –if you have Carol Dweck in charge of your intervention.

This is somewhat hyperbolic, but clarifies the issue at stake. Growth mindset theory fits fairly snugly into the basket of psychological "quick fixes" that make up the replication crisis, broadly cuts against long-standing findings from neuroscience on intelligence, and is racking up more failures-to-replicate as it garners more attention. Like other similar principles that came before, it's probably a bad bet that institutional interventions based on the theory will be worth the resources spent on them.

This post was initially written as an answer on Stack Exchange: Mathematics Educators. Thanks to the community there for reading and refining it.