Ergodic Switch, part 2: Chubby Thighs

Part 1

If people aren’t ergodic, that means in theory exams and other cognitive assessments have no predictive value for individuals, only for populations. Intuitively, of course, that’s not true. If I needed a Spanish translator, and all I knew from among the candidates was their grades in their most recent Spanish class, I’d pick someone with high scores. That wouldn’t guarantee success, but I’d expect it to correlate with success.

The example Rose uses is instructive. If you hold up a newborn so that his or her legs are dangling, he or she will move his or her legs in a walking reflex. This walking reflex disappears after a time, only to reappear later. Researchers hypothesized that the suppression of this walking instinct is related to cognitive development.

They measured the point at which babies on average lose the walking instinct and compared that average to cognitive development known to happen in that same timeframe and linked the two.

Some doctors, then, naturally, would express concern to parents if their baby didn’t lose their walking reflex in that timeframe. It became a symptom of delayed cognitive development.

That is, until a researcher came along and instead of worrying about how babies compared against the average baby, measured each baby’s loss of walking reflex against a variety of other physical measurements specific to that baby. The culprit: chubby thighs. As babies put on weight, their leg muscles simply can’t power their legs anymore. The reflex returns as their legs strengthen.

According to Rose, the difference between the two research approaches is “aggregate, then analyze” versus “analyze, then aggregate,” with the former being more typical given our drive to compare ourselves and others against averages.

I’m not positive Rose’s is a great example of the problems of bias toward comparisons with normal. The original researchers” focus on an incorrect hypothesis (cognitive development versus physical development) left them trying to find a correlation to something unobservable. Still, Rose’s point is that had researchers focused on measuring, in each subject rather than across subjects, the relationship between the walking instinct and cognitive development, they may have discovered decades earlier the lack of a correlation.

More to come…

Advertisements

2 thoughts on “Ergodic Switch, part 2: Chubby Thighs

  1. Rob Foshay

    Funny, in 1971 I first learned about the fallacy of applying averages to individual students from E.L.Thorndike, Jr. (son of one of the founders of psychometrics). He emphasized, repeatedly, that tests measure *correlates* of their constructs, NOT the constructs themselves. No test correlates 1.00 with the construct, so there are always boundary conditions to tests. Consequently, when a teacher needs to make a decision about a particular student, based only on test data, there is no way to know if the that particular student’s test results truly indicate that student’s level of whatever the construct is. This is the fallacy of particularizing an average, which Rose appears to use as his premise. Stated differently: the map is not the territory.

    This fallacy applies to the practice of clinical medicine, and is the source of a continual debate over to what degree data-driven practice should win out over a doctor’s clinical judgement. Or, the degree to which a health insurance company should use that data to guide (and override the doctor’s judgement) about what procedures to approve.

    Any teacher (or instructional designer) should understand this fallacy when dealing with individual students.

    Now you know why NCLB was predicted to cause such damage, and why it did. I could have told you in 1971.

    Incidentally, I proved this for myself using stepwise multiple regression analysis on my dissertation data. But that’s another story.

    Reply
  2. robertmulcahy Post author

    Hi, Rob! Love this: “The map is not the territory.”

    Related to your point on medicine, one of the books I read recently (Peak, I think) asserted that doctors in most fields don’t get reliable feedback about whether their diagnoses and prescriptions are correct, and thus fail to become significantly better at their craft (with notable exceptions like surgeons, where it is generally clear whether you got it right or not). I don’t know if that is true or not (certainly, a couple of doctor friends of mine didn’t think much of the assertion), but it was thought-provoking. Your comment provides a nice counterpoint.

    Reply

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s