Case Studies and Confirmation Bias

I liked The End of Average, but I did find one stretch difficult–Todd Rose spends a chunk of the book describing some inspiring case studies of major corporations who have abandoned HR and performance management systems that assume one path to success, to great commercial effect.

My issue isn’t with the examples, but rather the apparent lack of a critical search for disconfirming cases. Finding cases that support your point is great and instructive. But many companies who embrace traditional methods also succeed wildly, so how do we know it was abandoning average that made the difference. What about companies that tried similar approaches but failed? Is there anything we can learn from them?


Benjamin Bloom and Mastery-Based Learning

I’ve long been aware of Benjamin Bloom (of Bloom’s Taxonomy fame) and his work on mastery-based learning. The argument goes something like this: schools try to move everyone at the same pace, but since most students on a given topic can move either faster or slower than average, you guarantee that most of the class is either bored or falling behind.

But I’ll admit that I don’t think I’ve ever actually read Bloom (not even in graduate school; how is that possible?). So it was interesting when Todd Rose delved a little into Bloom’s work. According to Rose, Bloom found that if you create two identical classrooms, and make one of them move at a traditional pace while the other allows individuals to move at their own pace, the percentage of learners who master the material by the end of the course moves from 20% to 90%.

Bloom also found that learners were jagged in the pace they moved. All learners had content they breezed through, and all had content they struggled with, even in the same subject. As Rose puts it, “equating learning speed with learning ability is irrefutably wrong.” (P. 133)

For me, Bloom’s work raises questions. Chief among them: Bloom was doing this work forty years ago; has it been replicated?

The Pathways Principle


The next principle is the Pathways principle, which postulates that people assume there is one best path to goals, and that faster is better.

Rose uses learning to walk as an example. Parents worry if their babies are late walkers, or if they take a strange path to learning to walk, even though it’s fine; children learn to walk no matter which path they take.

He then moves on to discuss Bloom…

The Context Principle


Rose’s second principle of individuality is the Context Principle, which postulates that, as he puts it, “traits are a myth.” In other words, while we think of ourselves and others as having fixed traits, in fact our behavior is determined far more by context than by traits. I might believe that I’m an introvert, but how introverted or extroverted I act depends on where I am and who I’m interacting with.

This is hard to wrap one’s head around because fixed traits seem intuitively true. Part of this is because we generally only interact with people in one context–we only see co-workers at work-related functions, for instance. But also partly because we create narratives for ourselves and others and then let confirmation bias affirm those narratives. We see what we expect to see.

Rose contends that personality traits and actual behavior at best only correlates at a level of 0.3, suggesting that personality traits can only explain 9% of how we behave.

The context principle, then, asserts that behavior cannot be predicted outside of specific situations. To understand someone, then, you need to understand how they behave across a wide range of situations.

Labeling a child as aggressive, Rose argues as an example, ignores the likely reality that the child is aggressive under some situations but not others, which bypasses an opportunity to try and understand why.

Rose speaks of the famous marshmallow study, which people tend to point to as an example of how some people have good self-control and some don’t. But the original researcher himself didn’t view the study through this lens, and subsequent work had been done to show that children react very differently in situations where they have reason to trust the researcher than in situations of distrust. (This doesn’t really disprove the existence of traits, of course, since it could mean that people override traits when there is good reason to.)

Do I believe in traits? (In Rose’s words, am I an essentialist?) Maybe. It’s important to separate out adaptability from context dependence. Just because we constantly adapt to situations doesn’t mean there isn’t a baseline “me” in there somewhere. That said, I’m sure our true selves are far more variable and less in our control than we believe. I don’t like personality assessments or communication style instruments for this reason; they place us into boxes that limit our possibilities rather than in kaleidoscopes that celebrate our potential.

The Jaggedness Principle

Continuing a thread from here

Rose’s first principle of individually is the Jaggedness principle. This principle asserts that any complex physical or cognitive trait is a collection of multiple traits that can be measured individually and, crucially, any individual is likely to rate above average on some of the traits and below on others. Yet, we tend to look at such complex traits unidimensionally.

The example Rose leads with is big. He demonstrates that there is no definitive answer to the question of which of two men is bigger because big could mean height, weight, etc.

This isn’t a perfect example of the principle because big isn’t something that people really rank order. Intelligence might be a better example. We rank people by IQ or perceived intelligence assuming that intelligence is some monolithic thing and not a collection of jagged traits. We reduce people to a judgement rather than celebrate their internal diversity of skills and interests.

A shortcoming, perhaps, of jaggedness as a principle is that it is not fractal in nature. In other words, big might be a concept made up of multiple components, but the individual components themselves are not jagged. It’s certainly possible to order people by height.

I like the imagery of a jaggedness principle, though. It gives form to the recoil I feel when, as a leader of a team, I’m asked to reduce everyone in my team to a number or category.

Performance Ratings

When I was a kid, I was always enchanted by the notion of ranking things. I’d listen to Casey Kasem count down the top 40 (and once created my own top 100 list of all time favorite songs–“It’s a Mistake” by Men at Work FTW!). It was interesting to see every week in the newspaper which television shows were top ranked. I knew which teams were leading their divisions in the major sports leagues.

I work for a CPA firm; historically, CPA firms rank employees by assigning them a rating of one to five as part of a performance management process. Todd Rose points to Deloitte as an example of how this is changing, as Deloitte had gone away from ratings altogether.

Gartner published original research recently on the effectiveness of raising employee engagement by dropping numerical performance ratings. The Gartner data doesn’t support my intuition, though, as they found drops in engagement, time spent giving feedback, and employee confidence that better performance equals better pay in companies that have abandoned numerical performance ratings.

Gartner contends that only 5% of managers are skilled enough to realize gains in employee engagement from dropping numerical performance ratings.

All of that said, one interesting twist they did throw out is that dropping numerical performance ratings may still be the right answer in industries that place disproportionate importance on numbers (like CPA firms?).

None of this is definitive nor the last word, but it’s worth pointing out that the only quantitative evidence I know of runs against my intuitive position (now my job, as a creature of confirmation bias, is to keep looking until I can find evidence supporting my position!).

Ergodic Switch, Part 3

Part 2

Even if I’m not always enthusiastic about Rose‘s examples, I’ll admit the I find his premise alluring. As humans we constantly worry about how we compare against the average.

On the one hand, perhaps this is motivating, and therefore an essential part of who we are and how we evolved (I was about to write “to become the dominant species,” but that’s classic averagarianism thinking, as Rose calls it).

On the other hand, we aren’t even very good at it. We are all the heroes of our running narrative, and perceive ourselves on average as above average.

More fundamentally, this lens limits our vision. We can’t appreciate people for their unique strengths; we instead reduce them down to one judgement against a vague sense of what’s average or normal.

Rose gives some useful principles to reframe our thinking.

More to come…