The Jaggedness Principle

Continuing a thread from here

Rose’s first principle of individually is the Jaggedness principle. This principle asserts that any complex physical or cognitive trait is a collection of multiple traits that can be measured individually and, crucially, any individual is likely to rate above average on some of the traits and below on others. Yet, we tend to look at such complex traits unidimensionally.

The example Rose leads with is big. He demonstrates that there is no definitive answer to the question of which of two men is bigger because big could mean height, weight, etc.

This isn’t a perfect example of the principle because big isn’t something that people really rank order. Intelligence might be a better example. We rank people by IQ or perceived intelligence assuming that intelligence is some monolithic thing and not a collection of jagged traits. We reduce people to a judgement rather than celebrate their internal diversity of skills and interests.

A shortcoming, perhaps, of jaggedness as a principle is that it is not fractal in nature. In other words, big might be a concept made up of multiple components, but the individual components themselves are not jagged. It’s certainly possible to order people by height.

I like the imagery of a jaggedness principle, though. It gives form to the recoil I feel when, as a leader of a team, I’m asked to reduce everyone in my team to a number or category.

Performance Ratings

When I was a kid, I was always enchanted by the notion of ranking things. I’d listen to Casey Kasem count down the top 40 (and once created my own top 100 list of all time favorite songs–“It’s a Mistake” by Men at Work FTW!). It was interesting to see every week in the newspaper which television shows were top ranked. I knew which teams were leading their divisions in the major sports leagues.

I work for a CPA firm; historically, CPA firms rank employees by assigning them a rating of one to five as part of a performance management process. Todd Rose points to Deloitte as an example of how this is changing, as Deloitte had gone away from ratings altogether.

Gartner published original research recently on the effectiveness of raising employee engagement by dropping numerical performance ratings. The Gartner data doesn’t support my intuition, though, as they found drops in engagement, time spent giving feedback, and employee confidence that better performance equals better pay in companies that have abandoned numerical performance ratings.

Gartner contends that only 5% of managers are skilled enough to realize gains in employee engagement from dropping numerical performance ratings.

All of that said, one interesting twist they did throw out is that dropping numerical performance ratings may still be the right answer in industries that place disproportionate importance on numbers (like CPA firms?).

None of this is definitive nor the last word, but it’s worth pointing out that the only quantitative evidence I know of runs against my intuitive position (now my job, as a creature of confirmation bias, is to keep looking until I can find evidence supporting my position!).

Ergodic Switch, Part 3

Part 2

Even if I’m not always enthusiastic about Rose‘s examples, I’ll admit the I find his premise alluring. As humans we constantly worry about how we compare against the average.

On the one hand, perhaps this is motivating, and therefore an essential part of who we are and how we evolved (I was about to write “to become the dominant species,” but that’s classic averagarianism thinking, as Rose calls it).

On the other hand, we aren’t even very good at it. We are all the heroes of our running narrative, and perceive ourselves on average as above average.

More fundamentally, this lens limits our vision. We can’t appreciate people for their unique strengths; we instead reduce them down to one judgement against a vague sense of what’s average or normal.

Rose gives some useful principles to reframe our thinking.

More to come…

Ergodic Switch, part 2: Chubby Thighs

Part 1

If people aren’t ergodic, that means in theory exams and other cognitive assessments have no predictive value for individuals, only for populations. Intuitively, of course, that’s not true. If I needed a Spanish translator, and all I knew from among the candidates was their grades in their most recent Spanish class, I’d pick someone with high scores. That wouldn’t guarantee success, but I’d expect it to correlate with success.

The example Rose uses is instructive. If you hold up a newborn so that his or her legs are dangling, he or she will move his or her legs in a walking reflex. This walking reflex disappears after a time, only to reappear later. Researchers hypothesized that the suppression of this walking instinct is related to cognitive development.

They measured the point at which babies on average lose the walking instinct and compared that average to cognitive development known to happen in that same timeframe and linked the two.

Some doctors, then, naturally, would express concern to parents if their baby didn’t lose their walking reflex in that timeframe. It became a symptom of delayed cognitive development.

That is, until a researcher came along and instead of worrying about how babies compared against the average baby, measured each baby’s loss of walking reflex against a variety of other physical measurements specific to that baby. The culprit: chubby thighs. As babies put on weight, their leg muscles simply can’t power their legs anymore. The reflex returns as their legs strengthen.

According to Rose, the difference between the two research approaches is “aggregate, then analyze” versus “analyze, then aggregate,” with the former being more typical given our drive to compare ourselves and others against averages.

I’m not positive Rose’s is a great example of the problems of bias toward comparisons with normal. The original researchers” focus on an incorrect hypothesis (cognitive development versus physical development) left them trying to find a correlation to something unobservable. Still, Rose’s point is that had researchers focused on measuring, in each subject rather than across subjects, the relationship between the walking instinct and cognitive development, they may have discovered decades earlier the lack of a correlation.

More to come…

Ergodic Switch

Every measurement of cognitive skills or attributes, such as an academic exam or a personality assessment, is just a sample that, because of luck or fatigue or other factors, may or may not represent your true self. The only way to find out your true score would be for you to take the test a bunch of times and average your score.

But that doesn’t work in the real world because taking the test changes you. So instead a bunch of people take the test and we use the distribution of scores to statistically determine your likely range of scores.

This type of reasoning is ergodic. Ergodic theory, according to Todd Rose, allows you to “use a group average to make predictions about individuals if two conditions are true: (1) every member of the group​ is identical, and (2) every member of the group will remain the same in the future” (p. 63). Ergodic theory was developed by physicists to study the question of whether one could use the average behavior of a group of gas molecules to predict the average behavior of a single gas molecule.

Turns out, according to Rose, most gas molecules are not ergodic. Likewise, Rose cites the work of Peter Molennar to argue that people aren’t ergodic, either, throwing into doubt the validity of instruments like exams and diagnostic tests.

That’s interesting. More soon.

Design Flourishes

I designed a business simulation a few years ago where teams compete to be the most profitable. It is a simulation of a CPA firm, so they are competing to win the audit, tax, and consulting work of various local, fictional businesses. I named the businesses after jazz musicians I was listening to at the time, which was fun for me, but not something I expected anyone else to catch. I was delighted to receive an email recently from one of the instructors sharing the news that one of her students was a jazz piano minor in college and was very enthusiastic about the jazz references in the course. It enriched the course for him.

That made my day.

Way back at the beginning of my career, I was writing worskills exams for PLATO Learning (then TRO Learning) and I remember creating a whole little fictional universe with recurring businesses. It helped make development come alive for me, and I have to believe the more engaged the designer, the better the design, and in the end the better the learning.

At some level, elegant design is about small, delightful details, whether the details themselves get noticed or not (thought it’s great when they are noticed).

Learning Athlete, Part 2: Temperature

I hate freezing cold classrooms.

One of the areas of improvement we are looking at is whether we can gain better control over temperature in classrooms in the hotels and other venues we hold classes. First, we had to ask, what is the optimal temperature range for learning?

The team assigned to this couldn’t find research specific to learning and classrooms, but they did find research on workplace productivity. This metastudy talks about how productivity drops two percent for every degree Celsius outside of 21 to 24 degrees Celsius (70 to 75 degrees Fahrenheit). That’s a little warmer than I expected.

Of course, the age and gender of participants matters.

Then I got schooled by the folks at RSM who work with the hotels. Apparently, it’s not that hotels aren’t aware that their rooms are too cold or too hot; they just have less control than we realize. The way the HVAC systems are set up makes it often impossible to make all of their classrooms comfortable at the same time.

We decided to focus our efforts on making sure site staff are equipped to address concerns as honestly and realistically as possible. It’s no good to tell conference participants that the hotel staff has been notified and is working on it if we know darn well that the situation won’t change. Making someone believe that change is coming if it isn’t is worse than just leveling with them.

Acknowledge their concerns. Be honest about what’s been done and the prospects (or lack of) for change. Offer to get them a hot cup of coffee to hold. But don’t imply that it will get better because that will distract them from learning when it doesn’t.