8/10
- My One-Sentence Summary
- Content Extraction
- Takeaways
My book summaries are designed as captures for what I’ve read, and aren’t necessarily great standalone resources for those who have not read the book.
Their purpose is to ensure that I capture what I learn from any given text, so as to avoid realizing years later that I have no idea what it was about or how I benefited from it.
My One-Sentence Summary
While metrics can and do offer extraordinary benefits when they’re used carefully and properly, there is significant chance of them being chosen incorrectly, being gamed and corrupted by various parties, and ultimately becoming toxic to the very cause they were created to help.
Content Extraction
- Empower others, but be willing to step in with micromanagement temporarily if things get out of hand
- Don’t be so dominating and intimidating that your leaders can’t step up and lead themselves
- Discipline is great, but too much leads to a lack of creativity
- Too much creativity and not enough discipline leads to sloppiness and mistakes
The more a quantitative metric is visible and used to make important decisions, the more it will be gamed—which will distort and corrupt the exact processes it was meant to monitor.
An adaption of Campbell’s Law
And the second is a similar, more simplified version of the same, by Goodhart:
Anything that be measured and rewarded will be gamed.
An adaption of Goodhart’s Law
All quotes here are from the book itself unless otherwise indicated.
“The three components of great training are realism, fundamentals, and repetition.”
- Metrics can be and often are useful, but the thing to avoid is Metrics Fixation, which is where you replace judgement with numeric indicators, you think making metrics public will solve everything through motivation, and thinking that the best way to motivate people is by giving them money or ordinal rankings.
I’m not convinced this is true in theory, but it’s definitely true in practical terms given current society and technology.
- Not everything that matters is measurable
- Not everything that’s measurable matters
Common flaws
The most characteristic feature of metric fixation is the aspiration to replace judgment based on experience with standardized measurement.
Muller, Jerry Z.. The Tyranny of Metrics (p. 6). Princeton University Press. Kindle Edition.
- A common flaw is measuring what’s easiest
- You don’t want to measure the simple, when the outcome you want is complex
- Measuring inputs rather than outputs
- Gaming through creaming is when you find simpler targets or only choose inputs where you’re likely to have good metrics
- Lowering standards to have more successes
- Leaving out data or reclassifying incidents as higher or lower categories
- Outright cheating happens too, when the pressures are high enough (high Metrics Fixation)
The demand for measured accountability and transparency waxes as trust wanes.
Muller, Jerry Z.. The Tyranny of Metrics (p. 39). Princeton University Press. Kindle Edition.
Takeaways
- Metrics are a good way to align a team around a goal.
- The more numeric, visible, and reward-tied a metric is, the more likely it is to be gamed and turn toxic to its original purpose.
- When you use a metrics program, be sure to periodically ensure that undesired externalities have not emerged as a result. And be prepared to go digging, since the negative effects could be well-hidden.
- Moderation is key. Use metrics, but don’t let them control you or become a substitute for judgment.
- Remember to constantly revisit the spirit of what you’re trying to attain, and continuously ask yourself whether the tangible things you’re tracking are high-signal proxies for those goals.
You can find my other book summaries here.
—
Become a direct supporter of my content for less than a latte a month ($ 50/year) and get the Unsupervised Learning podcast and newsletter every week instead of just twice a month, plus access to the member portal that includes all member content.
