WatchNo. 03
Why does chance fool us?
You bet on a pattern of three flips, I pick mine after you, and with one fair coin I win at least two games in three: whatever you pick, there is a pattern that beats it. A positive result from a 99% accurate test for a rare illness is right only about 1 time in 11, and a kidney-stone treatment can do better for small stones and for large ones yet worse overall: each time, we counted the wrong thing.
Shorts from this episode
About a minute each
Pick any three coin flips, I still win
On YouTube: Pick any three coin flips, I still winThe 99% test that is usually wrong
On YouTube: The 99% test that is usually wrongWins for small stones, wins for large, loses overall?
On YouTube: Wins for small stones, wins for large, loses overall?
Explore it yourself
The explorations this episode covers
Notes and references
Penney’s game
Each player names a pattern of three flips, and one fair coin is flipped until one of the two patterns turns up; that player wins. Walter Penney published the game in the Journal of Recreational Mathematics in 1969, and Martin Gardner described it in Scientific American in October 1974; the exploration records both as not yet checked against the journals. The second player can always answer well: take the opposite of the first player’s middle call, then their first two calls. Against HHT that reply is THH, which wins exactly 3 games in 4, because HHT can come first only if the game opens with two heads (probability 1/4). The best reply wins 7 in 8 against HHH or TTT, 3 in 4 against HHT or TTH, and 2 in 3 against the other four patterns; a second player who always replies this way wins 71 games in 96 (about 74%) against a first player who picks at random. No pattern is best: THH beats HHT, HHT beats HTT, HTT beats TTH and TTH beats THH. Played alone, HHT and THH each take 8 flips on average to appear, so a lopsided race does not come from unequal waits. Every figure is recomputed on the Penney exploration; Guibas and Odlyzko proved the general overlap formulas in 1981.
The 99% test, made up for illustration
The test in the episode is invented to show the arithmetic; it is not a real test. Suppose 1 person in 1,000 has the illness, the test catches 99% of people who have it and clears 99% of people who do not. Of 100,000 people, 100 are ill and 99 of them test positive; of the 99,900 who are well, 999 also test positive. So 99 of 1,098 positives are ill, which is 11 in 122: about 9%, or 1 in 11. If a second test made independent errors and also came back positive, the chance of illness would be about 91%. The point is that the rarity of the illness, the base rate, matters as much as the accuracy of the test.
Real screening: counting people, not percentages
The screening figures in the episode come from the studies cited below, not from the invented test. Gigerenzer and colleagues describe 160 gynaecologists who were given breast-screening figures as conditional probabilities and asked what a positive mammogram means: the most frequent answer was 90%, and the correct value is about 10%. When the same information was given as counts of people, the most frequent answer was correct. Their worked example is 1,000 women, 10 with breast cancer, of whom 8 test positive, and about 95 of the 990 without it also test positive; that is about 103 positives, 8 with cancer, roughly 1 in 10. Hubbard and colleagues estimate that of women screened every year for ten years from their forties, 61% receive at least one false-positive recall (42% if screened every other year). A false positive is a woman without cancer who is recalled for more tests; a positive screen leads to further imaging and, if needed, a biopsy, not straight to treatment. Overdiagnosis is something else, a real but harmless cancer that gets treated. The Cochrane review by Gøtzsche and Jørgensen estimates that, of 2,000 women screened for ten years, about 10 healthy women are treated for a cancer that would never have harmed them. Those are the review’s own modelled estimates and they are contested, so they are attributed here and not stated as settled.
Kidney stones
Charig and colleagues compared open surgery with keyhole surgery (percutaneous nephrolithotomy) for kidney stones. For stones under 2 cm, open surgery succeeded in 81 of 87 cases (93.1%) and keyhole in 234 of 270 (86.7%). For stones of 2 cm or more, open surgery succeeded in 192 of 263 (73.0%) and keyhole in 55 of 80 (68.8%). Overall, open surgery succeeded in 273 of 350 (78.0%) and keyhole in 289 of 350 (82.6%). Open surgery is ahead in both groups and behind overall, because patients were not assigned at random: 263 of the 350 open-surgery patients had large stones, which are harder to treat, while 270 of the 350 keyhole patients had small ones. Compared like with like, open surgery did better in this study, which was an observation of patients and not a trial, so it does not show that open surgery is better. Julious and Mullee pointed out the paradox in 1994. The counts here agree across the accounts we checked; the 1986 paper itself was not inspected.
Counting the wrong thing
In each case chance did not cheat; we looked at one thing and missed what it left out. Judging a pattern on its own ignores which pattern it is racing, and the wait alone does not decide the race. Judging a test by its accuracy ignores how rare the illness is. Judging a treatment by its overall score ignores who ended up in which group. The dare at the end is to play twelve games of Penney’s game against a friend with the reply above, which should win about eight or nine.
Not medical advice
This page and the episode explain how to read numbers. They are not medical advice, and the figures describe groups of people, not any one person. Whether to be screened, and what a result means for you, is a question for your own doctor.
How the site classes a statement as proved, checked or open is set out in the method.
References
- L. J. Guibas, A. M. Odlyzko, String overlaps, pattern matching, and nontransitive games, Journal of Combinatorial Theory, Series A 30(2), pp. 183–208, 1981. Link to String overlaps, pattern matching, and nontransitive games
- G. Gigerenzer, W. Gaissmaier, E. Kurz-Milcke, L. M. Schwartz, S. Woloshin, Helping doctors and patients make sense of health statistics, Psychological Science in the Public Interest 8(2), pp. 53–96, 2007. Link to Helping doctors and patients make sense of health statistics
- R. A. Hubbard, K. Kerlikowske, C. I. Flowers, B. C. Yankaskas, W. Zhu, D. L. Miglioretti, Cumulative probability of false-positive recall or biopsy recommendation after 10 years of screening mammography, Annals of Internal Medicine 155(8), pp. 481–492, 2011. Link to Cumulative probability of false-positive recall or biopsy recommendation after 10 years of screening mammography
- P. C. Gøtzsche, K. J. Jørgensen, Screening for breast cancer with mammography, Cochrane Database of Systematic Reviews 2013, issue 6, CD001877, 2013. Link to Screening for breast cancer with mammography
- C. R. Charig, D. R. Webb, S. R. Payne, J. E. A. Wickham, Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy, BMJ 292, pp. 879–882, 1986. Link to Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy
- S. A. Julious, M. A. Mullee, Confounding and Simpson’s paradox, BMJ 309, pp. 1480–1481, 1994. Link to Confounding and Simpson’s paradox