The Base-Rate Observatory

Curriculum field sheet — A signal is not a verdict

Independent Sahilverse educational material. Fictional cases; no compulsory personal disclosure.

Opening provocation

A detector catches ninety percent of genuine faults. It also incorrectly flags five percent of good items. Is most of what it flags actually defective? Invite the intuitive answer, then ask what essential number is missing: how common are defects?

The mechanism

In a hypothetical group of one thousand products, users change prevalence, sensitivity and specificity. The program computes expected true positives, false positives, false negatives and true negatives, then shows the proportion of all positive alerts that are genuine faults. The arithmetic expresses Bayes' theorem through counts rather than opaque notation.

A serious disagreement

A positive alert with a modest positive predictive value can still justify action when missing a genuine fault would be disastrous and follow-up checks are affordable. In other contexts, too many false alarms impose serious costs. Decision policies depend on consequences, not on one percentage alone.

The second-order question

If a detector is rolled out from a high-defect factory to a low-defect one, its sensitivity and specificity might remain unchanged while the meaning of a positive alert changes dramatically. What happens when people assume precision transfers unchanged across populations?

Transfer exercise

Consider an invented spam filter or automated moderation tool. Write out a ten-thousand-item confusion matrix using hypothetical rates. Describe the experience of people wrongly flagged and those harmed by missed cases. Propose a confirmation process before a high-stakes decision.

Evidence boundaries

All rates are user-chosen and hypothetical. The population is assumed homogeneous and the rates constant. This room does not represent health screening, diagnosis or any specific commercial tool.

Debrief questions

• What denominator did your first intuition assume?
• How did prevalence change the meaning of “positive”?
• Is predictive value identical to sensitivity?
• How do error costs change a reasonable response?

Learning cycle

Predict → interact → describe the result → state a limitation → revise or retain → transfer to another context. Do not assign a grade for agreement with the lesson.