Survivorship Bias

Overview

Survivorship bias is drawing conclusions from a sample pre-filtered by survival — treating survivor traits as the cause of survival when non-survivors (absent from data by definition) may have had identical traits and still failed.

Canon: Wald (1943) reversed the Navy's bomber-armor recommendation — returning planes showed damage where hits were survivable; the missing planes (shot down) were hit where returning planes showed no damage. Armor the gaps, not the hits.

Composes with bayesian-reasoning (prior = population, not survivors), critical-thinking (what would non-survivors say?), first-principles (population is bedrock), and abductive-reasoning ("winners have trait Y" is one hypothesis; randomness is another).

When to Use

Someone draws lessons from "what successful X did"
Investment returns / fund performance / backtested strategies are cited
A business strategy is justified by pointing to companies that used it
Medical / treatment success rates are reported without dropout data
Career advice comes from what top performers did
Odds of building an AI startup are inferred from the visible AI winners (funded unicorns, "wrapper" success stories) amid the AI-bubble / AI-capex debate

Not when: population data available and filter already corrected; analysis is intentionally about survivors only with no population claim.

Coaching Novices (Adaptive Front Door)

Engine mode: user has a concrete claim and data → run The Process directly.
Coach mode: unfamiliar or no case → guide step by step.

In Coach mode, respond one step at a time. Each [WAIT] is a hard stop — output only that step's question, then stop.

One-liner: "Before believing 'X worked because winners did X,' ask whether the losers did the same X — they're not in your sample to refute it."
Check fit: if sample is explicitly restricted to survivors with no population claim, this lens doesn't apply.
Elicit their real claim and the visible data they have.

[WAIT — do not advance until user responds]

Run The Process one step at a time: what filter produced this sample? what's missing? if missing data looked like the sample, would the conclusion hold?

[WAIT — do not advance until user responds]

Close by naming the selection-corrected conclusion (or marking it unprovable from this data alone).

[WAIT — do not advance until user responds]

The Process

Step 1 — State the claim: What is being concluded, from what sample, from what source?

Step 2 — Identify the survival filter: What process produced this sample? What was the population before the filter? What fraction was removed? What did the filter select for/against?

Step 3 — Construct the non-survivor hypothesis: What did non-survivors likely have? Did they share the trait attributed to success? Would the claim hold if we could see them?

Step 4 — Re-estimate strength: Best case = trait explains survival (non-survivors lacked it). Worst case = trait doesn't explain survival (non-survivors had it too). What evidence distinguishes these?

Step 5 — Correct or mark: Get population data and re-run analysis with selection correction. If unavailable, mark conclusion as conditional on survivor sample.

Output Template

# Survivorship Bias Analysis: <claim>
Claim / sample / source:
Survival filter (what removed non-survivors, population size est., survival rate est.):
Non-survivor hypothesis (what they likely had/lacked, could they have had same trait):
Corrected inference (conclusion, confidence, what data would settle it):

→ Method in Action: Abraham Wald and the Statistical Research Group, 1943 · Mutual Fund Survivorship and Reported Returns, 1971–1996 → 2026 lens: AI-startup survivorship — funded unicorns vs the dead-wrapper graveyard (2023–2026)

Pack: Common Survivor Patterns

| Domain | Survivor sample | Missing non-survivor data | Biased claim | |---|---|---|---| | Business / startup | Surviving companies | Failed companies | "Successful companies do X" | | Investment returns | Active funds / listed stocks | Closed funds / delisted stocks | "Stocks return 10% annually" | | Career advice | Top performers | People who left the field | "To succeed, do X" | | Scientific findings | Published studies | Unpublished null results | "X is significant" | | Treatment efficacy | Patients who completed | Drop-outs, deaths during treatment | "X% recovered" |

→ Primary sources: references/sources.md

Common Rationalizations

[D] = designed upfront | [O] = observed in real use. [O] entries are more valuable.

| Fake move | Reality | |---|---| | [D] "Look at the data" (survivor sample) | Survivor data ≠ population data. Correct or mark as conditional. | | [D] Citing one famous example as proof | N=1 in survivor sample tells you nothing about the rate. | | [D] "X is the formula for success" | If failures did the same X, X is not the formula. Get non-survivor data or stop claiming. | | [D] "We use a backtested strategy" | If backtest excludes failed/delisted stocks, results are upward-biased. | | [D] "If we had non-survivor data, we'd see the same pattern" | Unfalsifiable without the data. Get it or hold the claim. | | → Add [O] entries here after each real use — paste the actual failure pattern | What went wrong and why |

Red Flags

Sample described as "successful X" or "the X who made it"
Data source is survivor-filtered (active funds, surviving companies, published studies)
Base rate of failure / dropout not stated
Conclusions about a population drawn from the survivor subset

Verification

[ ] Survival filter identified
[ ] Non-survivor population size estimated
[ ] Non-survivor hypothesis constructed
[ ] Conclusions conditional on survivor sample, or formally selection-corrected
[ ] Recommendation robust to worst-case non-survivor hypothesis

Part of deciqAI Knowledge Skills — 227 open-source thinking skills that make rigor executable for AI agents. The same skills power every deciqAI agent, which runs them autonomously to operate your company. See it run → https://www.deciqai.com/c/survivorship-bias · ⭐ Star the repo → https://github.com/deciqAI/knowledge-skills · Contributions welcome.

Agents: latest version & machine-readable metadata → https://www.deciqai.com/s/survivorship-bias.json