Causal Design

Design and audit identification strategies for observational causal inference.

Output Path

Per rules/review-artefact-routing.md (auto-loads in research projects (path-scoped to paper-*/ and paper/)):

Source slug: causal-design
Write reports to: reviews/<scope>/causal-design/<YYYY-MM-DD-HHMM>.md inside the project, where <scope> is the paper slug (e.g. paper-philtech) for paper-level audits or _project for project-level reviews. Path is relative to the research project root, not the Task-Management repo.
Never at project root (./CRITIC-REPORT.md-style filenames are forbidden — pre-rule layout).
Idempotency: if today's timestamp exists, append a same-day descriptor to the path base ({date}-HHMM-revision.md, {date}-HHMM-r2.md, {date}-HHMM-pre-submission.md) — never overwrite.
Index update: if reviews/INDEX.md exists, write a one-line entry under "Latest per source" pointing at the new file. Otherwise review-recap will rebuild the index next time it runs.
Infrastructure repos (Task-Management, atlas-workspace, etc.): this section does not apply — the path-scoped rule won't load there.

Modes

| Mode | What it does | Entry point | |------|-------------|-------------| | Design | Interview-driven strategy selection and memo production | "Design my causal strategy" / "What identification can I use?" | | Audit | 4-phase causal inference check on existing paper/scripts | "Check my identification" / "Audit my econometrics" |

Default: Design. If the user points to an existing paper or estimation script, auto-select Audit mode.

When to Use

Choosing an identification strategy for an observational study
Stress-testing whether an existing strategy is credible
Verifying that code implements the claimed identification design
Mapping causal claims to their identifying assumptions

When NOT to Use

Experimental design (RCTs, surveys, factorial) -- use experiment-design
Running the analysis or generating results -- use data-analysis
Literature search or citation gathering -- use an installed literature workflow or scholarly search
Proofreading or compiling the paper -- use proofread, latex

Shared References

Method probing questions: shared/method-probing-questions.md — ask before running any analysis (DiD, IV, RDD sections)
Validation tiers: shared/validation-tiers.md — declare tier before designing strategy
Escalation protocol: shared/escalation-protocol.md — escalate when identification is vague or unsound

Mode: Design

Phase 1: Interview

Before opening the interview, confirm the project's validation tier per shared/validation-tiers.md — Exploratory designs warrant lighter identification stress-testing than Publication-ready ones. Use shared/method-probing-questions.md as the interview backbone; the prompts below adapt those probes to causal identification specifically.

Conduct a structured interview to understand the research setting. Ask these questions (adapt to what the user has already shared):

Causal question: What causal effect are you trying to estimate? What is the treatment? What is the outcome?
Variation: What source of variation in treatment do you exploit? Is it natural, policy-driven, institutional?
Confounders: What are the main threats to identification? What unobservables worry you?
Data structure: Panel, cross-section, or repeated cross-section? What units and time periods?
Institutional context: Any thresholds, cutoffs, rollout dates, or instruments available?
Prior literature: What identification strategies have others used for similar questions?

Do not proceed until the causal question and data structure are clear.

Phase 2: Strategy Selection

Read references/design-decision-tree.md and walk through the decision tree with the user's answers:

Match the research setting to the strongest available strategy
If multiple strategies are viable, rank them by credibility and discuss trade-offs
If the setting does not support any strong strategy, say so explicitly -- do not force a weak design

Phase 3: Strategy Memo

Write a strategy memo using references/strategy-memo-template.md. Save to docs/causal-strategy.md (or project-appropriate location).

The memo must specify:

Estimand -- the exact causal parameter being estimated, in formal notation
Identification strategy -- how variation is generated and why it is exogenous
Key assumptions -- each one stated, with a defence or test plan
Threats and mitigations -- what could go wrong and how to address it
Diagnostics plan -- which tests to run before trusting the estimates
Robustness checks -- pre-committed alternative specifications
Alternative strategies considered -- why they were rejected

This memo is what data-analysis Phase 3 checks for before allowing estimation. It locks the research design per the design-before-results rule.

Phase 4: Adversarial Review

The reviewer follows shared/escalation-protocol.md — when identification is vague or assumptions are hand-waved, the reviewer escalates rather than accommodating.

Delegate an adversarial review to the domain-reviewer agent. Read references/causal-audit-prompt.md and pass it as the prompt to the fresh-context sub-agent mechanism:

Launch the domain-reviewer agent with this prompt:
"You are reviewing a causal identification strategy memo. [Insert contents of causal-audit-prompt.md, customised with the specific strategy chosen]. The memo is at [path]. Focus exclusively on identification credibility."

The agent will produce a report at reviews/<scope>/domain-reviewer/<YYYY-MM-DD-HHMM>.md in the project, where <scope> is the paper slug or _project.

Phase 5: Iterate

Present the domain-reviewer's findings to the user. For each issue flagged:

Discuss whether it is a genuine threat or can be addressed
Update the strategy memo if the design changes
If the strategy is fundamentally flawed, return to Phase 2

Mode: Audit

Phase 1: Extract Claims

Read the paper (.tex files) and/or estimation scripts to extract every causal claim:

What effects does the paper claim to estimate?
What language is used? ("causal", "effect of", "impact of", "leads to")
Are claims hedged appropriately or overstated?

Produce a numbered list of claims with their locations (file:line).

Phase 2: Map Estimands to Identification

For each causal claim, determine:

| Claim | Estimand | Strategy | Key Assumption | Stated? | Defended? | |-------|----------|----------|----------------|---------|-----------| | ... | ... | ... | ... | Yes/No | Yes/No |

Flag any claim where:

The estimand is undefined or vague
The identification strategy is not stated
Key assumptions are not listed or defended
The strategy does not match the claim (e.g., claiming ATE but estimating LATE)

Phase 3: Assumption Diagnostics

For each identification strategy found, check whether the required diagnostics are present and passing:

DiD / Event Study:

Pre-treatment parallel trends test (visual + formal)
Staggered treatment handling (TWFE bias check, Callaway-Sant'Anna or Sun-Abraham if staggered)
Anticipation effects check
Treatment effect heterogeneity assessment

IV:

First-stage F-statistic reported (> 10 for Stock-Yogo, > 104.7 for modern thresholds)
Exclusion restriction argument (quality of narrative)
Monotonicity discussion
Over-identification test (if multiple instruments)
Reduced form reported

RDD:

McCrary density test (no bunching at cutoff)
Bandwidth sensitivity (MSE-optimal + alternatives)
Covariate balance at the cutoff
Donut hole specification
Placebo cutoffs

Synthetic Control:

Pre-treatment fit quality (RMSPE)
Donor pool selection justification
Placebo tests (in-space, in-time)
Leave-one-out robustness

Event Study:

Pre-event coefficients jointly zero
Dynamic treatment effects plotted
Clean control group definition
Anticipation effects addressed

Phase 4: Code-Design Alignment

If estimation code exists, verify it implements the claimed design:

Does the regression specification match the paper's equations?
Are standard errors computed correctly for the design? (clustered at the right level, heteroskedasticity-robust)
Are the treatment and control groups defined as claimed?
Are the diagnostics actually run, not just mentioned?
Do robustness checks exist in code, or only in the text?

Audit Report

Produce an audit report at reviews/<scope>/causal-design/<YYYY-MM-DD-HHMM>.md (where <scope> is the paper slug or _project) with:

# Causal Audit Report

**Document:** [filename]
**Date:** YYYY-MM-DD
**Mode:** Audit

## Claims Inventory

[Numbered list of causal claims with locations]

## Estimand-Identification Map

[Table from Phase 2]

## Diagnostics Assessment

| Strategy | Diagnostic | Present? | Passing? | Notes |
|----------|-----------|----------|----------|-------|
| ... | ... | ... | ... | ... |

## Code-Design Alignment

[Phase 4 findings, or "N/A -- no code found"]

## Critical Issues

[List of issues that threaten identification credibility]

## Recommendations

[Ordered list of fixes, from most to least important]

Cross-References

| Resource | When read | |----------|-----------| | references/design-decision-tree.md | Design Phase 2 (strategy selection) | | references/strategy-memo-template.md | Design Phase 3 (memo output) | | references/causal-audit-prompt.md | Design Phase 4 (agent delegation prompt) | | design-before-results rule | Both modes enforce this | | domain-reviewer agent | Design Phase 4 (adversarial review) | | data-analysis skill | Consumes the strategy memo | | experiment-design skill | For experimental (not observational) designs | | experiment-design/references/identification-strategies.md | Quick-reference for strategies (shared knowledge) |