反事实推理引擎

Overview

CounterfactAgent performs dual-channel counterfactual policy analysis. The quantitative channel implements the Synthetic Control Method (Abadie, Diamond & Hainmueller, 2010) using pure numpy/scipy -- donor pool construction, constrained weight optimization, counterfactual prediction, and placebo testing. The qualitative channel leverages an LLM to generate structured narrative reasoning about causal mechanisms, with explicit confidence tags (HIGH_CONF, MED_CONF, LOW_CONF, SPECULATIVE). All claims are graded on a 4-tier A/B/C/D evidence quality scale. The result is a unified analysis that is mathematically grounded where data allows and narratively rich where it does not.

Capability Boundaries

What this tool does:

• Frame a counterfactual policy event with an intervention time, treated unit, and outcome variable.
• Build a donor pool from comparable units with pre-intervention covariates and outcome series.
• Fit a synthetic control using scipy.optimize.minimize with non-negativity and sum-to-one constraints on donor weights.
• Estimate the treatment effect as the gap between observed and synthetic outcomes over time.
• Run placebo tests by iterating the synthetic control over each donor unit.
• Generate structured LLM narrative explanations organized by mechanism type:
  • Direct effects
  • Spillover effects
  • Equilibrium adjustments
  • Heterogeneous responses
• Tag every narrative claim with one of 4 confidence levels:
  • HIGH_CONF: Multiple independent sources corroborate; quantitative evidence available.
  • MED_CONF: Supported by theory or partial evidence; reasonable inference.
  • LOW_CONF: Plausible but speculative; limited direct evidence.
  • SPECULATIVE: Theoretically possible; no direct evidence available.
• Grade all evidence claims on the A/B/C/D scale:
  • Grade A: Well-verified, multiple strong sources, rigorous method.
  • Grade B: Reasonable evidence, some corroboration, standard method.
  • Grade C: Weak evidence, single source, unverified method.
  • Grade D: Unsupported, speculative, or contradictory evidence.
• Produce a comprehensive AnalysisResult with both quantitative estimates and qualitative narratives, assembled by the Reporter.

What this tool does NOT do:

• It is NOT a general-purpose causal inference library (no DiD, RDD, IV, etc.).
• It does NOT automatically fetch or construct donor pools from external data sources -- the user must provide or construct the donor pool.
• It does NOT run formal statistical hypothesis tests beyond placebo testing.
• It is NOT designed for real-time or streaming analysis.
• The LLM narrative is probabilistic and should be treated as a reasoning aid, not a substitute for domain expertise.

Trigger Conditions

Use this skill when the user:

• Wants to estimate the causal effect of a policy, event, or intervention.
• Asks "what would have happened if X had not occurred?"
• Needs a counterfactual analysis combining quantitative and qualitative approaches.
• Mentions phrases like "counterfactual analysis", "synthetic control", "causal impact", "policy evaluation", "what if", "反事实分析", or "政策效果评估".
• Provides time-series data and an intervention point for analysis.

Usage

Python API

from counterfact_agent import CounterfactEngine

engine = CounterfactEngine(model="qwen2.5:7b")
result = engine.analyze(
    event_name="California Tobacco Control Program (1988)",
    treated_unit=california_data,
    donor_pool=other_states_data,
    intervention_time=1988,
)

print(f"Estimated effect: {result.effect_estimate}")
print(f"Narrative confidence: {result.narrative.confidence}")
print(f"Evidence grades: {result.evidence.grades}")

Dual-Channel Architecture

1. Quantitative Channel (synthetic_control.py):

   from counterfact_agent.synthetic_control import (
       SyntheticControl, DonorUnit, SyntheticControlResult
   )
   
   sc = SyntheticControl()
   result: SyntheticControlResult = sc.fit(treated, donor_pool)
   # result contains: weights, counterfactual series, gaps, placebo results
   

2. Qualitative Channel (narrative.py):

   from counterfact_agent.narrative import NarrativeGenerator, MechanismNarrative
   
   generator = NarrativeGenerator(llm_client, model_name)
   narrative: MechanismNarrative = generator.generate(event_context, sc_result)
   # narrative contains mechanism-level explanations with confidence tags
   

3. Evidence Grading (evidence.py):

   from counterfact_agent.evidence import EvidenceGrader, EvidenceGrade
   
   grader = EvidenceGrader(llm_client)
   report = grader.grade(narrative_claims)
   # report assigns A/B/C/D grades to each claim
   

Dependencies

• Quantitative: numpy, scipy, pandas
• LLM: OpenAI-compatible endpoint (default: Ollama)
• The Synthetic Control implementation is pure Python/numpy/scipy -- no LLM required for the math.

Output Example

An AnalysisResult contains:

• Event Frame: Intervention time, treated unit, outcome variable, event description.
• Synthetic Control Weights: Which donor units contributed to the synthetic control and their weights.
• Treatment Effect Estimate: Gap between observed and synthetic counterfactual trajectories.
• Placebo Test Results: Treatment effect compared against placebo effects from each donor.
• Mechanism Narratives: Structured explanations with:
  • Mechanism type (direct, spillover, equilibrium, heterogeneous)
  • Narrative text
  • Confidence tag (HIGH_CONF / MED_CONF / LOW_CONF / SPECULATIVE)
  • Supporting evidence citations
• Evidence Grades: Each narrative claim assessed as A, B, C, or D quality.
• Confidence Discussion: Aggregate confidence assessment with limitations and caveats.

FAQ

Q: How much data do I need for the Synthetic Control? A: The method requires a treated unit with pre- and post-intervention outcome data, plus a donor pool of comparable units. More pre-intervention periods and more relevant donor units improve the fit. The implementation handles missing data gracefully but will warn you.

Q: What happens when quantitative data is insufficient? A: When the donor pool is sparse or pre-intervention periods are too few, the engine falls back to qualitative narrative reasoning. The confidence tags make this transparent -- you will see MED_CONF, LOW_CONF, or SPECULATIVE instead of HIGH_CONF.

Q: Is the LLM narrative scientifically valid? A: The narrative is an LLM's structured reasoning about causal mechanisms. It is explicitly tagged with confidence levels and should be treated as a reasoning aid, not as definitive causal proof. Cross-reference with domain expertise.

Q: Can I use this for non-economic counterfactuals? A: Yes. The quantitative channel is domain-agnostic (it only needs time-series data). The qualitative channel can reason about any domain. The evidence grading system is also domain-independent.

Q: How is this different from just asking an LLM "what if"? A: Unlike pure LLM reasoning, CounterfactAgent runs actual mathematical synthetic control where data is available. The LLM is used to explain mechanisms behind the estimated gap, not to hallucinate the gap itself.