shield_lock
金大哥
← 返回 Skill 列表
extension
分类: 其它无需 API Key

research-harness

AI原生实验认知规范：LLM代理设定对照实验、设计可复现评估时触发。

person作者: zhelunsunhubclawhub
open_in_new仓库download下载资源包

research-harness

Version: 1.0.0 Cognitive discipline for AI-native scientific experimentation — guardrails, not recipes.

When to Use

Trigger this skill when the user:

  • Sets up a new AI-native research experiment repo
  • Designs controlled experiments with LLM agents
  • Needs reproducible evaluation, statistics, and error analysis
  • Wants to structure a research workspace for long-running agent collaboration
  • Wants agent-safe research governance that prevents overclaiming
  • Says anything like: "research harness", "experiment framework", "AI科研", "对照实验", "评分体系", "可复现性", "科研workflow", "agent协作科研", "可控实验", "效应量"

Core Philosophy

This skill does not prescribe what experiment to run. It prescribes how to think while running it.

Research agents fail not because they lack capability, but because they:

  • Scale before validating the minimum loop
  • Overclaim what the data proves
  • Treat surprising results as methodology failures before checking the execution chain
  • Delete failed runs to make progress look cleaner
  • Change baselines or rubrics silently

The antidote is cognitive discipline — a set of non-negotiable mental habits enforced by repo structure, not by prompt reminders. Detailed reasoning for each discipline is in references/scientific-thinking.md.

Five Cognitive Disciplines

| # | Discipline | Core Question | Deep dive | |---|------------|---------------|-----------| | 1 | Minimum Closed Loop Before Scale | Can the smallest version produce distinguishable signals? | references/experiment-design.md | | 2 | Isolated Variables & Attributable Baselines | Does each group add exactly one variable? | references/experiment-design.md | | 3 | Dual-Track Validation | Do two independent scoring systems agree? | references/scoring-statistics.md | | 4 | Effect Size Over Significance | What is the magnitude, not just the p-value? | references/scoring-statistics.md | | 5 | Pipeline Before Interpretation | Was the execution chain verified before the hypothesis was questioned? | references/scientific-thinking.md |

Disciplines 1-2: experiment design. 3-4: scoring & statistics. 5: critical reasoning.

Five Governance Rules

| # | Rule | Principle | |---|------|-----------| | 1 | Human Owns Direction; Agent Owns Execution | Agent cannot change research questions, promote evidence without review, or make academic decisions | | 2 | Evidence Has Status; AI Output Is Not Fact | All AI-generated evidence starts as candidate; only back-to-source verification promotes to verified | | 3 | Failed Runs Are Data, Not Trash | Register every run in the manifest; failures are process evidence against survivorship bias | | 4 | Protected Surfaces Change Only By Proposal | Baselines, rubrics, raw results, and schema require version bump + documented proposal | | 5 | Every Handoff Needs an Alignment Doc | Short doc replaces long chat history for agent onboarding |

Details in references/agent-collaboration.md.

Phase Workflow

Phase 0 · Scaffold

Goal: Set up the three-layer repo and root entry files.

  • thinking-space/ — research direction, claims, decisions (human)
  • execution-layer/ — briefs, logs, results, drafts (agent)
  • code-workshop/ — runnable artifacts, packages

Root files: AGENTS.md (workspace map), PLAN.md (phase panel), WORKFLOW.md (procedure), harness/README.md (governance).

Directory skeleton and rationale: references/repo-architecture.md.

Phase 1 · Harden

Goal: Make the repo self-checking before formal execution.

  1. Module contracts — Each core module gets a CONTRACT.md (purpose, inputs, outputs, invariants, local validator). Template in references/repo-architecture.md.
  2. Local validators — scripts/validate_<module>.py per module; scripts/validate_repo_state.py as aggregator. Gate rule: 0 FAIL before any formal run.
  3. Experiment manifest — experiments/results/manifest.csv as run-level provenance ledger (run_id, wave, task_id, group, model, version metadata, status, retry_of, git_commit).
  4. Protected surfaces — Baselines, rubrics, raw results, scoring config, schema. Require version bump + proposal to change.

Phase 2 · Design

Goal: Design attributable controlled experiments.

  • Progressive building: minimum artifacts → schema validation → small task set → dry run → scoring → expand. Design details in references/experiment-design.md.
  • Controlled groups: Baseline → incremental treatments. Adjacent groups differ by exactly one variable.
  • Gold checklists: Every task has must_include, forbidden, and scoring_notes.
  • Output contract: Agent output follows a strict schema (YAML/JSON). The scorer and analysis pipeline depend on this contract.

Phase 3 · Execute & Analyze

Goal: Run experiments, score, compute statistics, analyze errors.

Preflight gate: local validators must pass. Then:

  1. Dry run — print prompt, no API call
  2. Smoke run — 1 task × 2 groups, verify output parsing
  3. Wave 1 — small set × all groups, minimum viable data
  4. Scoring: Track A (rule-based) + Track B (semantic) cross-validation. Details in references/scoring-statistics.md.
  5. Statistics: Cohen's d primary, 95% CI, paired t, Wilcoxon. --reproduce flag for one-click reproducibility.
  6. Error analysis: hallucination, output depth, specificity, task appropriateness.

Phase 4 · Handoff & Writing

Goal: Package results for the next phase or agent.

  • Alignment doc: ~1 page with state, entry files, new surfaces, preflight commands, protected surfaces. Never pass chat history.
  • Upstream proposals: Any insight affecting direction goes to sync/upstream_proposals/ first. Template in references/agent-collaboration.md.
  • Writing markers: [REF-MISSING], [CRITICAL-CHECK], [TODO]. Never use AI numbers without verification.

Non-Negotiables

  1. No unverified citation becomes a research fact
  2. No debug result becomes a formal result
  3. No agent changes baseline, rubric, or metric definitions without a proposal
  4. No raw result is overwritten
  5. No failed experiment is deleted
  6. No phase gate passes before validators report zero FAIL

References

  • references/repo-architecture.md — three-layer repo, module contracts, manifest, validators
  • references/experiment-design.md — progressive building, controlled groups, gold checklists
  • references/scoring-statistics.md — dual-track validation, effect size, reproducibility
  • references/scientific-thinking.md — cognitive disciplines for agent-led research
  • references/agent-collaboration.md — governance, evidence status, alignment docs