Benchmark And MMS Planner

Goal

Design a verification and validation plan before trusting simulation results. The skill helps agents choose manufactured solutions, benchmark cases, refinement protocols, uncertainty checks, and pass/fail criteria.

Requirements

• Python 3.10+
• No external dependencies
• Works on Linux, macOS, and Windows

Inputs to Gather

| Input | Description | Example | |-------|-------------|---------| | PDE or model class | Governing family | diffusion, elasticity, phase-field | | Quantity of interest | Metric to validate | interface velocity, L2 temperature error | | Dimension | 1, 2, or 3 | 2 | | Expected order | Formal discretization order | 2 | | Reference availability | Analytic, benchmark, or none | analytic | | Risk level | Cost or consequence of wrong result | high |

Decision Guidance

• Use MMS when code correctness is uncertain and an analytic solution can be injected.
• Use canonical benchmarks when physical model validation matters more than code verification.
• Use grid/time refinement whenever the result is used for a claim, design decision, or comparison.
• Use uncertainty propagation when inputs are calibrated, noisy, or experimentally measured.

Script Outputs

scripts/benchmark_mms_planner.py emits inputs and results with:

• verification_strategy
• effective_model — the resolved model family actually used; unknown families fall back to general.
• mms_plan
• benchmark_cases
• refinement_protocol (dimension, levels, spacing_ratio, expected_order, accept_observed_order_min, include_time_refinement)
• uncertainty_plan (propagate_inputs, report_error_bars, separate_discretization_and_model_error) — propagation/error-bar guidance driven by risk level and reference type.
• acceptance_criteria
• warnings

The accept_observed_order_min is an engineering screening heuristic, not a certified bound: it is the formal expected_order reduced by a fractional tolerance (10% for high risk, 20% otherwise) and floored at first-order convergence (1.0). The relative band keeps strictness consistent across formal orders. See references/vv_patterns.md.

Workflow

1. Collect the governing model, quantity of interest, and risk level.
2. Run benchmark_mms_planner.py --json.
3. Treat warnings as blockers for high-risk claims.
4. Convert the returned protocol into tests, simulation runs, or review checklist items.

python3 skills/verification-validation/benchmark-and-mms-planner/scripts/benchmark_mms_planner.py \
  --model diffusion \
  --quantity "L2 error in temperature" \
  --dimension 2 \
  --expected-order 2 \
  --reference analytic \
  --risk high \
  --json

Error Handling

• If the dimension or expected order is invalid, stop and correct the model description.
• If no reference exists, use conservation and convergence checks but do not call the result validated.

Limitations

This skill plans verification work; it does not run the solver or prove that a physical model is appropriate for an experiment.

Verification checklist

Before trusting a result that used this planner, record concrete evidence for each item:

• [ ] Ran benchmark_mms_planner.py --json and saved the inputs block, confirming the echoed dimension, expected_order, reference, and risk match the actual run (a fallback to effective_model: general was intentional, not a typo in --model).
• [ ] Executed the refinement_protocol: used the reported levels (3, or 4 for high risk) of systematically refined grids at spacing_ratio 2, and recorded the observed order of accuracy from those runs.
• [ ] Confirmed the observed order is at or above accept_observed_order_min; if below, logged the investigation (mesh not yet asymptotic, boundary/source errors, limiter activation) rather than treating the result as passed.
• [ ] When include_time_refinement is true, ran a separate time-step refinement study and recorded the temporal observed order, not just the spatial one.
• [ ] When mms_plan.manufacture_solution is true, derived the symbolic source/forcing term, applied the matching boundary terms, and recorded the L2 and Linf error norms versus the manufactured solution.
• [ ] Checked every acceptance_criteria item with a number: conservation/balance closes within a documented tolerance, the quantity of interest plateaus under refinement, and any benchmark discrepancy from benchmark_cases is explained before production use.
• [ ] Treated all entries in warnings as blockers for high-risk claims and recorded how each was resolved (e.g. an independent analytic/published reference was added when reference was none or experimental).

Common pitfalls & rationalizations

| Tempting shortcut | Why it's wrong / what to do | |-------------------|-----------------------------| | "The planner ran and printed a plan, so the result is verified." | The script only plans V&V; it never runs the solver. Verification comes from executing the refinement_protocol, MMS, and acceptance_criteria, not from generating the plan. | | "Two grids converged, so the observed order is fine." | refinement_protocol.levels is 3 (4 for high risk) for a reason: you need >=3 systematically refined grids to estimate observed order and confirm the solution is in the asymptotic range before quoting it. | | "Observed order beats accept_observed_order_min, so it's certified." | That threshold is an engineering screening heuristic (formal order minus a 10%/20% relative tolerance, floored at 1.0), not a certified bound. For rigorous order verification run a Richardson/GCI study. | | "Steady-looking model, so I can skip include_time_refinement." | If the planner set include_time_refinement: true (any time-dependent or general fallback family), spatial refinement alone hides temporal error — run the time-step study too. | | "We matched a benchmark, so the code is validated." | Matching benchmark_cases or converging shows the code approaches some solution; it does not prove the physical model is correct. Validation needs an independent reference plus model-error separation, not convergence alone. | | "reference none is fine, the runs look physical." | With reference: none the strategy is verification-only; warnings says so. You may report convergence and conservation but must NOT call the result validated. | | "Unknown model name, so I'll ignore the general fallback." | An unrecognized --model silently resolves to effective_model: general; confirm that fallback is intended, since it changes both benchmark_cases and the time-refinement decision. |

Security

Input Validation

All inputs are command-line arguments parsed by argparse; validation happens in plan_vv (and partly in the parser). Any rejected input causes the script to print the error to stderr and exit with code 2.

• dimension must be exactly 1, 2, or 3; any other integer is rejected.
• expected_order must be a positive, finite number (NaN, infinity, zero, and negatives are rejected).
• risk must be one of the allowlist low, medium, high (enforced both as an argparse choice and re-checked in plan_vv).
• reference must be one of the allowlist analytic, benchmark, experimental, none (enforced both as an argparse choice and re-checked in plan_vv).
• model and quantity are capped at 256 characters (MAX_FIELD_LEN); longer strings are rejected.
• The string content of model and quantity is otherwise not allowlisted or sanitized: quantity is echoed verbatim into the output, and an unrecognized model family is silently resolved to general rather than rejected.

File Access

• The script reads and writes no files; all I/O is command-line args -> stdout JSON (or a short plain-text summary), with errors on stderr.
• Because no filesystem paths are accepted or constructed, there is no path-traversal surface and no path-sandboxing logic is needed.
• The only DoS-relevant size limit is the 256-character cap on model and quantity; numeric outputs are bounded by the validated inputs.

Tool Restrictions

The frontmatter declares allowed-tools: Read, Bash, Write, Grep, Glob.

• Bash is used solely to run the bundled scripts/benchmark_mms_planner.py (e.g. the python3 ... --json invocation in the Workflow).
• Read, Grep, and Glob are for inspecting the skill's own files and references (e.g. references/vv_patterns.md) when planning.
• Write supports turning the returned protocol into test stubs or checklist files; the planner script itself never writes.

Safety Measures

• No eval, exec, or dynamic code execution; the planner is pure Python computing a dictionary.
• The script spawns no subprocesses and invokes no external solvers, so there are no subprocess argument lists to escape.
• No pickle or other deserialization of untrusted data is performed; output is serialized with json.dumps.
• The 256-character field cap is the explicit DoS guard against pathological input strings.

References

• See references/vv_patterns.md for MMS, benchmark, and uncertainty planning notes.

Version History

• 1.1.3: Add a "Verification checklist" (evidence-based items tied to the planner's refinement_protocol, mms_plan, acceptance_criteria, and warnings) and a "Common pitfalls & rationalizations" table that pins down domain-specific V&V shortcuts (plan != verification, >=3 grids for observed order, screening band is not a certified bound, time refinement, convergence != validation, reference none, general fallback).
• 1.1.1: Make the eval suite discriminating by adding deterministic script_checks that pin the planner's specific output (resolved verification_strategy, the relative accept_observed_order_min band, refinement levels, include_time_refinement, uncertainty_plan flags, model-specific benchmark cases, and the exact warning strings) for each of the three cases.
• 1.1.0: Resolve unknown model families to general once so benchmark selection and the time-refinement decision agree (transient unlisted PDEs no longer skip time refinement); echo the resolved family as effective_model. Replace the fixed absolute observed-order offset with a relative tolerance floored at first order. Document uncertainty_plan, effective_model, and the acceptance heuristic. Add 256-character caps on string inputs.
• 1.0.0: Initial benchmark and MMS planning skill.