Parameter Optimization

Goal

Provide a workflow to design experiments, rank parameter influence, and select optimization strategies for materials simulation calibration.

Requirements

Python 3.10+
No external dependencies (uses Python standard library only)

Inputs to Gather

Before running any scripts, collect from the user:

| Input | Description | Example | |-------|-------------|---------| | Parameter bounds | Min/max for each parameter with units | kappa: [0.1, 10.0] W/mK | | Evaluation budget | Max number of simulations allowed | 50 runs | | Noise level | Stochasticity of simulation outputs | low, medium, high | | Constraints | Feasibility rules or forbidden regions | kappa + mobility < 5 |

Decision Guidance

Choosing a DOE Method

Is dimension <= 3 AND full coverage needed?
├── YES → Use factorial
└── NO → Is sensitivity analysis the goal?
    ├── YES → Use quasi-random (preferred; "sobol" is accepted but deprecated)
    └── NO → Use lhs (Latin Hypercube)

| Method | Best For | Avoid When | |--------|----------|------------| | lhs | General exploration, moderate dimensions (3-20) | Need exact grid coverage | | quasi-random | Sensitivity analysis, uniform coverage (preferred) | Very high dimensions (>20) | | sobol | Deprecated alias of quasi-random (emits a warning) | New code (use quasi-random) | | factorial | Low dimension (<4), need all corners | High dimension (exponential growth) |

Factorial sizing: the factorial grid is levels evenly spaced values per parameter, producing exactly levels ** params samples. Set the resolution explicitly with --levels (e.g. --params 2 --levels 4 -> 16 samples). If you use --budget instead, the script back-computes levels = round(budget ** (1/params)) and warns whenever the realized sample count differs from the requested budget (e.g. --budget 20 --params 2 realizes 16 samples). For an exact design, pass a perfect power (--budget 16) or, preferably, --levels.

Choosing an Optimizer

Is dimension <= 10 AND budget <= 100?
├── YES → Bayesian Optimization
└── NO → Is dimension <= 20?
    ├── YES → CMA-ES
    └── NO → Random Search with screening

| Noise Level | Recommendation | |-------------|----------------| | Low | Gradient-based if derivatives available, else Bayesian Optimization | | Medium | Bayesian Optimization with noise model | | High | Evolutionary algorithms or robust Bayesian Optimization |

Script Outputs (JSON Fields)

| Script | Output Fields | |--------|---------------| | scripts/doe_generator.py | samples, method, coverage (count, dimension; plus levels and a top-level requested_budget/note for factorial) | | scripts/optimizer_selector.py | recommended, expected_evals, notes | | scripts/sensitivity_summary.py | ranking, notes | | scripts/surrogate_builder.py | model_type, metrics (mse, cv_error, output_variance), notes |

Workflow

Generate DOE with scripts/doe_generator.py
Run simulations at DOE sample points (user's responsibility)
Summarize sensitivity with scripts/sensitivity_summary.py
Choose optimizer using scripts/optimizer_selector.py
(Optional) Fit surrogate with scripts/surrogate_builder.py

CLI Examples

# Generate 20 LHS samples for 3 parameters
python3 scripts/doe_generator.py --params 3 --budget 20 --method lhs --json

# Full factorial with 4 levels per parameter (2 params -> 16 samples)
python3 scripts/doe_generator.py --params 2 --levels 4 --method factorial --json

# Rank parameters by sensitivity scores
python3 scripts/sensitivity_summary.py --scores 0.2,0.5,0.3 --names kappa,mobility,W --json

# Get optimizer recommendation for 3D problem with 50 eval budget
python3 scripts/optimizer_selector.py --dim 3 --budget 50 --noise low --json

# Build surrogate model from simulation data
python3 scripts/surrogate_builder.py --x 0,1,2 --y 10,12,15 --model rbf --json

Conversational Workflow Example

User: I need to calibrate thermal conductivity and diffusivity for my FEM simulation. I can run about 30 simulations.

Agent workflow:

Identify 2 parameters → --params 2
Budget is 30 → --budget 30

Use LHS for general exploration:

python3 scripts/doe_generator.py --params 2 --budget 30 --method lhs --json

After user runs simulations and provides outputs, summarize sensitivity:

python3 scripts/sensitivity_summary.py --scores 0.7,0.3 --names conductivity,diffusivity --json

Recommend optimizer:

python3 scripts/optimizer_selector.py --dim 2 --budget 30 --noise low --json

Error Handling

| Error | Cause | Resolution | |-------|-------|------------| | params must be positive | Zero or negative dimension | Ask user for valid parameter count | | budget must be positive | Zero or negative budget | Ask user for realistic simulation budget | | argument --method: invalid choice: <value> (choose from lhs, sobol, quasi-random, factorial) | Invalid method (argparse) | Use decision guidance to pick a valid method | | could not convert string to float: <token> | Non-numeric value in --scores/--x/--y | Reformat as 0.1,0.2,0.3 | | scores must be a comma-separated list | Empty --scores input | Provide at least one numeric score |

Verification checklist

[ ] Recorded the exact doe_generator.py coverage.count and confirmed it matches the intended design — for factorial, verified count == levels ** params and that no note/requested_budget mismatch warning was emitted (or that the realized count is acceptable).
[ ] Confirmed the chosen --method matches the Decision Guidance for the actual dimension/goal, and that quasi-random was used instead of the deprecated sobol alias (no DeprecationWarning in output).
[ ] Recorded the optimizer_selector.py recommended strategy and expected_evals, and verified expected_evals <= budget so the plan is feasible within the stated evaluation budget.
[ ] Logged the sensitivity_summary.py ranking and checked whether the top sensitivity is < 0.1 (the "All sensitivities are low" note); if so, did not over-interpret the ranking and revisited the output metric.
[ ] For surrogate fits, judged quality with metrics.cv_error (leave-one-out), NOT in-sample mse — especially for rbf, where mse is near zero by construction — and compared cv_error against metrics.output_variance to confirm the surrogate beats the constant-mean baseline.
[ ] Confirmed any reported cv_error is a finite number (not NaN), i.e. there were enough samples for leave-one-out (poly: n > degree+1; rbf: n >= 3).

Common pitfalls & rationalizations

| Tempting shortcut | Why it's wrong / what to do | |-------------------|------------------------------| | "RBF surrogate mse is ~0, so the model is excellent." | RBF is an exact interpolant — in-sample mse is near zero by construction and says nothing about generalization. Judge fit with metrics.cv_error and compare it to output_variance. | | "I asked for --budget 20 factorial, so I got 20 samples." | Factorial honors levels ** params, not the budget; --budget 20 --params 2 realizes 16 samples and emits a note/warning. Use --levels for an exact, intended design. | | "sobol gives me a true Sobol low-discrepancy sequence." | sobol is a deprecated alias that emits a DeprecationWarning and uses a simplified golden-ratio additive recurrence, not a true Sobol sequence. Use quasi-random; for production Sobol use scipy.stats.qmc. | | "The optimizer recommendation is just advice — budget doesn't matter." | The recommendation is gated on dimension AND budget (BO only for dim<=10 AND budget<=100), and expected_evals is capped at the budget. Record both and confirm the plan fits the real budget. | | "One sensitivity score is highest, so that parameter dominates." | The script only sorts the scores you pass in; it computes no sensitivity itself. If the top score is < 0.1 it flags that all sensitivities are low — get the scores from a real screening/Sobol analysis before trusting the ranking. | | "It printed JSON without erroring, so the result is valid." | Exit success only means inputs parsed. Verify the design size, expected_evals <= budget, a finite cv_error, and that the surrogate beats output_variance before trusting any output. |

Security

Input Validation

sensitivity_summary.py validates --names against [a-zA-Z_][a-zA-Z0-9_ .-]* with a 200-char limit, preventing shell metacharacter injection via crafted parameter names
All numeric list inputs are validated as finite numbers (NaN/Inf rejected)
Comma-separated value lists are capped (10,000 for scores, 100,000 for surrogate data) to prevent resource exhaustion
doe_generator.py caps dimension at 1,000 and budget at 1,000,000; optimizer_selector.py caps dimension at 100,000 and budget at 10,000,000
--method is validated against a fixed allowlist (lhs, quasi-random/sobol, factorial); sobol is an accepted but deprecated alias of quasi-random
--noise is validated against a fixed allowlist (low, medium, high)
--model (surrogate type) is validated against a fixed allowlist (rbf, poly)
--levels (factorial grid resolution) is validated as an integer in [2, 1000]

File Access

Scripts read no external files; all inputs are provided via CLI arguments
Scripts write only to stdout (JSON output); no files are created unless the agent explicitly uses the Write tool

Tool Restrictions

Read: Used to inspect script source, references, and user data files
Write: Used to save DOE sample plans, sensitivity rankings, or optimizer recommendations; writes are scoped to the user's working directory
Grep/Glob: Used to locate relevant files and search references
The skill's allowed-tools excludes Bash to prevent the agent from executing arbitrary commands when processing user-provided parameter names and constraints

Safety Measures

No eval(), exec(), or dynamic code generation
All subprocess calls use explicit argument lists (no shell=True)
Reduced tool surface (no Bash) limits the agent to read/write operations only
Parameter names are sanitized before use, preventing injection via crafted identifiers

Limitations

Not for real-time optimization: Scripts provide recommendations, not live optimization loops
Surrogate is lightweight: surrogate_builder.py fits a real 1-D least-squares polynomial (poly) or Gaussian RBF interpolant (rbf) using only the standard library and reports honest residual mse, leave-one-out cv_error, and the data output_variance; for production use scipy/scikit-learn/GPyTorch. For rbf, in-sample mse is near zero by construction (exact interpolation) — judge fit quality with cv_error
No automatic simulation execution: User must run simulations externally and provide results

References

references/doe_methods.md - Detailed DOE method comparison
references/optimizer_selection.md - Optimizer algorithm details
references/sensitivity_guidelines.md - Sensitivity analysis interpretation
references/surrogate_guidelines.md - Surrogate model selection

Version History

v1.2.2 (2026-06-24): Added Verification checklist and Common pitfalls & rationalizations sections to drive evidence-based use of the DOE, optimizer, sensitivity, and surrogate scripts
v1.2.0 (2026-06-23): Real surrogate fits (poly least-squares, rbf interpolation) with honest mse/cv_error/output_variance; explicit factorial --levels with budget-mismatch warnings; BO dimension cutoff harmonized to dim<=10; corrected Security/Error-Handling/output-field docs to match script behavior
v1.1.0 (2024-12-24): Enhanced documentation, decision guidance, conversational examples
v1.0.0: Initial release with core scripts