Agent DOE Engine

The Problem

Tuning an agent the usual way fails twice. One-at-a-time testing is slow and blind — it misses settings that only matter in combination (a bigger batch size that only helps once you also add workers). And a single run can fool you: every run carries random variation, so a number that looks better might just be noise.

What I Built

Part of the design-of-experiments optimization pattern (see /toolkit/patterns/design-of-experiments) — here: design size is chosen automatically from the count of settings under test (2–3 → full factorial, 4–7 → fractional factorial, 8–11 → Plackett-Burman), and competing agent goals are resolved by whichever of three methods the run calls for — scalarize, desirability, or pareto — rather than a single fixed objective function.

A design-of-experiments engine that varies several settings together in one planned batch, then tells you which changes are real. You list the settings to test and the results you care about — speed, cost, quality, accuracy — and for each result it reports which settings moved it and by how much (including interaction-only effects), whether that movement is a real effect or a fluke, which effects are mathematically tangled in this design, and the single best configuration when goals compete.

Architecture

agent-doe-engine is a deterministic statistics engine, not a model-calling one. The matrix generation, sampled measurement, effects fitting, and multi-objective selection all run on numpy and stdlib with no LLM in the loop; the host coding agent’s LLM (Claude Code, Codex, or whichever plugin host is running the skill) is invoked only at three judgment points around that core — which factors are worth testing, what one atomic change to try next, and whether a kept result looks gamed. There are no vendor API calls inside the plugin itself: the skill instructs the host’s own model.

flowchart LR
  A["worktree.py init<br/>dedicated git worktree"] --> B["suggest_factors.py<br/>scan for factor candidates"]
  B --> C["Host LLM<br/>picks candidates to test"]
  C --> D["validate_factors.py<br/>snapshot/mutate/revert/verify"]
  D --> E{"doe.py detect k"}
  E -->|"k=1"| G["loop.py<br/>autoresearch mode"]
  E -->|"k=2-3"| F1["full factorial<br/>4-8 runs"]
  E -->|"k=4-7"| F2["fractional factorial<br/>8 runs"]
  E -->|"k=8-11"| F3["Plackett-Burman<br/>12-run screening"]
  F1 --> H["doe.py generate<br/>numpy ±1-coded matrix"]
  F2 --> H
  F3 --> H
  H --> I["apply factor values"]
  I --> J["metric_runner.py<br/>sampled measurement + guard"]
  J --> L[("results.jsonl")]
  L -->|"next run"| I
  L --> M["doe.py analyze<br/>OLS effects (doe_stats.py)"]
  M --> N["objectives.py select_best<br/>scalarize / desirability / pareto"]
  G --> G1["optimize-runner agent<br/>hypothesize -> edit -> commit"]
  G1 --> G2["metric_runner.py + loop.py --score"]
  G2 --> G3{"aggregate improved<br/>& guard ok?"}
  G3 -->|"keep"| G1
  G3 -->|"discard"| G4["git revert"] --> G1
  N --> O["overfitting-reviewer agent<br/>read-only, adversarial"]
  G3 -->|"converged / budget exhausted"| O
  O --> P["loop.py --archive"]
  P --> Q["worktree.py cleanup"]

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Factor candidate ranking	Host LLM (whatever coding agent runs the skill)	Phase 0.2, before any run	suggest_factors.py is a deterministic scan; picking which candidates are worth burning a run budget on is judgment, not math
Hypothesis generation (autoresearch)	Host LLM, Sonnet-tier per agents/optimize-runner.md	one call per loop iteration	Fires up to budget times per run, so it’s pinned to a fixed code-tier model rather than a slower thinking tier
Overfitting review	Host LLM, Sonnet-tier per agents/overfitting-reviewer.md	once per run, Phase 3, read-only	Adversarial pattern match over kept diffs (safety removal, metric-gaming, scope violations) needs judgment; the agent has no edit tools
DOE generation, effects fitting, selection, measurement, guard	none — deterministic	scripts/doe.py, doe_stats.py, objectives.py, metric_runner.py	numpy OLS fitting and closed-form scoring math; “the metric is the only judge,” per the skill — no model call decides a winner

Models: none inside the plugin’s own code path — the three LLM touch points above are the host coding agent’s model, called by the skill’s instructions, not by an SDK call in this codebase.

Tools & infra — which, why:

• numpy (>=1.24) — the only runtime dependency. Full factorial enumeration, fractional-factorial generators, and the Plackett-Burman 12-run Paley construction are numpy-only and were verified equivalent to pyDOE3 1.6.2 up to row/column permutation; the same array ops drive the OLS effects fit in doe.py analyze.
• Git worktrees (scripts/worktree.py, stdlib subprocess + argparse, zero deps) — every run gets its own <repo>-agent-doe-engine-<slug> worktree on branch agent-doe-engine/<slug> so the apply/measure/revert cycle across many DOE runs never dirties the user’s primary checkout.
• Filesystem JSON/JSONL, no database — state lives under .agent-doe-engine/optimize/ in the consumer project: factors.json, doe.json, results.jsonl, effects.json, objectives.json, plus experiment.json / results.tsv for autoresearch and an experiments/ archive. A legacy .multi-goal/ path from before the project’s rename is migrated automatically on first use.
• subprocess shell execution (metric_runner.py) — runs the caller’s own metric/guard commands, extracts a numeric result via a regex cascade (labeled values → percentages → time(1) output → trailing number → any number), and aggregates repeated samples (last/min/max/mean/median/p95) with configurable warmups to smooth noisy metrics.
• pytest (dev only) — the test suite (test_doe.py, test_doe_stats.py, test_objectives.py, test_validate_factors.py, test_worktree.py, test_manifest.py) exercises the design math and the validators.
• Dual plugin manifests — .claude-plugin/plugin.json (Claude Code) and .codex-plugin/plugin.json (Codex) point at the same skill/commands/agents directories, so the deterministic scripts and skill instructions are host-neutral.
• Distribution — npm package @tyroneross/agent-doe-engine (metadata-only package.json; no JS/TS source, ships the Python scripts, skill, commands, and agents as files) alongside a standard Python install via pyproject.toml and uv.

How it works

1. Isolate. worktree.py init creates or reuses a dedicated git worktree (<repo>-agent-doe-engine-<slug> on branch agent-doe-engine/<slug>) so the run never touches the user’s primary checkout.
2. Scan. suggest_factors.py walks the target repo for numeric/config knobs worth tuning, ranking candidates and flagging ones a research step could improve on (--research-levels).
3. Pick. The host LLM reviews the candidate list and selects which factors to test — the one reasoning step before any run spends budget; the user confirms.
4. Validate. validate_factors.py proves each accepted candidate is actually adjustable via a snapshot → mutate → re-read → revert → verify cycle; only adjustable candidates enter factors.json (dead constants, duplicate definitions, and failed mutations are rejected with a reason).
5. Design. doe.py detect <k> routes by factor count — k=1 → autoresearch; k=2–3 → full factorial (4–8 runs); k=4–7 → fractional factorial (8 runs, some effects aliased); k=8–11 → Plackett-Burman (12-run screening) — then doe.py generate builds the ±1-coded matrix with numpy.
6. Run the matrix. For each row: apply the factor values, measure every named objective with metric_runner.py (sampled, aggregated by last/min/max/mean/median/p95), run the guard command (must exit 0), append the result to results.jsonl, then revert — every run starts from the same baseline.
7. Fit effects. doe.py analyze fits an OLS model (intercept + main effects + two-way interactions, via doe_stats.py) per objective, ranks factors by effect size with p-values, confidence intervals, and a low-power warning, and reports which effects are aliased (tangled) in the chosen design.
8. Select. objectives.py select_best scores every run by the configured method — scalarize (weighted min-max-normalized sum), desirability (Derringer-Suich weighted geometric mean; Derringer & Suich, Journal of Quality Technology, 1980), or pareto (non-dominated front, always computed regardless of selection) — and returns best_factors, the concrete winning configuration.
9. Or, for one factor: autoresearch. loop.py drives a greedy loop instead of a matrix: the optimize-runner agent (Sonnet-tier) hypothesizes one atomic change, commits it, measures every objective, scores the aggregate improvement ratio against the recorded baseline (loop.py --score), and keeps the change or git reverts it based on whether the aggregate improved and the guard passed. Convergence: 5 consecutive discards, a regressing trend, or budget exhaustion.
10. Review. Phase 3 dispatches the overfitting-reviewer agent (read-only, Sonnet-tier) against every kept commit, checking for removed safety checks, fragile shortcuts, metric-gaming, scope violations, and Goodhart risk on any kept result driven by an unvalidated objective — reports strong_checkpoint / guidance findings, never a hard blocker.
11. Archive and clean up. The run is archived (loop.py --archive) and the worktree removed (worktree.py cleanup), keeping the branch by default so the user can review, merge, or cherry-pick before it’s deleted.

Real Effect vs Fluke

Run the same setting twice and the result won’t be identical — that spread is the noise. A real effect is a change bigger than the noise; a fluke fits inside it. Every effect gets a p-value and a confidence interval, with a low-power warning when there aren’t enough runs to tell the two apart. The point: you don’t ship a change that was never real.

The Designs It Uses

It picks the smallest design that still answers the question:

Settings	Design	What it does
2–3	Full factorial (4–8 runs)	Tests every combination — most accurate, nothing tangled
4–7	Fractional factorial (8 runs)	A carefully chosen subset; reports which effects are tangled
8–11	Plackett-Burman (12 runs)	Screening — finds the few settings that matter out of many

For a single setting it falls back to a “try a change, measure, keep it if better” loop — cheaper to set up, but blind to interactions.

Competing Goals

When you care about numbers that fight each other (faster, but cheaper, but more accurate), it offers three ways to choose: scalarize (best weighted blend), desirability (every goal must clear a minimum bar), and pareto (show all the best trade-offs before committing).

Worked Example

Using the two-setting case named in “The Problem” — batch size and worker count, the pair called out because a bigger batch size only helps once workers are also added:

Factors: batch_size (low / high), workers (low / high) — 2 settings under test.

Matrix: 2 settings falls in the 2–3 tier from the table above, so the engine selects a full factorial: 2² = 4 runs, every combination covered.

Run	batch_size	workers
1	low	low
2	high	low
3	low	high
4	high	high

Effect analysis: ⚠️ no benchmark yet. “Real Effect vs Fluke” defines the method applied to this matrix — a p-value and confidence interval per effect, a low-power warning when four runs aren’t enough to separate signal from noise, and explicit reporting of the interaction term (whether the batch-size effect actually depends on the workers setting) — but no measured p-values, confidence intervals, or effect sizes from an actual run are published in this write-up.

Tech stack

Python + numpy (only runtime dependency) for the DOE matrix math and OLS effects fitting; stdlib subprocess/argparse for the git-worktree isolation layer and sampled metric/guard execution — no other packages. pytest for the dev test suite. Dual plugin manifests (.claude-plugin/plugin.json, .codex-plugin/plugin.json) make the same skill/scripts host-neutral across Claude Code and Codex. Distributed as the npm package @tyroneross/agent-doe-engine (metadata-only — no JS/TS source) alongside a standard pyproject.toml/uv Python install. State is JSON/JSONL on disk under .agent-doe-engine/optimize/ in the consumer project — no database. Apache-2.0.