Six projects, one recurring loop. The gate in the middle is deterministic on purpose —
and two projects show what happens on either side of getting that right.
The short version
Plan, execute, hit a deterministic gate before anything counts as done, have a judge or
critic score what passed, then feed the result back so the next run starts smarter. The
gate has to be deterministic and separate from the thing being judged — the RossLabs Agent
Harness's gate is written in Rust specifically so the model can't compute its own policy,
and it rejected 6 of 7 garbage outputs on one OSS ingest sweep. When that gate is missing
or faked, as Prompt Test Lab's original scoring was — keyword overlap plus a hardcoded
accuracy of 0.75 — the number it produces is worse than no number at all.
The loop in one view
Five stages, one paragraph each — about a two-minute read. "Each stage, for real" below
expands every one of these into the actual mechanism, with the real numbers where the
project write-ups give them.
1Plan
Every build in the portfolio starts by naming the decision before making it — the goal, the files, the pass/fail criteria, and which design calls are being made. Build Loop's plan-verify runs a deterministic lint against that plan before any code moves. The RossLabs Agent Harness treats planning as a routing decision: which model is cheapest that still passes the task's oracle, and whether to invoke the harness at all. Interface Built Right's Design Director produces design-intent.json and concrete validation criteria before implementation starts.
Once a plan clears its check, execution runs local or dispatches to bounded parallel workers. Build Loop dispatches authorized parallel subagents with MECE file ownership. The RossLabs Agent Harness routes to the cheapest model that passes the task's oracle and runs it inside a sandbox with graceful fallback. Interface Built Right's Implement step applies Calm Precision and platform-specific routers to a plan that already named its constraints.
This is the stage the whole pattern is named for: a check that runs independent of the model producing the work, so the model can't grade its own homework. The RossLabs Agent Harness's gate rejected 6 of 7 garbage outputs on an OSS ingest sweep — encoded in Rust so the model never computes the policy. Build Loop runs two gates: a deterministic plan-verify lint before execution, and a seven-step Review chain with a single exit point before merge. Interface Built Right's two-tier scan feeds a Pixelmatch diff at a 0.1 threshold, classified into four verdicts, before a UI change counts as done.
Where a deterministic check can't reach — quality, coherence, whether a diff actually matches the plan's stated intent — an LLM judge steps in, but never ungated. Prompt & Model Benchmark Lab's judge is calibrated, can never set its own composite score, and gets flagged self-graded (confidence stripped) when it happens to be scoring one of its own candidate models. Build Loop splits judgment across two named Review sub-steps: Critic and Fact-Check.
The stage that keeps a loop from being a straight pass repeated forever. Build Loop's optional Learn phase detects recurring patterns across runs and auto-drafts experimental skills or agents with A/B tracking. Prompt & Model Benchmark Lab ingests real build-session signal as field observations, quarantined behind a hard wall so it can never move a calibrated score. The RossLabs Agent Harness's stated trajectory is self-hosting: once the core loop is reliable on a local model, the harness uses it to build its own later phases.
The paragraphs above are the map. These open into the actual thresholds, checks, and
numbers per project, drawn from each project's own write-up.
Plan. A plan that names its own decisions, checked by a deterministic lint before code moves. Read moreShow less
Build Loop's plan has to name the goal, files, pass/fail criteria, commands, and design decisions before build work starts. plan-verify runs a deterministic lint — delete-with-callers, numeric-drift, route-change-evidence, package-state, missing-evidence — and plan-critic runs a softer critique on top of that. Plans crossing five or more synthesis dimensions auto-route to the strong model in one pass, a threshold set from testing where the fast model lost cross-decision context below it.
The RossLabs Agent Harness's planning step is really a routing decision made before any model runs: which model is cheapest that still passes the task's oracle, and whether to invoke the harness at all versus a direct or deterministic solve. Per-model and per-domain profiles inform that routing, so the model is treated as a swappable data row rather than a fixed choice made once.
Interface Built Right's /ibr:build orchestrator runs a Design Director step that produces design-intent.json and concrete validation criteria before implementation starts. The orchestrator — not the agent doing the work — is the one place that knows the active design system and the gates the result has to pass, so an agent can't skip the design step on the way to a passing test.
Execute. MECE-owned parallel subagents, a swappable model behind a sandbox, or a scoped implement step. Read moreShow less
Build Loop's Execute phase runs the work locally or dispatches authorized parallel subagents with MECE (mutually exclusive, collectively exhaustive) file ownership; the orchestrator integrates the final diff rather than letting workers merge their own pieces.
The RossLabs Agent Harness treats the model as a swappable data row — local via Ollama, MLX, or llama.cpp, or cloud via OpenAI, Groq, Fireworks, OpenRouter, Together, or DeepInfra — and routes to the cheapest model that passes each task's oracle. A Program-Aided code_exec lane and sandboxed execution (Docker → macOS Seatbelt → Linux Bubblewrap → explicit opt-in) mean it never runs unsandboxed silently; the fallback chain is itself part of the design.
Interface Built Right's Implement step applies Calm Precision, the web/iOS/macOS design routers, component patterns, and data-viz guidance as needed. Execution here is scoped by the plan and validation criteria set in the step before, not improvised as the build goes.
Deterministic verification gate. 6 of 7 garbage outputs rejected on one sweep, a five-check plan lint, a four-way pixel verdict — none of it graded by the model itself. Read moreShow less
The RossLabs Agent Harness's verification gate is the project's headline result: on an OSS ingest sweep it rejected 6 of 7 garbage outputs a raw model call would have returned. It's encoded in Rust specifically so the model never computes the policy — the check runs the same way whether or not the model would have caught its own mistake, which is the point of a deterministic gate over an LLM one.
Build Loop runs two separate gates. plan-verify's deterministic lint (delete-with-callers, numeric-drift, route-change-evidence, package-state, missing-evidence) blocks execution before it starts. A fixed seven-step Review chain — Critic → Validate → Optimize (opt-in) → Fact-Check → Simplify → Auto-Resolve → Report — with a single exit point gates merge, and routes back to Iterate on a failure, up to 5 repair-and-revalidate passes in classic mode and 25 in autonomous mode.
Interface Built Right runs a two-tier scan on every check: a zero-token deterministic rule engine (wcag-contrast, touch-targets, calm-precision) and a sensor layer of structured summaries, feeding a Pixelmatch diff at a 0.1 threshold classified MATCH, EXPECTED_CHANGE, UNEXPECTED_CHANGE, or LAYOUT_BROKEN. A UI change isn't done when the code compiles or the test passes — it's done when that verdict lands.
Judge / critic. A composite score the judge isn't allowed to set, and a self-graded flag when it grades its own output. Read moreShow less
Prompt & Model Benchmark Lab runs deterministic graders where ground truth exists and an LLM-as-judge (rubric, temperature 0) where it doesn't, and every score carries a calibration confidence. The judge is gated by that calibration and can never set its own composite — the composite is computed in code from its per-dimension scores, not handed to the judge to total up. When the judge model happens to be one of the candidates being scored, the pipeline flags the cell self-graded and strips its confidence, because a model scoring its own work isn't evidence. A one-line Trust banner states plainly whether the evidence behind a score is objective, mixed, or judge-only-and-directional.
Build Loop splits judgment across two named sub-steps inside its Review chain: Critic, which evaluates the diff against the plan's stated design decisions, and Fact-Check, which verifies claims against the actual diff rather than the implementer's report of what it did.
The two projects that show the contrast sit outside the gate stage rather than inside it. Visual Test Suite left judgment to a human — accept a diff as the new baseline or flag it as a regression. Prompt Test Lab's original "judge" was keyword overlap and a hardcoded accuracy = 0.75, not a judge at all — what that cost, and what it changed, is the subject of the section below.
Learn. An optional phase that drafts its own skills, and a quarantine wall so unverified evidence can't move a score. Read moreShow less
Build Loop's optional sixth phase, Learn, detects recurring patterns across runs and auto-drafts experimental skills or agents with A/B tracking, so a lesson from one run becomes part of the next run's toolkit instead of staying in a transcript nobody rereads.
Prompt & Model Benchmark Lab's real build sessions across other projects generate their own model-fitness signal — which model shipped a chunk, whether it held. The lab ingests these as field observations, quarantined behind a hard wall: they can corroborate a score but can never blend into a calibrated score or move a recommendation's confidence. Observational evidence stays labeled observational, permanently — the wall is the point, not a temporary caveat.
The RossLabs Agent Harness states its own trajectory plainly: self-hosting. Once the core verification loop is reliable on a local model, the harness uses that loop to build its own later phases — the gate that verifies the work becomes the mechanism that extends the tool doing the verifying.
Where this shows up
Same five stages, one row each, against the six projects that instantiate the pattern — or
show what happens without it. A dash means the project's own write-up doesn't describe that
stage — not confirmed absent, just not stated.
A deterministic gate costs something — extra passes, extra latency, a Rust check to
maintain. Two of the six source projects show what it looks like on either side of skipping
that cost.
Prompt Test Lab's original "accuracy" scoring computed keyword overlap and a hardcoded
accuracy = 0.75 — not real grading. The gap surfaced during work on the
successor project, Prompt & Model Benchmark Lab, and the project's own page discloses
it rather than quietly patching it: "a benchmark you can't trust is worse than none; it
launders a guess as a number." The successor's entire design — deterministic graders, a
calibrated judge that can never set its own composite, a self-graded flag, a Trust banner —
reads as a direct response to that one failure.
Visual Test Suite is the inverse story — not a fake gate, but a gate that was the whole
product. It ran Playwright screenshots against Pixelmatch baselines for any URL and stopped
there: capture, diff, accept or flag. Its own write-up says plainly what happened next: "the
validation engine that used to be the product is now the verification layer underneath a
build orchestrator" in Interface Built Right. Comparison didn't go away — it became one step
inside a bigger loop instead of the whole product, with a Design Director, platform routers,
and a two-tier scan now built around it.
Read together: the pattern holds when the gate is deterministic and separate from the thing
being judged — a Rust check the model can't compute, a lint that runs before code moves, a
pixel-diff threshold nobody can talk their way past. It fails exactly where Prompt Test Lab
failed — when "grading" is dressed up in real-looking output but is actually a hardcoded
number, and nobody is checking the checker.