# RossLabs Agent Harness

Model-agnostic, local-first Rust agent harness: turns any local or cloud model into a coding agent with deterministic gates, routing, and sandboxed execution.

Status: active

> **In development.** Pre-release, private repo, committed to daily (most recent commit 2026-07-18). The routing policy, gate profiles, and command surface below reflect the current state of `main`, not a finished v1 — interfaces and defaults are still moving.

## Problem

A model call is only as trustworthy as the policy wrapped around it. Left on its own, nothing stops a cheap or open local model from returning garbage, nothing decides whether a task should hit the harness at all versus a deterministic solve, and nothing stops generated code from executing unsandboxed. Any of those left to the model's own judgment reintroduces the exact unreliability the harness is meant to remove.

## Approach

A portable, CLI-first agent harness that treats the model as a swappable data row — local (Ollama / MLX / llama.cpp) or cloud (OpenAI, Groq, Fireworks, OpenRouter, Together, DeepInfra). The value isn't the model; it's the deterministic scaffolding around it, encoded in Rust so the model never computes policy.

## Architecture

> Part of the verification-gated agent-work pattern (see /toolkit/patterns/verification-gated-agents) — here: a per-task oracle drives two decisions at once — whether to invoke the full agentic tool loop at all vs. a deterministic or Program-Aided solve, and (when a model does run) which model, escalating away from cheap/local only when measured evidence says it must.

A single-binary Rust workspace (10 crates under one Cargo workspace: `provider`, `tools`, `gates`, `context`, `engine`, `vault`, `cli`, `eval`, `bench`, `llmwiki`) built around one event-sourced `step()` loop — rehydrate session → assemble prompt → one model call → parse tool intents → gate → execute → append observation → repeat. The model never computes policy; every decision that determines *whether* and *which* model runs, *what* it's allowed to touch, and *whether its output is accepted* is a deterministic Rust function, not a prompt.

```mermaid
flowchart LR
  A["harness run &lt;task&gt;"] --> B["Task planner<br/>derive_solution_mode()"]
  B -->|deterministic transform| D["direct<br/>no model call"]
  B -->|compute task + verify_cmd| E["code_exec<br/>Program-Aided: model writes + runs code"]
  B -->|reliability cell: 6+ attempts, reliable| F["one_shot<br/>single model call"]
  B -->|coding/write or unproven| G["agentic<br/>full tool loop"]
  G --> H["provider::routing::route()<br/>role/task to model"]
  H -->|policy override| P1["pinned model"]
  H -->|reliable / above pass-rate floor| I["Local: Ollama / MLX / llama.cpp"]
  H -->|unreliable or below pass-rate floor| J["Escalate: bigger local, then cloud<br/>OpenRouter / OpenAI / Groq / Anthropic"]
  I --> K["Tool-call ladder<br/>native to schema-constrained to prompt+repair"]
  J --> K
  K --> L["Gate pipeline (deny-first)<br/>SAFE / RISKY / DECISION / PRODUCTION"]
  L -->|deny / unresolved ask| M["Blocked, fails closed"]
  L -->|allow| N["Tool exec: read/write/edit/bash/grep/test"]
  N -->|--sandbox| O["Docker to Seatbelt(macOS) to Bubblewrap(Linux) to refuse"]
  N --> Q["Session tree (JSONL)<br/>resume / fork / tree"]
  E --> R["Verification gate<br/>structural + source-fidelity"]
  N --> R
  R -->|reject| H
  R -->|pass| Q
  Q --> S["Compaction<br/>section-schema summarize, gate log kept verbatim"]
  T["harness bench"] -.folds pass/fail into.-> U[("reliability-*.json<br/>per model x task_class")]
  U -.read by.-> H
```

**Models — which, where, why** (rows drawn from `.harness/models.json`'s live `routes` table, not an exhaustive list):

| Route | Model | Where it runs | Why there |
|---|---|---|---|
| `default` | `qwen3:8b-q4_K_M` | Ollama, local | Small, always-pulled smoke baseline — kept stable so first-run UX and `scripts/e2e_smoke.sh` never depend on a large model download |
| `coding/read`, `coding/write` | `qwen3-coder:30b` | Ollama, local | Top-evidenced local coder among pulled models per live capability probe (`harness models use`), verified 2026-07-03 |
| `cheap` | `google/gemini-2.5-flash-lite` | OpenRouter, cloud | Cost floor for tasks that don't need a strong model |
| `code-cloud` | `moonshotai/kimi-k2.7-code` | OpenRouter, cloud | Cloud coding fallback when the local coder's reliability cell is unreliable or a task needs more context than fits locally |
| `experiment` | `nvidia/nemotron-3-super-120b-a12b` | OpenRouter, cloud | The only OSS model that passed the vault-ingest fidelity gate in the empirical bench (85% fidelity) — cheapest of the reasoning models tried, not the premium-priced ones |
| `think` | `claude-sonnet-5` | Anthropic, native (`/v1/messages`) | Top empirical performer on the same vault-ingest bench (95% fidelity) |
| `deepthink` | `claude-opus-4-8` | Anthropic, native | Escalation ceiling when local and cheap-cloud both fail their oracle |
| `vision` | `qwen/qwen3-vl-235b-a22b-instruct` | OpenRouter, cloud | Multimodal route for image-bearing tasks; no local vision model is in the default registry |

Local vs. cloud is a provider-trait seam, not a hardcoded branch: `provider::cloud` gives every OpenAI-compatible host (OpenAI, Groq, Fireworks, OpenRouter, Together, DeepInfra) one shared request/response mapping keyed by `(provider name, base_url, api-key env var)`; Anthropic gets its own native mapping since its wire format isn't OpenAI-compatible. `route()`'s escalation only fires when a `.harness/routing.json` policy opts in — the default policy (`privacy: allow`, empty escalation rule) means a fresh install can never send code off-device until a human explicitly configures it, and `privacy: strict` refuses cloud outright regardless of any other setting.

**Tools & infra — which, why:**

- **Rust workspace, single static binary (`harness`)** — no interpreter tax, no daemon IPC; `crates/cli` is the only crate with a `main.rs`, everything else is a library the CLI composes.
- **`gates`** — a deny-first permission classifier (SAFE/RISKY/DECISION/PRODUCTION tiers) ported from build-loop's Python `classify_action.py`/`autonomy_gate.py` into ~1.7K LOC of Rust (excluding tests), evaluated in-process before any tool executes, not left to the model to self-police.
- **`tools`** — the typed primitive set (read/write/edit/bash/grep/test) plus the sandbox dispatch: `--sandbox` routes `bash`/`test` through `exec::run_capture`, trying Docker (`docker info` succeeds + image present — the only backend with `--memory`/`--cpus`/`--pids-limit` caps), falling back to macOS `sandbox-exec` (Seatbelt) or Linux `bwrap` (Bubblewrap) — fs/network isolation, no resource caps — and refusing to run at all rather than executing unsandboxed, unless `--allow-no-sandbox` is passed explicitly with a printed warning.
- **`context` + NavGator** — reads `.navgator/architecture/` into an in-memory context graph for repo-map assembly; a missing or malformed map degrades to "unavailable" rather than panicking, since context absence must never kill the loop that calls it.
- **`bench`** — the oracle-measurement loop: re-invokes `harness run` as a real subprocess per fixture task (so a hung run is genuinely killable, not just in-process), runs the fixture's own `verify.sh`, and folds results into per-`(model × task_class)` reliability cells (`reliable`/`usable`/`unreliable`/`insufficient_data` + pass rate) that `routing::route()` reads to decide when to escalate.
- **`vault`** — the concrete instance of the gated-generation pattern: a structural gate (`verify.rs` — required frontmatter keys, allowed `type`/`status` values) plus a source-fidelity gate (`fidelity.rs` — at least 20% of the source document's top 20 salient terms must appear in the output), so a well-formed page about the wrong subject still gets rejected.
- **`llmwiki`** — the one crate wired to OpenTelemetry/OTLP for direct trace export; the rest of the engine doesn't yet emit spans (tracked, not shipped).
- **`tiny_http`** — backs `harness serve`, a local dashboard for routing/model state and ad hoc queries; no external web framework.
- **Rally Point** (`agent-rally-point`, the `rally` CLI) — not a harness dependency (absent from every crate's `Cargo.toml`); it's the multi-agent coordination tool (claims, handoffs, presence) used to develop the harness and the design target for a future bidirectional multi-LLM-team mode, reused rather than rebuilt.

## How it works

1. `harness run "<task>"` loads `.harness/models.json` (capability registry — falls back to a built-in seed if absent or malformed) and `.harness/routing.json` (routing policy — an absent file means local-only, never escalates).
2. `derive_solution_mode()` classifies the task before any model runs: a known deterministic transform short-circuits to **`direct`** (no model call at all); a compute task with a checkable `verify_cmd` routes to **`code_exec`** (the Program-Aided lane — the model writes and executes code, graded against an expected answer within a tolerance, up to 2 attempts); a role/task whose reliability cell has 6+ recorded attempts and a `reliable` verdict gets **`one_shot`** (skip the full loop, one call suffices); a workspace-mutating `coding/write` task, or anything with no cheaper mode proven, falls to **`agentic`** — the full tool loop.
3. For `one_shot`/`agentic`, `provider::routing::route()` picks the model: an explicit `.harness/routing.json` override wins outright; otherwise it reads the folded reliability cell for `(local model, role_task)` from `.harness/bench/reliability-*.json` (produced by step 9 below) — an unreliable/insufficient-data verdict or a pass rate below the policy's threshold escalates to a bigger local model first, then to the best-fit cloud model, in that order. `privacy: strict` bypasses all of this and refuses cloud unconditionally.
4. The chosen model is called through the provider trait — local via Ollama's `/v1`-compatible endpoint at `localhost:11434` (Metal-accelerated on Apple Silicon) or MLX/llama.cpp, cloud via the shared OpenAI-compatible client (OpenRouter/OpenAI/Groq/Fireworks/Together/DeepInfra) or Anthropic's native endpoint — through a tool-calling ladder that degrades native function-calling → schema-constrained (GBNF) → prompt-parsed-with-repair for weaker models or smaller quants.
5. The engine's `step()` loop parses the model's tool intents and evaluates each one through the gates pipeline before execution: a deny-first classifier resolves SAFE/RISKY/DECISION/PRODUCTION to allow/ask/deny; deny and any unresolved ask fail closed.
6. An allowed call executes through the typed primitive set (read/write/edit/bash/grep/test). With `--sandbox`, `bash`/`test` run inside the first available isolation backend — Docker, then Seatbelt (macOS), then Bubblewrap (Linux) — never silently on the bare host.
7. The observation is appended to the session tree (one JSONL file, `id`/`parentId` per entry — the structure `resume`/`fork`/`tree` navigate); when context exceeds the token budget, compaction folds old turns into fixed sections (Goal/Constraints/Progress/Decisions/Next/Critical), with every gate-decision entry kept verbatim, never summarized away.
8. Where a task has a checkable oracle — vault ingestion is the concrete, measured case — the produced output passes through a two-part gate before it's accepted as done: a structural check (required frontmatter keys, allowed field values) and a source-fidelity check (does the output share enough of the source's salient terms). On an OSS ingest sweep this gate rejected 6 of 7 garbage/wrong-subject outputs a raw model call would otherwise have returned as a "successful" page.
9. `harness bench` closes the loop offline: it replays fixture tasks through `harness run` as subprocesses, scores them against each fixture's own oracle, and folds the results into the reliability cells that step 3 reads — so routing escalation is grounded in measured pass rates per `(model, task_class)`, not a static price tier.
10. `harness serve` exposes the same routing and model state through a local dashboard for inspecting live decisions and running ad hoc queries.

## Tech stack

Rust, single-binary CLI (`harness`) over a 10-crate Cargo workspace — `provider` (capability registry, routing, tool-call ladder), `gates` (deny-first classifier, ported from build-loop), `tools` (typed primitives, sandbox dispatch), `context` (NavGator-backed repo map), `engine` (event-sourced loop, session tree, compaction), `vault` (gated document ingestion), `cli`, `eval`, `bench` (fixture runner, reliability folding), `llmwiki` (OTel/OTLP export). Model backends: local Ollama / MLX / llama.cpp; cloud OpenAI-compatible (OpenRouter, OpenAI, Groq, Fireworks, Together, DeepInfra) plus Anthropic Claude native. Sandboxing: Docker (default, resource caps) with macOS Seatbelt / Linux Bubblewrap fallback. `tiny_http` for the local dashboard. NavGator for the repo-map graph; Rally Point for multi-agent build coordination.

## Results

✅ **6 of 7** — on an OSS ingest sweep against the vault's structural + source-fidelity gate, 6 of 7 open-source models' outputs were rejected as garbage or wrong-subject (hallucinated subjects, empty runs, generic content); not one garbage page passed silently.

✅ In the same bench, only 1 of 7 OSS models passed outright — `nvidia/nemotron-3-super-120b-a12b` at 85% fidelity, the cheapest of the reasoning models tried, not the premium-priced ones (`glm-5.2`, `deepseek-v4-pro`, `minimax-m3` all failed to even emit a page). `claude-sonnet-5` topped the same bench at 95% fidelity.

⚠️ No benchmark yet on aggregate routing cost savings, sandbox-fallback frequency in practice, or the accuracy of the `direct`/`code_exec`/`one_shot`/`agentic` mode split across a broader task set — only the vault-ingest gate-rejection and per-model fidelity numbers above have been measured.

## Lessons

- **Self-hosting is the design bet**: once the core loop is reliable on a local model, the harness builds its own later phases, rather than treating self-hosting as a stretch goal to revisit after the harness is "done."
- **Price ≠ task fitness — benchmark, don't assume.** The vault-ingest sweep's cheapest reasoning model outperformed three higher-priced "thinking" models that failed to produce output at all; `harness bench` and reliability-cell-gated escalation exist because a static price-tier router would have picked the losers.
- **Fail closed on the axis you can't fully control.** Both `--sandbox` and cloud escalation share the same shape: an absent capability (no Docker, no explicit routing policy) resolves to the safer default (refuse-with-opt-in, local-only) rather than silently degrading to the riskier one.

## Links

- [HTML project page](https://rosslabs.ai/projects/rosslabs-agent-harness/)