Prompt Builder

Install

claude plugin marketplace add tyroneross/RossLabs-AI-Toolkit
claude plugin install prompt-builder@rosslabs-ai-toolkit

Problem

Prompts get rewritten by vibes. A prompt tuned for an Opus backend call gets pasted into a Haiku mobile agent and underperforms, but nothing points at which of the prompt’s parts caused it — was the constraint set too loose, the output format unenforced, or the role missing entirely? “Improve this prompt” from a general-purpose assistant returns a longer version with no way to check whether it is actually better, because nothing scored the original. A concrete instance: a raw prompt like "Summarize this support ticket and tell me what went wrong" looks reasonable on first read, but it has no role, no deliverable definition for “what went wrong,” no length or format constraint, and no anti-hallucination guard — four of the six parts a production prompt needs are silently absent, and there is no mechanism that surfaces that before the prompt ships.

Approach

The design bet: treat a prompt as a six-part contract (Role, Task, Constraints, Context, Output Format, Acceptance Criteria) and make every rewrite pass through the same pipeline that diagnoses the original — classify, diagnose, rewrite calibrated to tier and deployment, then re-score the rewrite on the identical rubric used to score the input. The score is the artifact, not the prose; “improved” is only a claim if there’s a before/after number backing it, and any dimension that lands at 2/5 or below is a hard blocker, not a note.

What was deliberately not built: no LLM API calls inside the tooling — the eval runner (evals/run-evals.mjs) spawns claude -p rather than calling the anthropic or openai SDKs, because Claude Code is the execution engine, not a library the plugin wraps. No built-in garbage collection on the saved-prompt library (.prompt-builder/prompts/<id>/vN.md) — old versions accumulate and get pruned manually, which is an accepted cost of “versions are never overwritten.” And no attempt to fix reliability failures (an agent claiming success it didn’t achieve, skipping a step, taking an unsafe action) through prompt tuning — that class of failure is documented-unreliable to prompt away, so the engine routes it to RISK_NOTES pointing at harness-level deterministic verification instead of iterating on wording.

Architecture

One skill, four progressive-disclosure reference files, five worked examples, five thin slash-command wrappers for human callers, and a zero-dependency eval harness. Agents invoke the skill directly with labeled inputs (raw_prompt, model_tier, deployment, …) per references/caller-contract.md; humans go through /prompt-builder:optimize|score|compare|save|list. There is no agent layer and no MCP server — the skill is a stateless transform, and orchestrating sub-agents around it was judged to add latency without adding quality.

flowchart TD
    A[raw_prompt + caller inputs] --> B["STEP 0 — Detect config\nMODEL_TIER / REASONING_MODE / DEPLOYMENT"]
    B --> C["STEP 1 — Classify\nfunction type, context dependency, output constraint"]
    C --> D["STEP 2 — Diagnose\ntop 3 issues vs 6-Part Stack"]
    D --> E["STEP 2.5 — Select output format\nP2 deployment > P3 downstream-LLM > P4 human-scan > P5 voice"]
    E --> F["STEP 3 — Optimize\nbuild 6-Part Stack, tier + deployment calibrated"]
    F --> G["STEP 4 — Enhance\nreasoning-mode gate, tier-gated CoT/few-shot"]
    G --> H["STEP 5 — Validate\n9-point checklist"]
    H --> I["STEP 6 — Score\nsame 5-dim rubric applied to original"]
    I --> J{prior version supplied?}
    J -->|yes| K["STEP 7 — Iterate\ndiff, re-score, regression flag ≥2pt drop"]
    J -->|no| L[Output contract]
    K --> L
    L --> M["CONFIG / DIAGNOSIS / OPTIMIZED_PROMPT /\nASSUMPTIONS / RISK_NOTES / TEMPERATURE_HINT"]

Reference files are loaded only when relevant (progressive disclosure keeps SKILL.md scannable — a ~400-line target the file currently sits just over at 443 lines): caller-contract.md for programmatic callers, deployment-modules.md for any DEPLOYMENT ≠ interactive, tier-calibration.md for the detailed T1/T2/T3 tables, type-rules.md for the nine classified function types, and scoring.md for the rubric and iteration protocol.

Models: none. No step in the eight-stage pipeline above calls a model API or SDK — package.json carries no dependencies block, and nothing in skills/ or commands/ imports anthropic or openai. Classification, diagnosis, rewriting, and scoring are markdown instructions interpreted in-context by whichever LLM is hosting the calling session (Claude Code, Codex, or any other MCP-compatible caller) — the skill is inference logic, not a wrapped inference call. The one process in the repo that does invoke a model is test infrastructure, not the product: evals/run-evals.mjs spawns the claude CLI (claude -p <prompt>, via Node’s spawnSync) once per regression case to exercise the skill end-to-end and assert on its output — a harness choice made explicitly so evaluation behavior and cost track whatever Claude Code session runs it, rather than pinning a model version inside the tooling.

Tools & infra — which, why:

Tool	Why
Node.js stdlib	Runs `evals/run-evals.mjs` (`spawnSync`, `node:child_process`) and the eval harness’s assertion logic — zero npm dependencies by standing invariant
Python 3 stdlib	Six CI test scripts (`scripts/test_*.py`, `collision_scan.py`) that check plugin-manifest correctness, skill-name collisions, MCP-registration shape, and bridge preflight patterns — stdlib-only so CI has no install step
GitHub Actions	Four workflows (`publish-npm.yml`, `publish-npmjs.yml`, `verify-install.yml`, `verify-manifests.yml`) gate every release on manifest verification before publish and catch version drift between `plugin.json` and the marketplace listing
GitHub Packages (npm registry)	Publish target for `@tyroneross/prompt-builder` — `publishConfig.registry` points at `npm.pkg.github.com`, scoped under `@tyroneross`
Dual plugin manifests (`.claude-plugin/plugin.json`, `.codex-plugin/plugin.json`)	Parallel install surfaces for Claude Code and Codex from one repo; a CI drift check keeps their version and metadata in lockstep so the Codex surface can’t silently fall behind
`claude` CLI	Invoked only by the eval harness (`claude -p`) to run the skill for regression assertions — not called by the skill itself at runtime, and not required to install or use the plugin

How it works

STEP 0 — Detect configuration. Reads three parameters from caller input, inferring and stating the inference if absent: MODEL_TIER (T1 frontier / T2 mid / T3 small-fast, default T2), REASONING_MODE (reasoning for o3/o4-class, GPT-5, Claude extended-thinking / standard otherwise, default standard), and DEPLOYMENT (interactive / backend / rag_pipeline / agent / plugin / eval_judge / personal_mobile, default interactive).
STEP 1 — Classify. Tags the raw prompt along three axes: function type (nine options — Instructional, Transformational, Analytical, Generative, RAG, Agent/Tooling, Evaluation/Judge, Data Pipeline, Reranker — each with a named risk, e.g. Transformational risks silently adding info), context dependency (Closed / Open / Grounded / Interactive), and output constraint (Free-form / Structured / Strict).
STEP 2 — Diagnose. Ranks the top 3 issues from an eight-item checklist (ambiguity, missing constraints, missing context, format issues, hallucination risk, tier mismatch, pipeline integration gap, privacy risk). In practice this resolves to: which of the six parts — Role, Task, Constraints, Context, Output Format, Acceptance Criteria — is absent. A missing part is the first diagnosis, and it predicts which of the five scoring dimensions will land low in Step 6.
STEP 2.5 — Select output format. Applies the override hierarchy (P2 beats P3 beats P4 beats P5): a deployment module’s format requirement wins over “output feeds another LLM → JSON+schema,” which wins over “human consumption → Markdown,” which wins over “voice-first → plain text.” When output must conform to a schema, constrained decoding (API-level response_format: json_schema, Anthropic tool-use, or grammar-constrained sampling via GBNF/Outlines/XGrammar) is preferred over prompt-only JSON — prompt-only adherence runs ~82–92%, constrained decoding exceeds 99%.
STEP 3 — Optimize. Builds the 6-Part Stack, each part calibrated to MODEL_TIER. T1 gets a 1–2 sentence role, a stated goal the model decomposes itself, and 3–5 high-impact constraints. T3 gets an explicit persona with behavioral anchors, numbered sequential task steps (one action per step), exhaustive constraints written as “If X, then Y,” and an exact output schema with 2–3 correct examples plus one error example with its correction. Deployment requirements override tier defaults here (Rule Priority P2 > P5): eval_judge injects a scoring rubric with calibration examples at 3-point (T2) or 5-point (T3) granularity plus explicit tiebreaker rules; rag_pipeline’s synthesis stage injects a grounding clause requiring a passage-ID citation for every claim and refusal to answer beyond source content.
STEP 4 — Enhance. Gated by REASONING_MODE first, then by tier. Reasoning-native targets suppress chain-of-thought and few-shot entirely — CoT was established for non-reasoning models and OpenAI’s own reasoning-model guidance directs writing these prompts without it — and depth is controlled via a named knob (reasoning_effort: low|medium|high or thinking.budget_tokens) surfaced in RISK_NOTES, not simulated with scaffolding. Standard-mode T3 targets get the small-model CoT guard: chain-of-thought is withheld, not merely optional, because small models emit fluent-but-wrong reasoning traces whose errors the final answer inherits; the alternative is least-to-most decomposition (pre-split into ordered sub-steps) or an escalation flag to a larger model.
STEP 5 — Validate. A 9-point pass/fail checklist (unambiguous task, hallucination minimized, output structure explicit, low variance, tier-appropriate, pipeline integrated, type rules applied, format matches consumer, token-efficient). Self-check runs inside the prompt for T1/T2 targets; it is deliberately omitted from T3 prompts, since heavy self-checking measurably degrades small-model output, and pushed to the deploying engineer to verify at design time instead.
STEP 6 — Score. Scores the rewrite on the same 5 dimensions used to diagnose the original — Accuracy Robustness, Clarity, Constraint Strength, Output Determinism, Completeness, 1–5 each, 5–25 total — so the before/after is a true diff rather than two independently-graded artifacts. Any single dimension at 2 or below blocks delivery. Content the rewrite had to invent to fill a gap the original left open is tagged [ASSUMED: ...] in the output rather than folded silently into the prompt text.
STEP 7 — Iterate. When a caller supplies a prior version, diffs the two, re-scores both, and flags a hard regression if any dimension drops ≥2 points (soft-flags at ≥1 point) — the caller must explicitly acknowledge a hard regression before the new version ships.

Tech stack

Plain Markdown + JSON + Node.js stdlib — no runtime dependencies (package.json carries publishing metadata only, no dependencies block, enforced as a standing invariant). Packaged as a Claude Code Skill (SKILL.md + references/ for progressive disclosure + examples/ for worked fixtures), with five thin slash-command wrappers (optimize, score, compare, save, list) and dual plugin manifests (.claude-plugin/plugin.json, .codex-plugin/plugin.json — the Codex adapter) kept in version lockstep by a CI drift check. The eval harness (evals/run-evals.mjs) spawns claude -p per case rather than calling a model SDK directly, so evaluation cost and behavior track whatever Claude Code session runs it. CI runs on GitHub Actions (four workflows: manifest verification, install verification, and two publish paths to GitHub Packages), with test/verification tooling written in stdlib-only Python (scripts/*.py — collision scanning, manifest and MCP-registration checks) alongside the Node-based eval runner.

Results

Worked example — raw prompt: "Summarize this support ticket and tell me what went wrong."

Classified Transformational (primary) + Analytical (secondary), Grounded context, Free-form output. Missing parts: Role, Constraints, Output Format, Acceptance Criteria — only Task is present, and even that is underspecified (“what went wrong” has no defined deliverable). Original score:

SCORE: 6/25 [Accuracy:1 | Clarity:2 | Constraints:1 | Determinism:1 | Completeness:1]

	T1 rewrite (frontier)	T3 rewrite (small/fast)
Role	1 sentence: “You are a support-ticket analyst who identifies root cause and summarizes clearly.”	Explicit persona + 3 behavioral anchors: what it IS (“a triage analyst”), IS NOT (“not a customer-facing responder”), and its scope boundary.
Task	Single stated goal: summarize the ticket and identify what went wrong. Model decomposes.	Numbered steps: (1) restate the issue in one sentence, (2) extract the failure point, (3) extract resolution status, (4) flag missing info.
Constraints	4 high-impact: no fabricated details, cite the ticket line for any claim, flag unresolved items, 150-word cap.	Exhaustive “If X, then Y” set (9 rules) covering missing fields, contradictory timestamps, multiple issues in one ticket, PII redaction.
Output Format	Described structure: summary + root cause + status, in prose.	Exact schema (`{summary, root_cause, status, unresolved: bool}`) + 2 worked examples + 1 malformed-input example with the correct handling.
Acceptance Criteria	Source-supported or tagged `[INFERRED]`.	Every claim traceable to a ticket line; zero fabrication; schema validated against both examples.

T1 — SCORE: 22/25 [Accuracy:4 | Clarity:5 | Constraints:4 | Determinism:4 | Completeness:5]   (+16 vs original)
T3 — SCORE: 25/25 [Accuracy:5 | Clarity:5 | Constraints:5 | Determinism:5 | Completeness:5]   (+19 vs original)

Both deltas are computed with the same rubric that diagnosed the original — that consistency is what makes “+16” and “+19” comparable numbers rather than two unrelated scores. ⚠️ These are the engine’s own rubric-scored outputs, not an independently graded benchmark — the project’s own CLAUDE.md documents this as a known limitation (“no external ground truth… two different calls may score the same prompt 1–2 points apart”), which is why eval thresholds are set conservatively (target ≥8/10 of 12 cases passing on first run, iterate to ≥9/10) rather than treated as ground truth. ⚠️ No logged run of evals/run-evals.mjs is available for this write-up — the 12-case harness (reranker-t3, backend-improver-t2, eval-judge-t2, regression-detection, reasoning-model-suppress-cot, etc.) exists and is checked after every SKILL.md edit, but no benchmark pass-rate is cited here because none was captured.

Lessons

Scoring the original prompt and the rewrite was chosen over scoring only the output, because the delta is what makes “improved” falsifiable — a rewrite landing at 22/25 means nothing without the 6/25 baseline it replaced; a system that only scores its own output can’t be wrong about whether it helped.

Reasoning-native CoT/few-shot suppression (added in v3.2) was chosen over keeping chain-of-thought as a tier-gated default, because reasoning-native models (o3/o4-class, GPT-5, Claude extended-thinking) internalize the plan internally — adding “think step by step” on top is redundant latency, not extra accuracy, and the depth lever moves from prompt text to a runtime parameter (reasoning_effort, thinking.budget_tokens) the prompt can only name, not set.

Reliability failures were scoped out of the prompt layer entirely (Rule Priority P7) rather than attempting to prompt-tune them away, because false-success claims and unsafe actions are documented-unreliable to fix with wording — the engine surfaces them in RISK_NOTES and points at harness-level deterministic verification instead of iterating on a prompt that was never the actual failure point.