Spectra

Install

claude plugin marketplace add tyroneross/RossLabs-AI-Toolkit
claude plugin install spectra@rosslabs-ai-toolkit

The Problem

Marketing captures span platforms that each demand their own stack: Chrome DevTools for web, Accessibility for macOS, simctl for iOS, and simulator workflows for watchOS. Stitching together screenshots, short clips, usage sequences, and exports across all of them is manual work that does not scale.

What I Built

Spectra is the primary Ross Labs capture tool for product demos: screenshots, guided usage sequences, smart framing, and a tagged export library across web, macOS, iOS, and watchOS. Video capture is supported where available and expanding.

Architecture

Spectra is an MCP server plus a local daemon — it exposes UI automation and capture as tools a host agent calls, rather than running its own planning loop. The primary path is host-routed: Claude Code, Codex, or another MCP-speaking coding agent reads a snapshot, decides the next action, and calls Spectra’s tools to execute it. Spectra.app, a standalone macOS menu-bar fallback, only plans locally (via Claude Haiku) when no host agent is present.

flowchart LR
  H["Host agent (Claude Code / Codex)<br/>plans from spectra_snapshot"] -->|MCP tool calls| D
  SA["Spectra.app standalone<br/>WalkthroughPlanner + Claude Haiku 4.5"] -.fallback, no host agent.-> D
  D["spectra_connect<br/>driver select"] --> CDP["CdpDriver (web)"]
  D --> NAT["NativeDriver (macOS AX)"]
  D --> SIM["SimDriver (iOS / watchOS)"]
  CDP --> SNAP["spectra_snapshot<br/>AX / DOM tree"]
  NAT --> SNAP
  SIM --> SNAP
  SNAP -.AX empty or thin.-> VIS["Vision fallback<br/>Apple Vision framework, on-device"]
  VIS --> STEP
  SNAP --> STEP["spectra_step<br/>Jaro-Winkler resolve engine"]
  STEP --> ACT["spectra_act<br/>driver executes"]
  SNAP --> AN["spectra_analyze<br/>6-factor importance scoring"]
  AN --> CAP["spectra_capture<br/>clean + auto-frame"]
  ACT --> CAP
  CAP --> LIB[("Library<br/>.spectra/library/index.json + media/")]
  CAP --> DEMO["spectra_demo<br/>ffmpeg spotlight + captions"]
  DEMO --> LIB

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Host-routed planning (primary path)	None — the host LLM (Claude Code, Codex, or another MCP-speaking coding agent) plans	Outside Spectra; the host reads spectra_snapshot output and calls Spectra’s MCP tools	Keeps Spectra a pure automation/capture layer — no API key, no model cost, no planning logic duplicated inside the tool for the path almost all sessions use
Standalone fallback planning	Claude Haiku 4.5 (default, overridable per call)	WalkthroughPlanner.swift + AnthropicClient.swift, Spectra.app only, one non-streaming call per turn	Small, fast model for single-turn action-plan JSON when no host agent is driving; temperature 0 for deterministic plans; the user’s own API key lives in Keychain and never reaches the daemon
Element-grounding fallback	Apple Vision framework (on-device, no network call)	computer-use/vision-fallback.ts, gated on AX-node-count (fires only when the AX tree is empty or thin)	AX reads take ~30–80ms and come pre-labeled; screenshot-grounding is ~5–50x slower and, per the code’s own cited benchmark, under 70% accurate on ScreenSpot-Pro — so vision is a fallback, never the default
Natural-language element resolution	Jaro-Winkler string similarity — deterministic, no model	core/resolve.ts, backs spectra_step	Matching “click the Log In button” to an AX label is a string-similarity problem, not a reasoning problem; resolving it without a model call keeps it instant and free
Screen importance scoring	6-factor weighted heuristic (fixed weights, derived from UEyes CHI 2023 eye-tracking research)	spectra_analyze	Role, position, interactivity, label quality, density, and prominence combine into a fixed formula — reproducible scoring with no inference cost
Change / scene detection	dHash perceptual hash + AX structural diff — deterministic	intelligence/change.ts, spectra_demo action=scan	A binary “did the screen change” signal doesn’t need a model; a gradient hash and a tree diff answer it directly

Models: minimal by design. Spectra runs model-free for its default host-routed usage; the one LLM in the stack (Claude Haiku 4.5) only activates in the standalone-app fallback path, and even the vision-grounding fallback is a local Apple framework, not a hosted model.

Tools & infra — which, why:

• MCP server (@modelcontextprotocol/sdk) — exposes 15 tool handlers (connect, snapshot, step, act, capture, demo, analyze, discover, session, library, llm-step, record, walkthrough, replay, computer-use) over stdio to any MCP-speaking host.
• Node.js daemon (node:http, custom JSON envelope in src/contract/wire.ts) — a long-running process holding driver connections and sessions, reachable over a Unix domain socket by default or optional loopback TCP with bearer-token auth, so the CLI/MCP process and the daemon can run as separate PIDs.
• CdpDriver — a hand-rolled Chrome DevTools Protocol client (own connection handling plus 7 domain wrappers), not Puppeteer or Playwright — keeps the web driver dependency-light for a narrow automation surface.
• NativeDriver / SimDriver — a Swift binary (spectra-native, built via swiftc against Foundation, ApplicationServices, AppKit, CoreGraphics, Vision, ScreenCaptureKit, AVFoundation) bridging the macOS Accessibility API and simctl for iOS/watchOS.
• FFmpeg — external binary, resolved via which ffmpeg rather than bundled — handles transcode, spotlight-focus rendering, lower-third caption burn-in, and scene-change scanning inside spectra_demo.
• Zero-dependency PNG codec built on node:zlib — decode/encode/crop/resize without pulling in sharp or jimp.
• Storage — no database. A flat .spectra/ directory per project (found by walking up to .git, package.json, or an existing .spectra, else ~/.spectra) holds sessions/<id>/ for ephemeral step recordings and library/index.json + library/media/<capture-id>/ for the persistent tagged catalog; index writes are atomic (tmp file + rename).
• ScreenCaptureKit + AVFoundation (Swift, spectra-composite-capture binary) — native screen recording with hardware H.264/HEVC encoding.
• Next.js dashboard (web-ui/) — local-only browse/manage/export UI served at :4300, separate from the MCP/daemon runtime.
• Vitest — 30 test files, 329 tests per the repo’s own README, run via npm test.

How it works

1. A host agent (or Spectra.app standalone) calls spectra_connect with a target — a URL, an app name, or sim:device — which selects a driver: CdpDriver for web, NativeDriver for macOS, SimDriver for iOS/watchOS.
2. spectra_snapshot returns the current accessibility tree: a compact element list with roles, labels, bounds, and available actions. If the AX tree comes back empty or thin, the Apple Vision on-device fallback grounds the screen from pixels instead and returns AX-shaped nodes so the rest of the pipeline doesn’t need to know which source produced them.
3. In the host-routed path, the host LLM reads that snapshot and decides the next action itself, then calls spectra_act or spectra_step directly — no model runs inside Spectra for this step. In the standalone-app path, WalkthroughPlanner sends the snapshot (plus an optional screenshot) to Claude Haiku 4.5 and parses the returned action plan.
4. spectra_step takes a natural-language intent (“click the Log In button”) and resolves it to a specific element with the Jaro-Winkler string-similarity engine, returning a confidence score alongside the resolved element ID.
5. spectra_act executes the resolved action (click, type, clear, select, scroll, hover, focus) on the driver and returns the updated snapshot.
6. spectra_analyze scores every element on screen with the 6-factor importance heuristic and groups them into labeled regions (Navigation, Form, etc.), also classifying overall UI state (loading, error, empty, populated).
7. spectra_capture takes the screenshot or recording. Visual cleanup (hide scrollbars, clean the simulator status bar, strip cursor artifacts) runs first if requested, then framing — full, element, region, or auto — crops to the relevant content using the importance scores from step 6.
8. spectra_discover automates steps 2–7 across an entire app: BFS-crawls navigable elements, captures and frames each new screen, deduplicates by role+label fingerprint, and flags (without capturing) any screen showing password or credential fields.
9. spectra_demo turns a raw recording into a polished clip: scan runs dHash-based scene-change analysis to find active segments, polish-clip/polish-script apply spotlight focus and burn in numbered lower-third captions via FFmpeg, and run-script can drive a live CDP browser through scripted beats while capturing.
10. Finished captures land in .spectra/sessions/<id>/ (ephemeral) or get promoted into .spectra/library/ (persistent, tagged) via spectra_library action=add — a flat JSON index plus a media/<capture-id>/ file, no database involved.

Tech stack

Node.js + TypeScript for the MCP server, daemon, and drivers; Swift for the native macOS/iOS/watchOS bridge (Accessibility API, simctl, ScreenCaptureKit, Apple Vision framework). The web driver is a hand-rolled Chrome DevTools Protocol client, not Puppeteer/Playwright. Element resolution (Jaro-Winkler), importance scoring, and change detection are all deterministic algorithms — no model in the loop. The one LLM in the stack, Claude Haiku 4.5, runs only in the standalone Spectra.app fallback planner when no host agent is present. FFmpeg (external binary) handles video transcode and demo-clip production. Storage is a flat .spectra/ directory — JSON index + media files, no database. A Next.js dashboard serves a local browse/export UI.

Impact

Spectra supersedes Showcase. Showcase proved the value of development-time capture; Spectra makes that workflow cross-platform and organized enough to become the long-term demo library. The site should now treat Spectra as the default capture recommendation and Showcase as the legacy migration path.

Worked Example

Consider a demo that needs to show the same product flow on its web app and its macOS companion. Previously this required Chrome DevTools scripting for the browser session and a separate manual walkthrough for the native app, with no shared place for the resulting assets.

With Spectra, a single capture session spans both surfaces. For the web leg, the Node runtime attaches over Chrome DevTools Protocol and steps through the flow, taking screenshots at each point in the guided usage sequence. For the macOS leg, the Swift native bridge drives the same kind of walkthrough through the Accessibility API, so the app does not need scripted UI hooks to be captured. iOS/watchOS legs of the same flow route through simulator control instead.

Each screenshot in the session carries framing metadata and tags, and the full set — web and macOS together — lands in one local library entry with an export manifest, rather than in two disconnected asset folders. This is the mechanism that lets Spectra replace Showcase as the default: Showcase’s development-time capture covered one platform at a time; Spectra’s session model is what makes the cross-platform library possible.

Capture duration, session count, and library size: ⚠️ no benchmark yet — the write-up does not report usage figures for this workflow.