Back to projects
Active Started Jan 2026

Research

Token-efficient research knowledge base with SQLite FTS5, source tier scoring, and claim verification. Persists to ~/research with project symlinks.

Source Code
Python SQLite FTS5 PyYAML Claude Code Plugin Codex Plugin Omniparse (Node.js, vendored)

Install

claude plugin marketplace add tyroneross/RossLabs-AI-Toolkit
claude plugin install research@rosslabs-ai-toolkit

The Problem

Research done inside a Claude Code session disappears when the session ends. The same questions get re-asked, the same sources re-fetched, contradictions across sessions never surface. Storing research as scattered markdown fixes persistence but leaves search, retrieval cost, and source trust unsolved.

What I Built

A personal research knowledge base that keeps findings searchable across projects, tiered by source quality, and cheap on tokens. One ~/research/ directory, symlinks into every project that needs it.

Architecture

The plugin itself runs no model — it is a deterministic Python script (research.py) plus Claude Code / Codex slash commands and a skill. The LLM work (framing a question, deciding what to fetch, judging relevance, drafting the entry) is done by the host coding agent that invokes those commands; research.py only scores, indexes, persists, and verifies. That split is a documented rule in the plugin’s own CLAUDE.md (“Don’t call external LLM APIs from scripts. Claude does any LLM work.”) and it is what keeps the tool cheap and host-portable — the same research.py runs unmodified under Claude Code or Codex.

flowchart LR
  A["User request<br/>('research X', /research:*)"] --> B["research skill activates<br/>(phrase match)"]
  B --> C["research.py depth<br/>light / standard / deep"]
  C --> D["Phases 1-5 (host agent LLM)<br/>frame -> source -> execute -> synthesize"]
  D --> E["WebFetch / Read<br/>Claude/Codex built-ins"]
  D -.PDF/Excel/PPTX/py/dir.-> F["research:extract<br/>vendored Omniparse (Node.js)"]
  E --> G["Domain tier scoring<br/>T1-T4, deterministic rules"]
  F --> G
  G --> H["3-layer entry drafted<br/>TL;DR / Notes / Raw"]
  H --> I["research:verify<br/>sympy symbolic check (optional)"]
  I --> J["research:save (Phase 6)"]
  J --> K[("SQLite FTS5<br/>RESEARCH_INDEX_DIR")]
  J --> L["Markdown corpus<br/>RESEARCH_CONTENT_DIR"]
  J --> M["Project symlinks +<br/>PORTFOLIO.md regen"]
  N["research:search"] --> K
  N --> L

Models — which, where, why:

Models: none run inside the plugin. research.py never makes an LLM API call — search, source-tier scoring, symlink/portfolio generation, and the depth classifier are all deterministic Python. Every LLM step (choosing sources, extracting judgment calls, drafting the TL;DR/Notes/Raw entry, deciding what’s worth verifying) happens in the host coding agent — Claude via Claude Code, or Codex — that is already running the session and calling the plugin’s subcommands. This was a deliberate choice, not an omission: keeping the script model-free means it costs nothing to run, needs no API key, and behaves identically regardless of which agent or model is driving it.

Tools & infra — which, why:

  • SQLite FTS5 (stdlib sqlite3, no server) — full-text index over the markdown corpus, BM25-ranked; chosen explicitly over a vector database because the corpus is personal-scale and keyword/BM25 search stays in the low milliseconds without an embedding pipeline or extra dependency.
  • PyYAML — parses entry frontmatter (tier, projects, lifecycle status); the one required Python dependency.
  • sympy (optional) — symbolic verification for math claims in /research:verify; the script degrades gracefully if it isn’t installed.
  • Vendored Omniparse (Node.js, self-contained bundle, FSL-1.1-MIT, upstream tyroneross/Omniparse) — /research:extract routes PDF, Excel, PPTX, Python, and directory parsing through it instead of an LLM; a global omniparse on PATH overrides the vendored copy when present. Requires Node.js ≥18 on PATH.
  • Claude Code Plugin + Codex Plugin — dual-host distribution from one repo; both resolve the runtime root via RESEARCH_PLUGIN_ROOT / CLAUDE_PLUGIN_ROOT / CODEX_PLUGIN_ROOT so the same commands, skill, and script work under either agent.
  • Advisory PostToolUse hook (hooks/hooks.json) — fires on Write/Edit to trigger post-write-reingest.sh, re-indexing entries edited outside the normal save path.
  • Seed data files (data/domain-scores-seed.json, data/iffy-domains.csv, data/topic-velocity.yaml) — deterministic inputs to source-tier scoring (e.g. arXiv/DOI/.gov/.edu domains score T1, bare github.com defaults T2, forums/personal-blog platforms score T3, the iffy-domains list forces T4).

How it works

  1. A request triggers the workflow either via the research skill auto-activating on phrasing like “investigate X” / “what’s the current state of Y”, or a direct /research:* slash command.
  2. research.py depth classifies the request as light (0-2 sources, persistence optional), standard (3-8 sources, target 5), or deep (7-15 sources, target 10, verify + persist by default) — a deterministic pre-flight, not a hidden override.
  3. The host agent runs Phases 1-5 (frame → source → execute → synthesize → deliver) using its own built-in WebFetch/Read tools to gather source material — no Python extraction library is involved for web or plain-text sources.
  4. As each source is gathered, research.py scores its domain against a four-tier rubric (T1 official docs/peer-reviewed through T4 forums/personal blogs), using deterministic rules plus the seed data files above — the same URL always scores the same tier.
  5. For binary or office documents (PDF, Excel, PPTX, Python files, directories), /research:extract routes parsing through the vendored Omniparse Node.js bundle instead of an LLM, caching results to RESEARCH_INDEX_DIR/.extract-cache/.
  6. The host agent drafts a three-layer entry — TL;DR (≤150 words, extractive), Notes (bolded key passages + citations), Raw (verbatim excerpts for future re-verification) — with frontmatter recording tier, projects, and lifecycle status.
  7. /research:verify optionally re-checks claims in the entry; math claims get a symbolic check via sympy if it’s installed, else the check is skipped.
  8. /research:save (Phase 6) persists the entry: upserts it into the SQLite FTS5 index, writes the canonical file to RESEARCH_CONTENT_DIR/topics/<top>/<slug>.md, creates a project symlink under RESEARCH_CONTENT_DIR/projects/<name>/ for each tagged project, and regenerates PORTFOLIO.md.
  9. /research:search retrieves later via FTS5 BM25 ranking across canonical entries and any linked external project files, auto-quoting punctuation-heavy terms and returning highlighted snippets; raw FTS5 syntax is available via --fts-query.
  10. Quantitative claims route separately: /research:table-profile or /research:db-profile profile the raw data, /research:analyze-plan generates a self-contained stdlib Python analysis script, and /research:analyze-run executes it, writing results and an audit trail graded High/Medium/Low certainty.
  11. Lifecycle commands keep the corpus honest over time: /research:review surfaces stale/review-due entries, /research:archive moves an entry out with a redirect stub (nothing is ever deleted), and /research:recategorize suggests splits for topics that have grown too large.

Tech stack

Python (stdlib sqlite3 + argparse, one script — research.py) is the entire runtime; PyYAML parses frontmatter and sympy is an optional dependency for symbolic math verification. Storage splits by role: RESEARCH_CONTENT_DIR holds the canonical markdown corpus, archive, inbox, and project symlinks; RESEARCH_INDEX_DIR holds the SQLite FTS5 database, the linked-project registry, verifier logs, and the Omniparse extract cache — split so the plugin code can be replaced without touching the knowledge base. A vendored, self-contained Omniparse (Node.js ≥18, no npm install required) handles binary/office document parsing for /research:extract. The plugin ships as both a Claude Code plugin and a Codex plugin from the same repository, sharing commands, skill, hooks, and script via a portable plugin-root environment variable. No vector database and no external LLM API calls from the script — both were deliberate exclusions, documented in the repo’s own CLAUDE.md.

Impact

Sources are scored against a four-tier rubric (T1 official docs and peer-reviewed through T4 forums and personal blogs), so downstream answers weigh evidence honestly. Optional symbolic verification with sympy catches math claims language models commonly fumble. Lifecycle fields mark notes as active, stale, or archived, so last quarter’s confident claim does not get cited as current.