Market Research Platform

Problem

A research project mixes confidential internal strategy PDFs with public job-board exports and competitor decks — the kind of project shown in the platform’s own “AI Market Research” workspace, where a single ingest zone holds both. The analyst needs to answer questions like “what have we already gathered about this competitor’s hiring plans” without re-reading forty documents by hand, and the strategy material can’t leave the machine to get an answer. General chat tools lose track of which claim came from which source; cloud research tools require uploading the documents to someone else’s servers. For competitive and strategic material, neither is acceptable.

Approach

The design bet is that ingestion should be a deterministic, zero-LLM pipeline — parse, chunk, embed, extract entities — so the same document produces the same chunks and the same entities on every run, and LLM calls are reserved for the analysis step, where non-determinism is the point. Everything downstream of ingestion (search, entity lookup, citations) is then reproducible and auditable independent of any model.

What was deliberately not done: no managed vector or graph database. The platform runs SQLite with the sqlite-vec extension (ANN search) plus FTS5 (BM25 keyword search) in one file, and an embedded Kuzu graph in another — both local, both single-process, neither requiring a network hop. It also dropped LangGraph as a pipeline dependency: ingestion, research, and reporting are all linear node sequences with exactly one conditional-edge shape (abort to end on error), so a ~100-line from-scratch async runner (LinearGraph) replaced the framework. And it does not use an LLM to extract entities or relationships during ingestion — spaCy’s bundled statistical NER model runs offline and deterministically, at the cost of being limited to general-purpose entity types rather than domain-specific ones an LLM extractor could name.

Architecture

Uses the standard knowledge-graph pipeline (see /toolkit/patterns/knowledge-graphs) — what differs here: entity extraction is deterministic spaCy NER over each ingested chunk, not LLM- or rule-based relation extraction, and the graph schema defines an entity-to-entity RELATED_TO edge that the current ingestion path never populates — graph queries traverse Document -[MENTIONS]-> Entity co-occurrence, not entity-to-entity relationships.

A multi-service Docker stack, local-first by design:

• Frontend — React + Vite + TypeScript (sources rail, search panel, report panel, settings)
• Backend — FastAPI + Python, orchestrating three pipelines (ingestion, research, reporting) through the in-house LinearGraph runner
• Storage — SQLite + sqlite-vec for vectors and FTS5 for keyword search (vectors.db), an embedded Kuzu graph for entities (Entity, Document, MENTIONS, RELATED_TO), and a separate SQLite file for project/source metadata
• Document parsing — in-process Python (PyMuPDF, python-docx/pptx, openpyxl, pytesseract OCR); docling is a PDF fallback but ships as an opt-in extra, not wired into the default parser routing (kept out of the slim install because it pulls in torch). A Node sidecar (services/parser-sidecar) handles only URL scraping now that file parsing moved in-process.
• Models — see below; two independently-tiered local generation paths (Ollama default, MLX opt-in) plus a cloud fallback

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Embeddings	Ollama bge-m3 / mxbai-embed-large / nomic-embed-text (auto-picks the strongest installed, in that order)	embed_node, ingestion	Stronger semantic similarity on noun-heavy domain text than the bundled MiniLM, which collapses toward keyword overlap
Embeddings fallback	all-MiniLM-L6-v2 (384-dim, bundled sentence-transformers)	embed_node, when Ollama is unreachable	Works fully offline, no daemon dependency — keeps ingestion functional with no local model pulled
Image vision description	Ollama multimodal tag if installed (llama3.2-vision, llava, bakllava, moondream, gemma3)	parse_image, ingestion (image uploads only)	Best-effort — the one ingestion path that is not zero-LLM; skipped silently if no vision model is pulled, so OCR alone still produces a usable chunk
Research analysis	Whatever llm_factory.get_provider() resolves (Ollama primary, MLX if local_llm_provider=mlx, Claude if processing_mode=cloud_allowed and Ollama is down)	analyze_node, research	Single call over already-retrieved context — no map-reduce needed, so it uses the general local/cloud provider rather than a dedicated tier
Report map step	Smallest installed match from ["qwen3:8b","llama3.1:8b","qwen2.5:7b","llama3.2:3b","mistral:7b"]	extract_brief map phase, reporting	Runs once per ~9,000-char packed unit (adjacent chunks from one source), up to 4 concurrent calls — a fast small model keeps per-chunk latency down across the 20-40 units a typical transcript produces
Report reduce step	Strongest installed match from ["mistral-small:22b","gpt-oss:20b","gemma4:26b","qwen2.5:14b","phi-4:14b","gemma2:27b","qwen2.5:32b"]	extract_brief reduce phase, reporting	One synthesis call over the clustered map output — worth the larger/slower model since it runs once, not per-chunk
Cloud fallback	Claude Sonnet	Any stage, only when processing_mode="cloud_allowed" and the local provider is unreachable	Opt-in only — the default processing_mode is "local", so confidential documents never leave the machine unless the analyst explicitly allows cloud

Local-model runtime: Ollama is the default backend (local_llm_provider="ollama"), reached over HTTP at localhost:11434, running on the host rather than in a container so it can use host GPU/Metal — the Docker Compose backend reaches it via host.docker.internal. MLX is an additive, opt-in alternative (local_llm_provider="mlx" / embedding_provider="mlx"), Apple-Silicon only, talking to an OpenAI-compatible local server (mlx-serve or mlx_lm.server, default localhost:8080) — default generation model mlx-community/Llama-3.2-3B-Instruct-4bit, default embedding model mlx-community/bge-small-en-v1.5.

Two different “default generation model” values exist in the repo and both are real, at different layers: Settings.ollama_model falls back to mistral:latest if unset, but the shipped docker-compose.yml overrides it to qwen2.5:14b-instruct-q4_K_M. Neither setting drives report generation, which resolves its own map/reduce tiers independently (table above) — the single ollama_model setting only matters for analyze_node’s one-shot research analysis call.

flowchart LR
    subgraph Ingest["ingestion pipeline — deterministic, one LLM exception"]
        P[parse_file] --> C[chunk_node]
        C --> E[embed_node<br/>Ollama/MLX embedder]
        E --> N[ner_node<br/>spaCy en_core_web_sm]
        P -.image upload only.-> V[vision describe<br/>Ollama multimodal, best-effort]
    end
    N --> VEC[(vectors.db<br/>sqlite-vec + FTS5)]
    N --> KG[(Kuzu graph<br/>Entity / Document / MENTIONS)]

    subgraph Research["research agent"]
        RS[retrieve_semantic] --> AN[analyze_node<br/>1 LLM call]
        RG[retrieve_graph] --> AN
    end
    VEC --> RS
    KG --> RG
    AN --> OUT[analysis + citations]

    subgraph Report["reporting pipeline — map-reduce"]
        GF[gather_findings] --> MP["map: small model x N chunks<br/>(4-wide concurrency)"]
        MP --> RD["reduce: large model x 1<br/>(top-12 chunks re-injected)"]
        RD --> GO["generate_outputs<br/>parallel, no LLM"]
    end
    VEC -.project chunks.-> GF
    GO --> FMT["pptx / docx / email / notes / outline"]

    AN -.cloud fallback.-> CL[Claude<br/>cloud_allowed only]
    RD -.cloud fallback.-> CL

This is a workspace a human operates — distinct from Stratagem, which is an autonomous research agent. The platform keeps the analyst in the loop: upload, search, review, and decide, with the documents never leaving the machine unless the analyst opts into cloud processing.

How it works

1. A source is queued and parse_file() routes by extension to an in-process parser (PyMuPDF for PDF, python-docx/pptx, openpyxl for Excel); URLs are the one case still sent to the Node sidecar’s /scrape endpoint. A standalone image upload also runs OCR (pytesseract) and — the one non-deterministic step in ingestion — a best-effort vision description from an installed Ollama multimodal model, skipped silently if none is pulled.
2. chunk_node splits the parsed text with a from-scratch recursive splitter (split_text, chunk_size=3000, chunk_overlap=200, separators ["\n\n", "\n", ". ", " ", ""]), page-aware when the parser returned multiple pages.
3. embed_node embeds each chunk (embed_texts) with the active embedder — Ollama (bge-m3 / mxbai-embed-large / nomic-embed-text) by default, MLX if configured, falling back to the bundled all-MiniLM-L6-v2 (384-dim) if neither is reachable — into a (project_id, embedder_name)-namespaced table in vectors.db.
4. ner_node runs spacy.load("en_core_web_sm") over each chunk (first 10,000 characters only, for latency), keeps entities labeled ORG, PERSON, GPE, PRODUCT, EVENT, or WORK_OF_ART, and drops names under 2 characters or already seen in that document (seen_entities — deduped per-document, not per-project).
5. Each surviving entity gets a stable id, md5(f"{name}:{label}:{project_id}"), and is written to Kuzu via graph_store.add_entity() (an Entity node) and graph_store.add_mention() (a Document -[:MENTIONS {chunk_index, context}]-> Entity edge, where context is the entity’s containing sentence truncated to 200 characters).
6. At query time, the research agent’s retrieve_graph node calls graph_store.search_entities(query, project_id) — a CONTAINS substring match against entity names, not embedding-based entity linking — and for the top 5 matches calls get_related_entities() (walks RELATED_TO, currently always empty) and get_entity_documents() (walks MENTIONS back to source documents).
7. retrieve_semantic independently pulls the top 10 vector hits from vectors.db; analyze_node merges both result sets into an analysis prompt (general / swot / competitive / trend), resolves a single provider via llm_factory.get_provider(), and sets confidence = min(len(semantic_results) / 10, 1.0) — confidence is scored from the semantic leg only, not the graph leg.
8. Citations returned to the UI carry source_id, chunk_index, and a 200-character snippet, so an answer traces back to the exact chunk, not just the source file.
9. Report generation (gather_findings) pulls every stored chunk and source record for the project — not scoped to one query — as input to a StrategyBrief extraction.
10. extract_brief resolves two independent model tiers unless the caller pins one: a map model (smallest installed match from a preference list favoring 3B-8B tags) and a reduce model (largest installed match from a preference list favoring 14B-32B tags) — see the Models table above.
11. The map step packs adjacent same-source chunks into ~9,000-character units and runs one call per unit against the map model, up to 4 concurrent (DEFAULT_MAP_PARALLELISM), extracting theme candidates, claims, quotes, and financials as JSON.
12. The reduce step clusters map output into top themes, re-injects the verbatim text of the 12 most-cited chunks (600 characters each) so the final synthesis call is grounded against real spans rather than a pure summary-of-summaries, and produces one StrategyBrief JSON via the reduce model.
13. generate_outputs fans out in parallel (asyncio.gather, no further LLM calls) to whichever formats were requested — pptx, docx, email (html), notes (md), outline (md) — each a pure-Python generator writing to data/reports/<project_id>/.
14. processing_mode gates cloud use end to end: "local" (the default) restricts every stage to the local provider; "cloud_allowed" tries local first and falls back to Claude only if the local provider is unreachable — confidential material only reaches Claude on an explicit, per-request opt-in.

Tech stack

React, Vite, TypeScript on the frontend; FastAPI on the backend. LinearGraph — an in-house ~100-line async runner — replaces LangGraph for the three linear pipelines. Storage is SQLite throughout: sqlite-vec for ANN search and FTS5 for BM25 keyword search in one file (replacing an earlier LanceDB/pyarrow store), plus an embedded Kuzu graph for entities. spaCy (en_core_web_sm) does NER. Ollama is the default local model backend for both embeddings and generation, reached over HTTP on the host (not containerized, so it keeps GPU/Metal access); MLX is an additive, opt-in on-device backend for Apple Silicon, speaking an OpenAI-compatible API. Report generation runs its own map/reduce model-tiering independent of the single-call research path. Anthropic Claude Sonnet is the cloud fallback, used only when processing_mode="cloud_allowed" and local is unreachable. Docker Compose ties the services (backend, frontend, parser sidecar) together for local-first operation.

Results

⚠️ no benchmark yet — there is no labeled query set or precision/recall comparison of graph-assisted retrieval against semantic-only retrieval.

Worked example, traced through the real pipeline: a source named q3-competitor-brief.pdf contains the sentence “Salesforce announced a partnership with Slack to expand its GPT-integrated workflows.” ner_node extracts two ORG entities, Salesforce and Slack, each hashed into a stable id and written as Entity nodes with MENTIONS edges back to the document — context holds that sentence, chunk_index points at the exact chunk. A later query, “what is Salesforce doing with Slack,” reaches retrieve_graph: search_entities("Salesforce") matches by substring, get_entity_documents("Salesforce") returns q3-competitor-brief.pdf with that same chunk_index and context, and it’s merged into analyze_node’s prompt alongside the top semantic hits. But get_related_entities("Salesforce") returns an empty list — no RELATED_TO edge was ever written between Salesforce and Slack, because nothing in the ingestion path populates that table. The graph reliably answers “which documents mention X”; it does not yet answer “how are X and Y connected,” despite the schema being built for it.

Lessons

• Deterministic NER was chosen over LLM-based entity/relation extraction for ingestion because ingestion needed to be reproducible and free of LLM cost and latency — the same document yields the same entities on every run. The tradeoff is coarser entity types (spaCy’s six statistical labels) and no relation extraction out of the box, which is why RELATED_TO exists in schema but sits empty.
• A from-scratch LinearGraph runner was chosen over keeping LangGraph as a dependency once all three pipelines turned out to be linear node sequences sharing one conditional-edge shape (abort-on-error) — carrying a general-purpose orchestration framework for that shape was more surface area than it bought.
• SQLite + sqlite-vec was chosen over LanceDB to collapse vector search and keyword search (FTS5/BM25) into a single dependency-light file, trading LanceDB’s native vector-format tooling for one store that also namespaces by (project_id, embedder_name) so swapping embedding models never corrupts existing tables.