Atomize AI: Research Intelligence

Problem

Tracking AI research across arXiv, Anthropic, and OpenAI required either expensive tooling or manual synthesis; existing aggregators didn’t connect related findings across sources. Atomize AI succeeds Atomize News Search, which handled RSS feeds and general tech aggregation. The move to academic and corporate research sources demanded deeper synthesis: entity extraction that preserves directional relationships (distinguishing “Google acquired DeepMind” from “DeepMind acquired Google”) and LLM inference that stays cost-efficient at scale. Atomize News Search remains live as the earlier, narrower iteration; Atomize AI is the current, actively developed system.

Approach

The design bet: verify LLM-extracted relationships against a second, non-LLM signal rather than trusting single-pass extraction. LLMs alone hallucinate relationships and reverse subject-object positions on directional claims, so a dedicated grammatical-dependency check was designed and built to run against every extracted relationship before it enters the graph — a Flask/spaCy microservice, deployed on Railway and reachable in production. That verification step is not currently wired into the live extraction path, though; the extraction and knowledge-graph-population services don’t call it, so relationships enter the graph unverified today. The same bet shows up in entity admission — rather than accepting every entity an LLM surfaces, a multi-signal scoring gate decides what’s worth keeping, because unrestricted extraction causes a knowledge graph to grow into a soup of every name that ever appeared in any article; this gate is live, filtering noise from real entities during RSS ingestion. What was deliberately not built: a single-model extraction pipeline (rejected for hallucination risk and cost — routing across Groq and OpenAI by content complexity was chosen instead) and an ungated graph (rejected because pruning has to be answerable to evidence, not ad hoc filtering after the fact).

Architecture

Atomize AI is a knowledge-graph system end to end. The full pipeline — sources → model routing → entity/relationship extraction → six-signal entity filtering → graph → synthesis — is described below, including which model runs at each stage and why, and which datastores hold what. A dependency-grammar verification microservice was built and deployed alongside this pipeline but is not currently called from it (see below). For how this same pattern recurs across other projects, see the knowledge-graph pattern; what’s specific here is six-signal entity filtering, a built-and-deployed (but not live-wired) dependency-grammar verification service, 9,556 entities scored in the first full-corpus run, and 76% cheaper inference via Groq LPU routing.

flowchart LR
  A["Sources: arXiv, Anthropic, OpenAI"] --> B[Ingestion]
  B --> C["10-model router<br/>Groq LPU / OpenAI"]
  C --> D["Entity + relationship extraction"]
  D --> F["Six-signal entity filter<br/>(live, per-article on ingestion)"]
  F -->|above threshold| G[("Knowledge graph<br/>Postgres + pgvector")]
  F -->|below threshold| H["Admin review queue<br/>promote / reject / merge"]
  H -.labeled data.-> F
  G --> I["Progressive pyramid synthesis"]
  G --> J["AI Search"]
  G --> K["AI Trends"]
  I --> L["LangSmith: IntelligentSearchCompositeEvaluator<br/>(4 evaluators)"]
  B --> M["OpenTelemetry + SigNoz + Prometheus"]
  D --> M
  N["spaCy dependency-grammar service<br/>(Railway, built + deployed, NOT wired into D)"]

How it works

1. Ingestion pulls source material from arXiv, Anthropic, and OpenAI.
2. A 10-model router across Groq and OpenAI selects a model per request based on content complexity — Groq handles straightforward summarization and entity extraction, OpenAI handles analysis requiring deeper reasoning.
3. The extraction stage pulls entities and relationships from each document, assigning an initial confidence score.
4. A spaCy NLP microservice (Flask, deployed on Railway, $5/month) is built to analyze grammatical dependencies — subject-verb-object structure — and verify extracted relationships, responding in 55ms with a designed 20% confidence boost for verified relationships. The service is live and reachable, but the extraction and graph-population code doesn’t call it yet — its only caller in the codebase today is a standalone diagnostic script, so this verification step isn’t part of the live pipeline.
5. A six-signal filter scores every extracted entity as it’s ingested (inline during RSS processing, not a separate nightly job): word frequency, capitalization/part-of-speech pattern, entity type, multi-word count, position in the article title, and mention frequency. It’s a noise-versus-real-entity filter rather than a significance ranker.
6. Entities above the scoring threshold pass into the canonical graph; entities below it land in an admin review queue for promote/reject/merge decisions. A separate admin-facing tool analyzes 30 days of those correction decisions and proposes threshold-weight adjustments with supporting evidence — a suggestion generator a human reviews, not an automated retraining job.
7. The progressive pyramid layers findings from specific papers (base) to cross-source patterns (middle) to emergent insights (apex), with pgvector similarity search connecting related entities across documents.
8. LangSmith’s IntelligentSearchCompositeEvaluator runs four evaluators — timeline alignment, answer relevance, insight quality, source tracking — against AI Search summary generation, feeding model-selection and prompt-refinement decisions. A separate five-evaluator set (factuality, coherence, relevance, business impact, format adherence) was built for an earlier Smart Brevity summary format and remains in the codebase but has no live caller.
9. OpenTelemetry and SigNoz instrument every ingestion, extraction, and graph-update span (latency, model-selection rationale, confidence-score evolution); Prometheus tracks cost per synthesis and accuracy trends over time.

Tech stack

Next.js 16, Groq LPU + OpenAI (multi-model routing via LangChain), PostgreSQL + pgvector, spaCy NLP (Railway microservice), Prisma, LangSmith (evaluation), OpenTelemetry + SigNoz + Prometheus (observability).

Results

• 76% cost reduction vs. OpenAI-only inference, from Groq LPU routing on straightforward extraction/summarization tasks.
• spaCy dependency-grammar verification is built and Railway-deployed, responding in 55ms with a designed 20% confidence boost — but it isn’t wired into the live extraction path yet, so relationships enter the graph without this check today.
• First full-corpus entity-scoring run: 9,556 entities scored; roughly 40% landed below the auto-accept threshold and routed to the review queue. (Point-in-time database snapshot, not a fixed or repeatable benchmark.)
• Worked example — knowledge graph: 50 entities, 124 connections extracted from a coverage set, with typed relationships (owns, integrates, develops, regulates). (Point-in-time database snapshot.)
• Worked example — AI Search: query “Enterprise AI Adoption” pulls from 50 articles, returns a structured summary with numbered source citations. (Point-in-time database snapshot.)
• Worked example — AI Trends: featured trend “Industry Tightens AI Governance” — 132 articles across 8 sources. (Point-in-time database snapshot.)

Lessons

1. Dependency-grammar verification via a dedicated spaCy microservice was chosen over trusting LLM-only extraction because LLMs reverse subject-object positions on directional claims — the “who acquired whom” failure mode needed a non-LLM check, not a better prompt. The service was built and deployed to prove the approach out; it isn’t wired into the live extraction path yet, which is the gap between “designed” and “shipped” on this bet.
2. Six-signal entity filtering during ingestion was chosen over accepting every LLM-extracted entity because unbounded extraction drifts the graph toward noise; making admission answerable to measurable signals (word frequency, capitalization/POS, entity type, multi-word count, title position, mention frequency) keeps pruning defensible instead of arbitrary.
3. A 10-model router across Groq and OpenAI was chosen over standardizing on one provider because task complexity varies enough within the pipeline that matching cheap, fast inference to simple extraction and reserving OpenAI for deeper reasoning produced the 76% cost delta without a quality tradeoff on the tasks that mattered.