Atomize News: Intelligent Search

Successor: This project’s ingestion and knowledge-graph pipeline evolved into Atomize AI, which extended entity extraction with grammatical verification and moved from general tech-news RSS aggregation to academic and corporate AI research sources. Atomize News Search remains live as the earlier, narrower iteration.

Problem

Product and strategy executives tracking tech news face a firehose with no signal filtering. RSS readers deliver every item unranked; news aggregators cluster by keyword but have no semantic understanding of which stories are actually the same event, so coverage fragments instead of collapsing. Concrete instance: a search for “AI regulation” needs to also surface articles filed under “tech policy laws” — semantically the same topic, lexically distinct — which a keyword-only aggregator misses entirely.

Approach

The system is built on a Retrieval-Augmented Generation (RAG) pattern extended with a “Fact-First” architecture: instead of feeding retrieved articles straight to an LLM, the pipeline first structures retrieval results into a Knowledge Graph of entities and a Timeline of dated events, then generates the summary against that fact layer rather than against the raw articles directly. What was deliberately not done: single-pass RAG (top-k articles dumped into an LLM with a “summarize this” prompt) — rejected because the summarization step, not the retrieval step, is the primary hallucination vector in RAG pipelines; a pre-computed, checkable fact layer catches drift that a raw-document prompt can’t.

Architecture

Uses the standard knowledge-graph pipeline (see /toolkit/patterns/knowledge-graphs) — what differs here: news-article chunking at the RSS/HTML ingestion boundary (BullMQ workers normalize each article before extraction), entity/relationship extraction tuned for event-oriented news facts (e.g., “OpenAI” owns “ChatGPT”) that feed a dated Timeline layer rather than just a static entity graph, and a 3-tier fallback chain on the synthesis call so a search never returns empty when one LLM provider is down. A standalone spaCy microservice for grammatical-relationship parsing exists in the repo (spacy-service/, deployable to Railway) but its TypeScript client isn’t imported anywhere in the live extraction path — the directionality problem it targets is unsolved here; Atomize AI is where that check actually runs in production.

flowchart LR
  subgraph Ingest["Ingestion — BullMQ workers"]
    A["RSS/HTML sources"] --> B["Normalize + chunk per article"]
    B --> C["Embed<br/>text-embedding-3-small"]
    B --> D["Extract entities/relations<br/>gpt-4o-mini, temp 0.1"]
  end
  C --> E[("Postgres + pgvector<br/>HNSW index")]
  D --> F[("Knowledge Graph")]

  subgraph Retrieve["Query time"]
    G["User query"] --> H["ComplexityDetector<br/>simple / medium / complex"]
    H -->|query expansion, rerank| GX["gpt-5-nano / gpt-5-mini / gpt-5"]
    G --> I["Query Orchestrator"]
    I -->|vector, 70%| E
    I -->|keyword, 30%| J["Postgres full-text search"]
  end
  E --> K["Hybrid retrieval<br/>50-200 candidates"]
  J --> K
  F --> L["Timeline extraction<br/>dated, themed events"]
  K --> L

  subgraph Synth["Fact-checked synthesis — 3-tier fallback"]
    L --> M{"Tier 1: Groq<br/>llama-3.1-8b / llama-4-scout / llama-3.3-70b<br/>by intent complexity"}
    M -->|on failure| N["Tier 2: OpenAI gpt-5-nano"]
    N -->|on failure| O["Tier 3: deterministic template<br/>no LLM, free, instant"]
  end
  M --> P["Key Insight + citations"]
  N --> P
  O --> P
  K --> Q["Trend clustering<br/>Events -> Industry Trends<br/>llama-3.3-70b-versatile"]

Models — which, where, why:

Stage	Model / tier	Where it runs	Why there
Embeddings	OpenAI text-embedding-3-small (1536-dim)	EmbeddingService / BatchEmbeddingService (lib/services/), ingestion + query time	Code comment marks it as the replacement for text-embedding-ada-002 — “5x cheaper, better accuracy”; article and query vectors share the same model so they compare in the same space
Entity/relationship extraction	OpenAI gpt-4o-mini, temperature 0.1	ExtractionService + HybridExtractor (lib/knowledge-graph/), ingestion queue	Low temperature for consistent structured JSON; mini-tier keeps per-article extraction cost down at ingestion volume
Query complexity routing	OpenAI gpt-5-nano / gpt-5-mini / gpt-5 / gpt-5-chat-latest	ComplexityDetector (lib/utils/complexity-detector.ts), query time	Simple keyword/entity lookups get the cheapest tier; multi-hop or comparison queries escalate to gpt-5; gpt-5-chat-latest is reserved for premium-tier or high-urgency requests
Fact-checked synthesis — Tier 1 (primary)	Groq llama-3.1-8b-instant / llama-4-scout-17b-16e-instruct / llama-3.3-70b-versatile, chosen by selectSynthesisModel() on a 0–1 intent-complexity score (<0.25 / <0.65 / else)	ProgressivePyramidService, query time	Groq LPU inference is fast and cheap enough to be the default path for the large majority of synthesis calls
Fact-checked synthesis — Tier 2 (fallback)	OpenAI gpt-5-nano	Same call site, only if the Tier 1 Groq call throws	Keeps synthesis available on an OpenAI-only path independent of a Groq outage, without paying its cost on the common path
Fact-checked synthesis — Tier 3 (guaranteed)	None — deterministic template analyzer	Same call site, only if both Tier 1 and Tier 2 throw	Zero-cost, instant fallback so a search never returns empty because both LLM providers were unreachable
Trend clustering	Groq llama-3.3-70b-versatile (via LangChain ChatGroq)	UnifiedTrendDetectorService	Two-stage clustering (articles → Events → Industry Trends) runs once per batch, not per-article, so the larger/slower model’s cost is amortized

Tools & infra — which, why:

• PostgreSQL + pgvector, HNSW index — one datastore for relational article/entity data and vector similarity search; migration 20251013000653_upgrade_to_hnsw_index replaced an earlier IVFFlat index for faster ANN queries.
• Prisma (@prisma/client 7.x) + pg — ORM and migrations against Postgres.
• BullMQ + ioredis (Redis) — ingestion, embedding, and prefetch job queues (lib/queues/bullmq-enhanced.ts); connects via REDIS_URL.
• Railway — hosts the BullMQ worker process (railway.json: node --import tsx scripts/start-hybrid-workers.ts).
• Vercel — hosts the Next.js 16 app and its cron-triggered ingestion/trend endpoints (vercel.json: /api/cron/refresh-rss, /api/cron/detect-trends, every 6h).
• LangChain (@langchain/core, @langchain/groq, @langchain/openai) — common chat-model interface wrapping both Groq and OpenAI calls for extraction and synthesis.
• LangSmith — traces every LLM call (wrapOpenAI, explicit tags/metadata on Groq calls) and backs the eval scripts (eval:pyramid, eval:reranker, eval:themes).

How it works

1. Ingestion — BullMQ workers scrape and normalize thousands of articles per run from RSS and HTML sources in parallel, indexing each with Publisher Name and Credibility Score.
2. Embedding — EmbeddingService and BatchEmbeddingService generate 1536-dimension vectors via text-embedding-3-small for every ingested article and incoming query, stored in Postgres + pgvector with HNSW indexing.
3. Entity/relationship extraction — ExtractionService and HybridExtractor call OpenAI gpt-4o-mini at temperature 0.1 to pull people, companies, and relationships from each article as structured JSON, populating the Knowledge Graph (e.g., “OpenAI” owns “ChatGPT”).
4. Query complexity routing — ComplexityDetector classifies each query simple / medium / complex by word count and keyword patterns, then picks an OpenAI GPT-5 tier for query expansion and reranking: gpt-5-nano for simple lookups, gpt-5-mini for medium, gpt-5 (or gpt-5-chat-latest for premium/high-urgency requests) for complex multi-hop queries.
5. Hybrid retrieval — the Query Orchestrator runs Vector Search (70% weight, pgvector similarity) and Keyword Search (30% weight, Postgres full-text search) in parallel, fetching 50-200 candidate articles, then applies Cluster Boosting (injecting articles from a matched Trending Cluster) and a weighted-ranking blend to produce the Related News list.
6. Timeline construction — before synthesis, the system extracts dated Timeline Events from the retrieved articles, filters spam/promotional signals, and groups events by Theme rather than raw chronology.
7. Fact-checked synthesis, 3-tier fallback — ProgressivePyramidService first tries Groq, with selectSynthesisModel() picking llama-3.1-8b-instant, llama-4-scout-17b-16e-instruct, or llama-3.3-70b-versatile off an intent-derived complexity score; the prompt requires the model to ground the Key Insight in the TIMELINE events and cite specific entities. If that call throws, Tier 2 retries the same prompt against OpenAI gpt-5-nano. If that also throws, Tier 3 falls back to a deterministic, no-LLM template analyzer so the request still returns a result.
8. Trend clustering — a two-stage algorithm groups articles into discrete Events, then aggregates Events into Industry Trends; UnifiedTrendDetectorService classifies each cluster (Single Entity vs. Industry Trend) using llama-3.3-70b-versatile on Groq via LangChain’s ChatGroq.
9. Source display — logged-in users can apply a Curated Sources filter stored in user preferences (currently disabled in the UI); clustered events show a source count (e.g., “5 sources reported this”) as a consensus signal.

Tech stack

Next.js 16 on Vercel, PostgreSQL + pgvector (HNSW indexing), BullMQ + Redis (ingestion/embedding/prefetch queues) on Railway, Prisma. LangChain wraps model calls; LangSmith traces them and backs the eval scripts. OpenAI: text-embedding-3-small (embeddings), gpt-4o-mini (entity/relationship extraction), gpt-5-nano / gpt-5-mini / gpt-5 / gpt-5-chat-latest (query-complexity routing, and gpt-5-nano as the Tier-2 synthesis fallback). Groq: llama-3.1-8b-instant, llama-4-scout-17b-16e-instruct, llama-3.3-70b-versatile (Tier-1 synthesis and trend clustering).

Results

• Hybrid retrieval fetches 50-200 candidate articles per query at a 70% vector / 30% keyword weighting before cluster boosting and blending.
• HNSW indexing on pgvector is documented in code comments as a 5-20x speedup over a flat similarity scan; not independently re-benchmarked for this write-up.
• The synthesis call has a 3-tier fallback (Groq → OpenAI gpt-5-nano → deterministic template) so a search request never returns empty because a single LLM provider is down.
• Worked example — a query for “AI regulation” retrieves articles indexed under “tech policy laws” via vector similarity, which a keyword-only search would have missed.
• ⚠️ no benchmark yet — end-to-end query latency, retrieval precision/recall, and synthesis factuality have not been measured against a held-out set.

Lessons

1. Fact-First RAG (extracting and grouping Timeline Events before synthesis) was chosen over single-pass RAG because the summarization step is where RAG pipelines hallucinate, not the retrieval step — grounding the Key Insight against a pre-computed, checkable fact layer catches drift a raw-document prompt can’t.
2. Hybrid search at a 70/30 vector-to-keyword weighting was chosen over pure vector similarity because embedding search alone lets exact-match entities (e.g., “Sam Altman”) get outranked by conceptually similar but less relevant articles; keyword search recovers the exact-match case vector search is weakest at.
3. A rule-based query router (ComplexityDetector) was chosen over classifying every query with an LLM because most user queries are simple keyword lookups — routing only complex/analytical queries to a larger model avoids paying for an LLM call on trivial searches.