How escalation works

A page enters at json-ld.

1. JSON-LD extracts publisher-provided structured article data: the schema.org Article.articleBody field, when the publisher includes one. It is the cleanest source when present: no heuristics, no boilerplate to strip.
2. Readability applies Mozilla's heuristic content-extraction algorithm to the raw HTML. It is the same engine behind Firefox's reader view.
3. Cheerio makes a selector-based structural attempt against that same HTML, using a fixed list of likely content selectors. It is the fallback for pages where Readability's heuristics find nothing.
4. Browser launches headless Chrome and retries extraction against the rendered DOM. It exists specifically for sources gated on browser fingerprint or client-side rendering. A plain fetch either gets blocked as a bot, or returns a shell whose article body JavaScript injects afterward. It runs only on the Railway worker; Vercel never launches it. Known limit: this recovers client-rendered pages, not every blocked page. Sources gated by an anti-bot wall or a paywall are out of scope for this tier. They stay out of scope after it runs.
5. Docparse hands the page to a Railway-hosted document-conversion service for formats HTML extraction cannot parse, principally PDFs. Its primary use case is arXiv papers whose HTML rendering path produced nothing. Most often the paper is too new, or its LaTeX too unusual to render.
6. LLM is the final fallback: a paid model call that attempts extraction when every structural and heuristic approach has failed.

At each tier, the candidate must pass the article gate. If it passes, extraction stops. If it fails, the page escalates to the next tier.

for extractor in [json_ld, readability, cheerio, browser, docparse, llm]:
            candidate = extractor.extract(page)
            if article_gate.passes(candidate):
                return candidate
        return ExtractionFailure(page.url)

The gate is the economic control point It limits expensive repair mechanisms to pages that actually need them.

What the design trades off

Chain cost is driven less by total corpus size than by the escalation rate:

A clean, well-formed source list should terminate mostly in the cheap tiers. A source list with many client-rendered, malformed, or non-HTML pages will escalate more often. That shifts latency and cost toward browser, document parsing, and LLM extraction.

Deployment levels

Tiers 1–3: cheap-only json-ld → readability → cheerio

Tiers 1–4: browser fallback adds browser

Tiers 1–6: full chain Live

Cost and latency: adds document parsing and, as a last resort, a paid model call. Capability: extends coverage to non-HTML documents and pages too malformed or unusual for deterministic extractors. Risk: LLM spend scales with the rate of pages that defeat every earlier tier. A noisy or adversarial source list can make this tier the dominant extraction cost. Quality: the LLM tier increases recovery, but its output still requires validation. A model-produced extraction is not automatically trustworthy merely because deterministic extraction failed.

Why gate escalation

The cheap tiers handle most well-formed news pages. Launching a browser or calling an LLM for every page would turn a rare recovery path into a universal tax.

Gating changes the role of expensive tiers:

Without a gate: expensive processing is the default.
        With a gate:    expensive processing is targeted repair.

The architecture follows a general systems principle: defer advanced, expensive behavior until the specific case requires it, rather than exposing all complexity on every request. This is the same reasoning Nielsen Norman Group documents for interface design as progressive disclosure.

What a new tier has to bring

Every tier beyond cheerio should have:

• a defined page class it recovers
• a measurable success rate after escalation
• a bounded latency and cost profile
• per-source instrumentation showing why pages reach it
• a validation rule for its output

If LLM-tier spend rises, inspect which sources and failure modes are causing escalation first. The likely fix is a cleaner source list, a source-specific extractor, better document classification, or a stronger article gate, not simply a cheaper model.

log1p compresses the reward for length so quality still decides close calls

The formula multiplies a quality signal by log1p(textLength) rather than by raw length. A 4,000-word page scores roughly 1.2× a 1,000-word page under log1p, not 4×.

Length still matters, since a longer article generally does carry more signal. But its influence flattens out fast. It functions as a modest tiebreaker rather than the dominant term.

The counterfactual makes the design choice legible. Suppose the formula used raw textLength instead of log1p(textLength). A candidate padded with boilerplate would then systematically outscore a shorter, cleanly extracted candidate, purely because it has more characters.

That padding is real: repeated navigation text, related-article link lists, comment sections dragged in by an imprecise extractor.

Raw length would reward exactly the failure mode the tier chain and article gate are trying to filter out. That failure mode is extractors that grab everything on the page rather than the article itself. log1p keeps length as a real but bounded input, so it can't overwhelm the quality term.

The gate is the filter; the score is the tiebreaker

Selection runs in two passes over the candidates for a page:

1. Gate pass: every candidate is checked against the six hard-reject thresholds (title length, word count, paragraph count, list:paragraph ratio, caps ratio, keyword stuffing). Any candidate failing even one threshold is dropped, independent of what its score would have been.
2. Score pass: among the candidates that survived the gate, quality × log1p(textLength) ranks them, and the highest scorer is stored as the article.

This ordering makes gate-pass precedence a real guarantee rather than a tendency. A failing candidate cannot win, because failing candidates are removed before scoring ever happens.

Take a page that produces one candidate failing the gate that would have scored very high, and another passing the gate with a modest score. The modest passer wins outright. There is no threshold at which a high enough score overrides a gate failure.

Selection score design comparison

Why not score everything and take the top result

A single ranking without a hard gate would let quality issues that the gate is specifically designed to catch, navigation-menu candidates, keyword-stuffed pages, compete purely on score. A sufficiently long or superficially well-formed bad candidate could still win. Splitting filtering from ranking removes that possibility structurally.

Why not gate everything and pick arbitrarily among survivors

Multiple tiers can all produce gate-passing candidates for the same page with real quality differences between them. readability's output and cheerio's output for the same article are rarely identical. An arbitrary pick (e.g. "first tier that passed") would leave quality on the table among candidates the gate has already approved.

When a good candidate loses to a worse one, both passed the gate. That is when the quality or length weighting is the right thing to touch.

Each threshold targets one publisher pathology

No single quality theory produced these six. Each maps onto one way a page can look like an article without being one, or fail to look like one while being one.

Each threshold decides which pages enter the corpus at all, and nothing rejected here re-enters the pipeline. Admitting more lets that kind of page compete for the top rank on a real query, which no reranker weight downstream can undo. Admitting fewer removes that kind of real article from every later stage permanently. Reach for more when a real article was wrongly rejected, and for fewer when that junk is reaching readers, always against a failure you have actually seen rather than a general feeling.

The list-to-paragraph ratio carries the most weight

Losing this one threshold would admit the highest-volume category of obvious junk the gate exists to stop.

• Navigation menus, category pages, and link hubs are the most common non-article shapes a crawler meets.
• They share one signature: many list items, few real paragraphs. A ratio ceiling reads that signature directly.
• No other threshold sees it. Word count and title length cannot tell a long nav menu from a long article, and caps ratio and keyword stuffing judge quality inside paragraphs rather than the shape of the page.

Independent filters compose differently than a score

The six thresholds are ANDed, not summed. A page must clear every one rather than accumulate enough combined strength across several.

That changes how a publisher fares. Take a publisher whose pages sit right at the word-count floor and right at the caps-ratio ceiling. Both checks have to clear on the same page, so its rejection rate runs meaningfully higher than the two individual pass-rates would predict.

An additive score behaves the opposite way. It lets strength on one signal pay for weakness on another, which is why the content quality floor is scored and this gate is not.

The formula rewards metadata completeness, not article quality directly

Seven signals contribute to the score:

• 300+ words contributes +0.3
• a title over 10 characters contributes +0.2
• an excerpt over 50 characters contributes +0.1
• author presence contributes +0.1
• published-date presence contributes +0.1
• Open Graph metadata contributes +0.1
• an Open Graph image contributes +0.1

The maximum achievable score is 1.0, and reaching it requires every single signal to be present simultaneously.

None of the seven components measures whether the article is well-written, accurate, or even coherent. The score is a proxy for "this page has the shape of professionally published content." It is built entirely from signals a real content-management system tends to emit and a scraped aggregator repost tends to drop.

Length and title strength carry the largest weights, +0.3 and +0.2, because those are the hardest signals for a low-effort page to fake. The four remaining signals are each worth only +0.1 and are individually replaceable. Together they distinguish a page with real publishing infrastructure behind it from one without.

0.0 – 0.3

Only the structural article-gate thresholds apply; the quality score adds essentially no additional filtering. Capability: maximum source coverage, since almost any page that clears the gate is admitted regardless of metadata. Quality: metadata-poor aggregator reposts dominate the corpus. Downstream entity extraction produces attribution with no authors and no dates to anchor it, which weakens every claim's traceability.

0.4 – 0.6 Live 0.5

Requires roughly "long enough, plus a real title, plus one further metadata signal." This is the balance point: a page has to look like a real article by more than a single measure. A page can't pass purely on length with zero attribution, or purely on having a title with almost no content.

0.7 – 0.9

Demands nearly complete metadata, most of the seven signals present simultaneously. Quality: resulting corpus is very high in metadata completeness. Capability: excludes most independent blogs and any publisher not emitting Open Graph tags. That is a large fraction of technical primary sources, including personal blogs and some academic sites that publish real, citable content without a modern CMS's metadata conventions.

1.0

Unsatisfiable in practice: the maximum achievable score is exactly 1.0, and reaching it requires every one of the seven signals present on the same page with no slack. Effect: the corpus goes empty, because no realistic floor should ever be set here.

Measure a proposed floor before committing it

Raising or lowering CONTENT_QUALITY_FLOOR in production changes what gets rejected immediately and irreversibly for that page. A rejected page isn't retried later at a different floor unless it's re-crawled.

ENFORCE_CONTENT_QUALITY_GATE=false exists specifically to break that irreversibility. It runs the same scoring logic and logs the score for every page, but rejects nothing.

That makes it possible to compute what a candidate floor value would have done to real, current traffic: how many pages would be rejected, and which publishers they'd come from, before that floor is ever live.

Operational pattern Test a new floor value in log-only mode first, review which pages and publishers it would have rejected, and only then move the enforced value. This converts a "did that break coverage" question, answerable only after the fact, into a "will that break coverage" question, answerable before the change ships.

When to raise and when to lower

• Raise it when metadata-poor aggregator noise is visibly diluting the corpus.
• Lower it when a specific class of legitimate source is being excluded: an independent blog, an academic site without Open Graph tags.

Set the floor to the point where a page has to be a real article by more than one measure, not just one.

Splitting breaks cross-section context; the cap exists to avoid needing to split

When a long investigative piece exceeds the block cap, it gets split into multiple blocks and processed separately. Below roughly 20,000 characters, this becomes a real problem: the model extracting facts from a later block has lost the context established earlier in the piece.

A fact like "the company reported the delay" can end up in a block that never introduced which company. The antecedent is in an earlier block the model isn't seeing. The extraction can still run, but individual facts start losing their subject.

Below ~20,000 chars

Long investigative pieces get split into multiple blocks, and the model loses cross-section context between them. Quality: mid-article facts lose their subject, since the antecedent that would resolve a reference like "the company" can fall in a different block than the reference itself.

80,000 chars Live

Covers essentially every real news article in a single pass, so cross-section context is preserved for the pieces this tier is actually meant to rescue. Cost: the worst case is spending a full 80,000-character prompt on a junk page that every cheap tier has already failed on. Because this tier only runs after the cheap tiers have already rejected the page, that worst case is not hypothetical. It is what happens on every page that reaches tier 6 and still fails to extract cleanly.

Above ~120,000 chars

Approaches context-window limits on smaller models like the one this tier uses, and raises tail latency, without recovering meaningfully more real articles. Real news articles are essentially never this long, so the extra headroom has no corresponding upside in coverage.

Why headroom above the cap buys nothing

Tier 6 only ever runs on pages that every cheaper tier has already rejected. A larger cap mainly increases the worst-case cost of processing a junk page, rather than the coverage of real articles.

Extra headroom above the longest real article buys no additional coverage, only additional worst-case cost.

Why the boundary sits at 800 tokens (roughly 3–5 paragraphs)

128 – 256 · sentence level

400 – 600 · paragraph level

800 Live

Roughly 3–5 paragraphs. One chunk usually contains a claim plus its attribution and its qualifier, which is the minimum unit a grounded answer needs. Accuracy: slightly diluted on narrow factoid queries versus 256.

1,200 – 2,048 · section level

How much duplication is worth paying for

0%

No duplication. Quality: any claim spanning a cut is unretrievable by either chunk, because neither contains the whole statement. This is the most common silent RAG defect.

10 – 15% Live 12.5%

A boundary-spanning sentence survives in at least one chunk. Cost ~12% more storage and embedding spend.

25%+

Accuracy: near-duplicate chunks crowd the top-K, so a 10-result page returns 4 distinct passages. Cost rises proportionally with no recall gain.

Critical companion setting Overlap tokens are excluded from the pooling weight (coreTokenCount), so duplicated text cannot double-count when chunk vectors pool into the article vector. Without that exclusion, raising overlap silently biases the article-level embedding toward whatever text sits on chunk boundaries.

What each safeguard actually enforces

What dimensionality changes at each tier

384 – 768 dims

Speed 2–4× faster search, 2–4× less storage. Accuracy measurably lower on nuanced semantic distinctions.

1,536 Live

Cost roughly 1/6th of text-embedding-3-large. Sufficient for topical news retrieval, where queries are entity- and topic-shaped rather than subtle.

3,072

Accuracy higher on fine distinctions. Cost 2× storage, 2× index RAM, ~6.5× embedding spend, and slower HNSW traversal at every layer.

Capability boundary The dimension is fixed at table creation. Changing it is a full table rebuild plus a full re-embed of the corpus, so treat it as an architecture decision, not a tuning knob.

What happens with and without the stamp

1. Without a stamp a model swap goes untracked. Every health metric stays green because the new vectors are the right shape. Retrieval quality degrades anyway, because they occupy the wrong space. There is no automatic signal that anything changed.
2. With a stamp a version bump marks every existing row stale. The backfill script reads that stale flag and re-embeds those rows automatically. Capability: this is what makes an embedding-model migration a routine operation rather than a corpus-wide outage.

How the pooled vector is built

Chunk vectors are L2-normalized, weighted by core-token count, summed, then re-normalized into one article vector.

Why 800 tokens (the retrieval size) breaks extraction

Why not reuse 800

Extraction needs to see "Company A acquired Company B" and "the deal closed Tuesday" in the same window to emit one relationship with a date. At 800 tokens those land in different blocks and the relation is never formed.

1,000 – 4,000

Quality cross-paragraph relations degrade. Cost more model calls per article.

8,000 Live

One call covers essentially any news article, so the model sees the whole story.

10,000+

Approaches model context limits, and attention over long inputs degrades before the limit is reached. Accuracy mid-document entities get missed.

Design principle worth copying One chunker, two configurations. Retrieval optimizes for precision, extraction optimizes for context, and neither compromises for the other.

Model size: what the extra capacity buys

Determinism and output-validity settings

temperature: 0

Deterministic. The same article always yields the same graph, so a re-run is a repair rather than a second opinion. At 0.2+, re-extraction produces drift indistinguishable from real corpus change.

max_tokens: 1500

Caps entities-plus-relations per block. Raising it recovers long-tail entities from dense articles; lowering it truncates JSON mid-object and fails the schema.

Strict json_schema vs. json_object

Strict mode rejects malformed output at the provider rather than at your parser, which eliminates the retry-on-parse-failure loop entirely for models that support it.

Why the two thresholds are asymmetric

0.3 – 0.5

0.6 – 0.7 Live

The relationship bar sits lower than the entity bar deliberately. An orphan entity pollutes the entity namespace permanently, while a weak relation is cheaper to drop later.

0.85+

How a quote gets verified against the source

1. No grounding requirement. The model asserts, you store. Unfalsifiable claims accumulate with no way to audit them.
2. Fuzzy or semantic matching (rejected). Quality worse than no grounding, because a paraphrase "resolves" and the resulting span looks verified while pointing at text that does not say it.
3. Exact match, then whitespace-normalized fallback Live. Whitespace normalization handles real formatting variance (newlines, non-breaking spaces) without admitting paraphrase.
4. Sliced from the original text, never the model's echo. Closes the remaining gap: a model can otherwise return a subtly altered quote that still passes a match against its own output.

What grounding buys Capability every claim carries block_id, char_start, char_end, so any answer is traceable to a character offset in the source.

What each retry policy actually does under provider failure

No retry

A provider rate-limit spike silently drops articles from the graph, and the gap is invisible because nothing errored.

3 attempts / 30s cap Live

Survives normal rate-limit turbulence. Worst-case per-article latency is bounded, so a backlog drains predictably. Honors the provider's Retry-After header.

Unbounded retry

A sustained provider outage converts into an unbounded queue backlog.

Matching the operator to how the embedding model normalizes vectors

OpenAI documents text-embedding-3-* as cosine-normalized, which is why vector_cosine_ops is correct here. It compares angle only and ignores magnitude, so it doesn't matter that normalization already stripped magnitude information out. The other two operator classes each assume something about magnitude that may not hold.

Why higher-dimensional vectors get a higher m

Index memory grows roughly linearly with m, and so does build time and insert latency. m is a cost paid continuously, not once.

At 1,536 dimensions the article index needs more edges per node to keep the graph navigable than the chunk index does. That is why it runs 24 against the chunk index's default of 16.

2 – 8

16 Live · default

The documented balance point. Index roughly 1.2–1.5× the raw vector size.

24 – 32 Live · article index

48 – 100

Why a bad build can't be tuned away later

Every future query pays for whatever graph quality ef_construction bought at build time. That makes it the one parameter here where the mistake to avoid is going too low, not too high.

Going too high has a real cost too: a large corpus can take hours to build with the ingest pipeline stalled underneath it. That cost is bounded and known in advance, unlike a permanently damaged graph.

4 – 32

64 Live · default

Adequate for corpora in the tens of thousands.

100 – 200 Live · article index

400 – 1000

Trading recall for latency at query time, per transaction if needed

Because ef_search is a runtime setting, one query class can run high-recall while everything else stays fast. That is not true of m or ef_construction.

That flexibility is what makes it the right first move whenever a recall complaint shows up, before touching either of the build-time parameters.

1 – 20

40 · pgvector default

100 Live

Accuracy notably better recall. Speed roughly 1.5–2.5× the latency of 40, typically single-digit milliseconds at this corpus size. UX the extra latency is invisible next to the LLM synthesis call that follows.

200 – 500

1000

Sitting just above the near-duplicate band on purpose

0.75 is deliberately close to the near-duplicate threshold used elsewhere in the pipeline: 0.7, the dedup constant. A cluster therefore reads as genuinely related coverage rather than a loose topical grouping.

Moving the threshold in either direction changes what a cluster means, not just how many articles land in one.

What the floor changes

Hold the corpus and the query fixed and move only the floor. The corpus never changes. What changes is which question the index can answer without misleading the reader.

Score bands for this model on this corpus

These bands describe OpenAI text-embedding-3-small over news articles. They describe one system.

Why the floor sits at the bottom of its band

One bound is recorded and the other is not, which is worth saying plainly on a page about how well numbers are known.

Off-topic queries top out around 0.08 to 0.09 on this corpus. That sets the lowest defensible floor. Nothing records where fair questions begin coming back empty, so there is no measured upper bound: 0.40 is the label on the next band, not an observed cutoff.

What can be said is that 0.25 is the lowest value clearing the noise. Raising a floor trades recall for precision, because a reader who gets "no results" for a fair question decides the search is broken.

How well this is known The source comment says empirical tests returned 0.08 to 0.09, so it claims a run. No fixture, eval or log in the repository carries that number. No sample size, corpus snapshot or date sits behind it. Treat it as the author's calibration note, and re-measure before relying on it.

The constant kept deliberately separate

SEMANTIC_SIMILARITY_NEAR_DUPLICATE = 0.7 belongs to the dedup pipeline alone. Holding it apart from the search floor prevents one specific failure: someone tightens duplicate detection, a single shared number moves, and search quietly returns nothing while every component still reports healthy.

What carries to another system

The principle carries. A retrieval system needs an absolute relevance gate, because ranking alone cannot say that every candidate is wrong.

The method carries. Derive the gate from the top score of queries you know should fail, and record that ceiling next to the floor so the number stays re-derivable.

The band does not carry either, and the reason matters. The source comment credits OpenAI's embeddings guidance with putting useful relatedness near 0.2 to 0.3. That guidance says no such thing. It recommends cosine similarity and adds that the choice of distance function rarely matters, and it publishes no threshold at all. No vendor number stands behind this band. Checked against OpenAI's embeddings guide on 2026-08-24.

The exact value does not carry. 0.25 describes this chunking and this corpus. Cosine scores are not comparable across embedding models. Across distance measures the difference is scale, not ranking: these embeddings are unit length, so cosine, dot product and Euclidean distance order results identically. A floor of 0.25 in cosine is simply not the number 0.25 in another measure.

Why keyword-leaning defaults are the safer failure mode early on

Users tolerate empty results better than confidently wrong ones. That is the real argument for keyword-leaning defaults early in a system's life.

The live default, hybrid-weighted, sits at 0.6 vector / 0.4 keyword for that reason. It rarely fails badly and rarely wins decisively, a safer place to sit than either extreme.

Timeout scaling

Timeouts track expected work: conceptual-deep gets 15s because deep vector traversal is slow; keyword-primary gets 5s because full-text search that hasn't returned in 5s won't return usefully.

entityBoost · 0.05 – 0.40

Additive bonus for query-entity matches. Above ~0.5 it dominates semantic relevance entirely and the system becomes an entity lookup.

temporalBoost · 0.05 – 0.50

Recency weight. At 0.5 a two-day-old article beats a materially better week-old one: correct for "what happened today," wrong for "how did X evolve."

Why a healthy-looking system can still return nothing without this

A partially backfilled index doesn't fail any health check. The vectors that exist are fine, there just aren't enough of them yet.

That is precisely the condition conceptual-deep's 0.9 vector weight is blind to: it has no fallback of its own. The degradation blend supplies one, interpolating toward keyword-primary weighting as the live vector-quality score falls.

Without it

A sparse or partially-backfilled index makes conceptual-deep's 0.9 vector weight starve recall completely, and the system returns nothing while every component reports healthy.

With it Live

Degradation is graceful: as vector quality falls, keyword carries more load. Capability the system stays usable during an embedding backfill instead of going dark.

Sizing the pool to what the reranker, not the fetch, can use

The fetch scales cheaply as the cap rises, so the real cost curve belongs to the reranker, not the database. That is why 100–200 sits where it does. It is the smallest pool that gives the reranker genuine material to re-rank, not the largest pool retrieval can cheaply return.

10 – 25

Speed fastest. Quality the reranker has too small a pool to improve on, so reranking stops paying for itself.

100 – 200 Live

Gives the reranker real material. Retrieval cost rises sub-linearly; the reranker is the bottleneck, not the fetch.

500+

Payload size and reranker token cost dominate with no measurable quality gain.

topK, plus four supporting decisions that keep the reranker honest

40 is deep enough to give a badly-ranked-but-correct result a second chance, at the cost of 40 title-plus-snippet pairs scored per query.

The rest of this setting is four smaller decisions that all serve the same goal. They make sure the highest-volume, highest-visibility LLM call in the system is fast, cheap, stable, and genuinely optional when it isn't needed.

topK 10 – 20

Speed fast, cheap. Quality the genuinely best result frequently sits at rank 25 and is never seen.

topK 40 Live

Deep enough that a badly-ranked-but-correct result gets a second chance. Costs 40 title-plus-snippet pairs per query.

topK 80+

Prompt size and latency grow linearly; gains flatten because vector rank below ~50 is mostly noise.

Skip-below-5 rule

LLM overhead exceeds any possible reordering benefit on 4 items, so narrow queries return without a model call at all.

Model choice · 20B not 120B

Relevance scoring is bounded and well-anchored, so the 20B model matches the 120B here. Cost this is the highest-volume LLM call in the system, so right-sizing it matters more than anywhere else.

temperature: 0

The same result set always ranks the same way. UX refreshing a page does not reshuffle results, which users read as instability.

Two knobs, two different failure modes if pushed too far

Push the blend toward cluster weight and heavily-covered stories start dominating regardless of query fit, trading "relevant" for "popular."

The cache has a different failure mode entirely. It isn't about the ratio, it's about staleness. A fixed TTL is safe only as long as nothing changes underneath it, which is why an embedding-model version bump has to invalidate it.

Blend 1.0 / 0.0

Pure semantic ranking; loses the "this story is being widely covered" signal.

Blend 0.7 / 0.3 Live

Cluster membership supplies mild corroboration weight.

Blend 0.5 / 0.5

Quality heavily-covered stories dominate regardless of query fit, pushing the system toward "popular" and away from "relevant."

Query-embedding cache · 30 days, 10,000 entries

Embeddings for a fixed model are deterministic, so a long TTL is free correctness-wise and removes 100–200 ms from every repeat query. The 10,000 cap is what makes the cache safe under a novel-query flood.

Truncation depth constrains coverage more than article count

Raising the per-article character limit recovers more real coverage than raising the article count does. Most articles put their most specific fact past the lede: a statistic, a quote, or a caveat that sits in paragraph four or six.

At an 800-character cap, that fact is cut before the model ever sees it. Adding more 800-character articles instead adds more truncated ledes, which mostly repeats the same shallow view from additional sources rather than deepening any one of them.

10 articles × 500 chars

Speed fastest synthesis, lowest cost. Quality single-source answers that miss contradicting coverage. Users spot the omission immediately when they know the story.

50 × 800 Live

Enough breadth to represent disagreement across outlets. Prompt assembly plus generation dominates total response time.

100 × 2,000

Quality better coverage on broad questions. Speed latency and cost roughly 5×. Accuracy long-context attention degrades, so mid-list articles get under-weighted. More evidence does not monotonically improve answers.

The sharper constraint is the 800-character truncation rather than the 50-article count, because an article whose key fact sits in paragraph six contributes only its lede, so raising per-article characters buys depth from sources already retrieved while raising the count buys more truncated ledes.

Only the 0.25 threshold changes what actually runs

Where the 0.25 threshold sits is the only score value that changes behavior right now. A simple query like "latest AI news" scores near 0 and stays on the cheap model.

Lowering the threshold to roughly 0.1 sends nearly everything to the expensive tier, which buys little quality on queries that were never complex.

Raising it to roughly 0.4 sends genuine comparisons to the weak tier instead. That's the failure users actually notice: the model gets a question it can't answer well.

Seven signals add into the complexity score, and 0.25 is the line between the two models. Four of them clear that line on a single occurrence, so one strong signal is often the entire routing decision.

• At or above 0.25 on one occurrence: analyze +0.40, compare +0.35, trend +0.30, rank +0.25.
• Below 0.25 on one occurrence: velocity +0.20, entities +0.10 each to a 0.25 cap, domains +0.05 each to a 0.15 cap.

Threshold at 0.25 Live

A simple "latest AI news" query scores near 0 and routes to the cheap model. Lowering the threshold to roughly 0.1 sends nearly everything to the expensive tier. Raising it to roughly 0.4 sends genuine comparisons to the weak tier.

Defect in the live config The middle and top tiers resolve to the same model today, so the three-way split is a routing signal with only two real destinations. A from-scratch build should either wire a genuinely larger top tier or collapse to two tiers and stop paying the complexity-scoring cost for a distinction that doesn't exist.

Fabrication rate rises once temperature passes the 0.1 – 0.2 band

Any temperature meaningfully above the 0.1 – 0.2 band measurably raises how often the model states a detail the source material doesn't support. Grounded prompts constrain the model less effectively as temperature rises.

The same "only use the supplied evidence" instruction that reliably holds at 0.1 stops holding at 0.4. Two users asking the same question at that higher setting can get different facts back, not just different phrasing.

0 Live · extraction, rerank, routing

Fully deterministic. Correct for anything structured: the same input must produce the same JSON.

0.1 – 0.2 Live · prose

Enough variation for readable narrative without inventing detail. The practical ceiling for grounded summarization.

0.4+

Accuracy fabrication rate rises measurably and grounded prompts stop reliably constraining output. UX two users asking the same question get different facts.

A named escape value for missing data prevents fabrication that phrasing can't

A named escape value for missing data prevents fabrication that instruction phrasing alone cannot catch. The prompt hard-bounds the model: every numeric value must come from the supplied articles. Missing data sets dataQuality: "insufficient" rather than filling the gap.

Without that explicit escape, the model fabricates to satisfy the requested output shape, because "return a chart" is a stronger instruction than "be accurate." This single affordance does more for grounding than any amount of "do not hallucinate" phrasing.

Cache invalidation and rate limiting

Cache invalidation works because GROUNDING_CACHE_VERSION and EVIDENCE_PRIORITIZATION_VERSION are part of the cache key. Capability prompt iteration becomes a deployable change rather than a wait, instead of something that quietly does nothing until old answers expire on their own.

Rate limiting runs on INTELLIGENT_SEARCH_LIMIT enforced in Redis rather than in-process, because in-process limiting doesn't survive serverless scale-out. Novel-query traffic can't be cached by definition, so without a limit it spends uncapped on paid LLM calls. That's the specific failure mode that turns a traffic spike into a bill.