Agentic RAG that stays current when the corpus never stops moving

The demo is easy: chunk a folder, embed it, ask a question, get a cited answer. The production problem is different. Enterprise knowledge is not a snapshot. Contracts get revised, runbooks drift, tickets close, wikis fork. If retrieval is a one-shot index, the agent is confidently answering last quarter.

Agentic RAG is not “an LLM with a vector store.” It is a retrieval system that can plan, tool-call, and verify — sitting on a data plane that refreshes at the speed of the organisation. The hard part is not the chat handler. It is keeping the index honest at terabyte scale.

Split the jobs you can fail independently

Treat the path as five stages you can observe, replay, and evaluate on their own:

1. Ingestion — parse, clean, reject garbage, attach provenance
2. Indexing — embeddings plus the metadata filters you will actually query
3. Retrieval — recall first, then precision
4. Generation — structured output, citations, tool calls
5. Evaluation — offline golden sets and online traces

If a bad answer lands and you cannot say which stage failed, you are debugging folklore. Agents make this worse: they will retry the wrong retrieval with more confidence.

Provenance is not optional metadata

Every chunk needs a stable identity: source system, document id, version, page or heading path, ingested_at, and a checksum of the raw bytes. Without that:

• citations become theatre
• refresh cannot do incremental deletes
• you cannot prove what the model saw

Store the raw document and the chunk as two records. The vector row is a projection. When legal asks “what did the agent use on Tuesday?”, you reconstruct from provenance, not from a prompt log that already dropped the context window.

Never let the model invent identifiers. Tool schemas return the ids you already have. Generation may quote; it may not mint a doc_id.

Chunking is a retrieval decision, not a tokenizer trick

Fixed 512-token windows are a starting point, not a strategy. I size chunks around how a human would look the fact up:

• policies and contracts — section-aware, keep headings in the chunk
• code and runbooks — function or procedure boundaries
• tables — do not split a row from its header
• slides — one slide, plus the deck title

Overlap helps recall; it also duplicates cost. Prefer a parent document you can expand after a child hit (small-to-big) over blindly overlapping every window. Attach the heading path to every child so hybrid search can filter source=policy AND product=payments before you spend rerank budget.

Refresh is a pipeline, not a cron that “re-embeds everything”

Full reindex is the last resort. At TB scale it is also how you burn money and freeze retrieval for hours.

A refresh job should be:

• Change-driven — object events, webhooks, or a watermark on updated_at
• Idempotent — same checksum means skip embed
• Deletable — source gone ⇒ chunks gone, not “soft stale forever”
• Bounded — max documents per tick, dead-letter for poison files
• Observable — lag from source commit to searchable chunk

async def refresh_document(doc: SourceDocument, index: VectorIndex) -> None:
    if await index.checksum_matches(doc.id, doc.checksum):
        return
    chunks = chunk_with_provenance(doc)
    await index.replace_document(doc.id, chunks)

Replace-by-document-id, not “insert more vectors.” Orphan embeddings are silent wrong answers.

Continuous refresh also means cache invalidation. If you cache retrieval or final answers, the cache key must include a corpus generation or per-document version. Otherwise the fastest path is the stale one.

Hybrid search, then rerank — not “more embedding dimensions”

Dense embeddings are weak on identifiers, error codes, and exact clause numbers. Sparse or keyword search is weak on paraphrase. Use both, then a cross-encoder or a small reranker on the union.

A practical default:

1. metadata prefilter (tenant, product, doc type, time window)
2. hybrid retrieve (dense + sparse), k large enough for recall
3. rerank to a budget the generator can actually attend to (often 6–12 chunks)
4. refuse to answer if top score is below a threshold — “I don’t have this” is a product feature

Agents should be allowed a second retrieve with a rewritten query. They should not be allowed a tenth. Cap tool iterations. Log every retrieve: query, filters, hit ids, scores. That log is how you improve chunking, not another prompt adjective.

Multi-tenant knowledge is a filter, not a vibe

Enterprise RAG is usually multi-tenant. Isolation belongs in the query planner and the index, not in the system prompt. If a filter can be omitted, it will be omitted under load or under a clever jailbreak.

• partition or strictly filter by tenant at the store
• never take tenant id from the model
• evaluate leakage with cross-tenant probes in the golden set

This is identity work wearing an AI costume. Treat it that way.

Evaluate retrieval before you evaluate prose

A fluent wrong answer is a retrieval bug until proven otherwise. Score the stages separately:

Ship a golden set before you ship the chatbot. Rerun it on every prompt, chunker, or embedding-model change. Online, sample traces — not just thumbs-up. Offline metrics without traces is how you overfit a spreadsheet.

Cost and latency are architecture

Embedding a terabyte is a budget line. So is reranking every query with a large cross-encoder.

Budget from the outside in: p50 for a calm user, p95 for support tickets, timeouts the gateway actually enforces. Cache embeddings. Cache retrieval for identical normalised queries. Do not cache final answers across a refresh generation.

A smaller model with honest retrieval and a tight context will beat a 70B model stuffed with twenty mediocre chunks. Agentic RAG earns its keep when the agent fetches less, better — not when it talks more.

The goal is not a clever retriever. The goal is an index you can explain, refresh, and distrust on purpose — so the agent still behaves at 3 a.m. when the wiki moved and the contract did not.