Enterprise RAG: How to Build Retrieval-Augmented Generation That Works in Production

Before selecting a vector database or embedding model, define the knowledge the application needs.

Create a source inventory.

The inventory determines ingestion, freshness, permissions, and retrieval design.

A PDF is not a sequence of clean paragraphs.

It can contain headers, footers, tables, multi-column layouts, footnotes, images, captions, and scanned pages.

A parser that flattens a pricing table into meaningless text can destroy the evidence before retrieval begins.

Evaluate parsing on representative source types.

Questions include:

• Are headings preserved?
• Are table rows understandable?
• Are page references retained?
• Are lists preserved?
• Are repeated headers removed?
• Are document dates and versions captured?
• Are images or diagrams relevant to the use case?

Chunks should preserve enough local context to answer likely questions.

Very small chunks can lose relationships. Very large chunks can reduce retrieval precision and consume context.

There is no universal chunk size.

A contract clause, API reference, support article, product catalog, and research report have different structures.

Structure-aware chunking gives the retrieval system more useful units than blind character windows.

Useful metadata can include:

• source ID;
• document title;
• section;
• owner;
• creation date;
• last updated date;
• product;
• region;
• customer;
• document type;
• version;
• permission group;
• validity period.

Metadata supports filtering, freshness, debugging, and citations.

Enterprise search benefits from combining methods.

A user asking about invoice INV-49328 needs exact matching. A user asking which policy applies to an unusual refund case needs semantic retrieval. A production system can use both.

Permission filtering belongs inside the retrieval path.

The agent should not retrieve documents the current user cannot access and rely on the model to keep them secret.

A useful path is:

identity
  ↓
authorized query
  ↓
candidate retrieval
  ↓
permission filter
  ↓
rerank
  ↓
evidence

This is especially important when a shared index contains information from several departments, customers, regions, or security classes.

Initial retrieval aims for recall: find a useful candidate set.

Reranking aims for precision: put the strongest evidence near the top.

This separation lets a fast method gather candidates and a more precise one order them.

A reranker can score candidates against the actual query after the initial search.

More context is not automatically better.

The system should send the model evidence that is relevant, current, authorized, and sufficiently complete.

Excess context adds cost and can dilute the strongest evidence.

Anthropic's work on context engineering describes context as a finite resource and argues for curating high-signal information during inference.

Citations should preserve a path back to the underlying source.

A useful citation object can include:

{
  "source_id": "policy-2026-04",
  "title": "Refund Policy",
  "section": "Enterprise Accounts",
  "updated_at": "2026-04-12",
  "passage_id": "p-184"
}

The model can then attach source references to claims that depend on retrieved material.

Create test queries with known relevant evidence.

For each query, record the document or passage that should be retrieved.

Then measure:

Only after retrieval works should final-answer evaluation carry much weight.

Failure A: evidence is absent from the corpus

Fix ingestion or source coverage.

Failure B: evidence exists but retrieval misses it

Fix query construction, indexing, filters, retrieval, or ranking.

Failure C: evidence is retrieved but the answer is wrong

Fix context assembly, instructions, model choice, or answer evaluation.

Failure D: evidence and answer are correct

Preserve the case as regression coverage.

Suppose an employee asks:

The system needs several kinds of retrieval.

1. 1Exact entity lookup for XR-440.
2. 2Customer lookup for Acme.
3. 3Region filter for Germany.
4. 4Date filter for the sale.
5. 5Semantic retrieval over warranty language.
6. 6Current-policy check.
7. 7Permission check for account records.

Pure vector similarity is an incomplete architecture for this question.

Changing information needs a freshness policy.

Possible mechanisms include:

• event-driven updates;
• scheduled synchronization;
• document versioning;
• validity dates;
• source priority;
• stale-document suppression;
• deletion propagation.

A RAG system that retrieves a superseded policy accurately is still wrong.

An agent can use retrieval as a tool and pull context as it works.

Anthropic's context-engineering guidance describes "just in time" retrieval, where an agent maintains lightweight references and loads information as needed during execution.

That approach is useful for long-running work because it avoids filling the context window with information that never becomes relevant.

Charts based on survey findings identify their source in the caption. Mathematical examples are labeled as illustrations. Architecture diagrams and decision frameworks are original explanatory assets created for this post.