Enterprise RAG: Architecture, Security, and How to Evaluate It

Most RAG demos work. That's the problem.

A prototype that answers questions over a folder of PDFs takes an afternoon. Point the same architecture at a company's real document estate and it fails in ways the demo never showed you. It answers from a document the person asking wasn't cleared to see. It cites a policy that was superseded in March. It produces a confident, fluent answer assembled from two chunks that have nothing to do with each other.

Enterprise RAG is the set of decisions that separate those two outcomes. The retrieval mechanics are identical. What changes is everything around them: who is allowed to retrieve what, how the corpus stays current, what the system does when retrieval misses, and how anyone finds out that it did.

This guide covers what changes at production scale, where retrieval actually breaks, how to handle security and tenant isolation, and what to evaluate when you're comparing platforms.

What Makes RAG "Enterprise" RAG

Retrieval-augmented generation was introduced in a 2020 paper by Patrick Lewis and colleagues at Facebook AI Research. The idea is straightforward: instead of relying on what a model memorized during training, fetch relevant documents at query time and give them to the model as context. Large language models stop guessing and start reading.

That definition hasn't changed. What the word "enterprise" adds is a list of constraints the original paper never had to consider.

Permissions. In a prototype, every document is readable by everyone. In a company, a contractor and a VP querying the same agent should get different answers from the same corpus, and the difference has to hold at retrieval time rather than being patched afterward in the prompt.

Freshness. A demo corpus is a snapshot. A real one changes daily, and a policy document that was correct last quarter is now a liability. Somebody has to own the refresh cadence and the removal of superseded content.

Tenancy. If you serve multiple customers from one deployment, tenant A retrieving tenant B's documents isn't a quality problem. It's an incident.

Auditability. When an agent gives a wrong answer, "the model hallucinated" is not a diagnosis. You need to know whether retrieval returned the wrong chunks, returned the right chunks and the model ignored them, or returned nothing at all. Those are three different bugs with three different fixes.

Measurability. Prototypes are judged by whether the demo went well. Production systems need a test set and a number that moves.

The thing is, none of these are retrieval problems. They're operational ones. That's why teams who nail the vector math still ship RAG systems that don't survive contact with their own company.

How Enterprise RAG Architecture Differs From a Prototype

The pipeline looks the same on a whiteboard. Documents get chunked, embedded into vectors, and stored. A query gets embedded too, matched against stored chunks by similarity, and the top matches go to the model along with the question. If you want the mechanics of that matching step in isolation, semantic search is the same retrieval primitive without the generation half.

Here's where a production build diverges.

Ingestion stops being a one-liner. Prototypes read clean text. Real corpora are PDFs with tables, slide decks, wiki pages, ticket histories, and spreadsheets. How you parse them determines the ceiling on everything downstream, and most teams underinvest here because it's the least interesting part.

Retrieval stops being purely semantic. Vector similarity is good at paraphrase and bad at exact terms: product codes, policy numbers, error strings. Production systems usually run keyword retrieval (BM25) alongside vector search and merge the results, then rerank the merged set with a model that scores relevance directly rather than by embedding distance.

The query stops being taken literally. Users ask short, ambiguous, context-dependent questions. Rewriting a query before retrieval, or routing it to the right subset of the corpus first, often moves accuracy more than any change to the embedding model. This is a context engineering problem as much as a retrieval one.

Chunking becomes a tuning parameter with real consequences. Chunk too small and you sever the context that made a passage meaningful. Chunk too large and you dilute the signal and burn tokens. There's no universal answer, which is why the number of chunks you retrieve needs to be adjustable rather than hardcoded.

Frameworks like LangChain and LlamaIndex give you these pieces as components. Managed services on Amazon Bedrock, Vertex AI, Azure, and Snowflake give you more of it assembled, with less control over the seams. Both are legitimate. What neither gives you is the operational layer above: permissions, environments, evaluation, and a way to tell your failure modes apart.

What Retrieval Failure Actually Looks Like

This is the part most RAG content skips, so it's worth looking at a case where someone published their numbers.

Uber runs an internal on-call copilot called Genie that answers engineering questions about security and privacy policy across more than 40 internal documents. In May 2025 the team published what happened when they moved it from a conventional RAG setup to an agentic one, measured against a golden test set of over 100 queries curated by subject matter experts.

The result: a 27% relative increase in acceptable answers and a 60% relative reduction in incorrect advice.

The failure modes they named are more useful than the headline.

Their PDF loaders were destroying tables. Cells were being extracted as isolated, standalone text, disconnected from the rows and columns that gave them meaning. A table of retention periods by data class became a list of unrelated numbers. Switching the source format and adding model-assisted formatting for structured content fixed it at the source.

Semantic similarity was retrieving plausible, wrong context. Where two policies differ in a subtle clause, embedding distance can't tell them apart. Both passages look equally relevant to a vector, and the model confidently answers from the wrong one.

The fix was mostly not the retriever. They added a query optimizer and a source identifier ahead of retrieval, layered BM25 alongside vector search, and added a post-processing step to deduplicate and structure the retrieved context before it reached the model.

Read that list again and notice what isn't on it: a better embedding model. The gains came from parsing, from routing, and from what happened either side of the retrieval call. That's the pattern in most enterprise RAG work, and it's why a platform decision that only optimizes the vector step is solving the wrong tenth of the problem.

RAG Security and Multi-Tenant Isolation

Ask a vendor how their RAG handles permissions and listen for whether the answer happens before or after retrieval.

Filtering after retrieval is the wrong shape. If the system fetches chunks first and then removes the ones the user shouldn't see, those chunks existed in the request path, and every downstream bug becomes a disclosure risk. Filtering has to constrain the search itself.

The substrate for this is metadata. Every chunk carries fields, and queries carry filters that are applied at retrieval. Tag documents by department, clearance, region, or tenant, and pass the querying user's attributes as filter values so the search never sees what it shouldn't return.

There's a sharp edge here worth knowing about in advance, because it bites in production rather than in testing. Filter operators are not all exact-match. In Voiceflow's knowledge base, the agent-facing search tool matches filters with CONTAINS, so filtering on user_id: "alice" also matches alice.smith and charlie.alice. The convention that fixes it is to wrap identity values in a delimiter that never appears inside a real identifier, storing and querying |alice@example.com| so that substring matching and exact matching produce the same result. The REST query path supports a fuller operator grammar ($eq, $ne, $in, $nin, and numeric ranges) when you need it.

That's a small detail. It's also exactly the class of detail that decides whether multi-tenant RAG is safe, and you will not find it in a vendor's overview page.

What to Look For in an Enterprise RAG Platform

The honest version of this list is short, because most platforms converge on the same retrieval quality and diverge on operations.

Can retrieval be filtered at query time, by fields you define? Covered above. This is the one that's hardest to retrofit.

Can you tell a retrieval miss from a grounding miss? When an agent gives a bad answer, you need to know which half failed. A system that returns only a final string makes every failure look identical.

Is content environment-scoped? If staging and production share one corpus, you cannot test a knowledge change without exposing it. Editing documents in a place that is also serving customers is not a workflow, and environments are the mechanism that separates the two.

Can you evaluate it? Uber's numbers exist because they had a golden test set before they made changes. Without one, every RAG improvement is a vibe. The platform should let you run a fixed question set against a version and compare, which is the same discipline as measuring what your agents actually do once they're live.

Can you see what it retrieved? Retrieval traces belong in your observability stack alongside everything else. Tools like Arize exist because a system you can't inspect is a system you can't improve.

Does retrieval compose with actions? Most real questions don't end at an answer. "Where's my order" needs a lookup, not a policy document. A retrieval layer that can't hand off to a tool call caps the agent at answering questions nobody needed answered.

Where Retrieval Sits in an Agent, Not Beside It

RAG used to be the whole architecture. Now it's one step inside a loop.

In a Voiceflow project, the knowledge base is a semantic-search corpus the runtime queries at retrieval time. Content goes in as one of five document types (table, text, URL, file, or integration), gets split into chunks and embedded, and is matched by cosine similarity. An agent reaches it through a search tool, the same way it reaches any other tool, which means retrieval is something the agent decides to do rather than something that happens to every turn. That distinction is the whole of what people mean by agentic RAG.

Three behaviors matter for the operational questions above.

Grounding is reported separately from retrieval. When synthesis runs and the retrieved chunks don't support an answer, the system returns a NOT_FOUND signal that the runtime surfaces as a null output. So chunks returned with a null output is a specific, detectable state: retrieval worked, grounding didn't. That's the distinction that turns "the bot was wrong" into an actionable bug, and it's the single most useful thing an enterprise RAG stack can expose.

Knowledge base content is environment-scoped. Each environment carries its own copy of the rows and chunks, cloned when the environment is created. Staging content and production content are genuinely separate rather than separated by convention.

Retrieval depth is a parameter. The number of chunks returned defaults to 3 and can be raised to 30. Tuning it is one of the cheapest accuracy levers available, and it's the first thing to move when answers are correct but thin.

What all three have in common: they're about knowing what your system did. Retrieval quality is a solved-enough problem. Knowing which part failed is not, and that's where production time actually goes. If you're tracking this over a deployed agent, retrieval outcomes belong in the same view as your other agent analytics.

Is RAG Still Relevant?

It's a fair question, asked often enough to show up in the search results for this topic. Context windows have grown enormously, and "just put the documents in the prompt" is a real option now in a way it wasn't in 2020.

For a small, stable corpus, that's often the right call. Skip the infrastructure.

It stops working on three axes. Cost, because paying to process a large document set on every single turn adds up fast at volume. Latency, for the same reason. And permissions, which is the one that doesn't have a workaround: if the corpus contains documents this particular user isn't allowed to see, you cannot put the corpus in the prompt. Retrieval isn't just an efficiency trick there. It's the enforcement point.

Fine-tuning is the other comparison, and it answers a different question. Fine-tuning changes how a model behaves. Retrieval changes what it knows. If your problem is tone, format, or task-specific behavior, fine-tune. If your problem is that the model doesn't know your refund policy, and your refund policy changes, retrieve. Most teams need both and reach for the wrong one first.

So RAG is still relevant, but less as an architecture and more as a component. The interesting work moved up a layer, to the operational questions this article is mostly about.

Getting Enterprise RAG Right

The gap between a RAG demo and a production system isn't retrieval quality. It's permissions that hold at query time, content that's environment-scoped so you can test a change before customers see it, and enough visibility to tell a retrieval miss from a grounding miss when something goes wrong.

Voiceflow gives you the knowledge base, filtered retrieval, per-environment content, and the traces, inside the same platform where the agent that uses them is built. Book a demo to see how it fits your stack.