What is enterprise RAG, and what does a demo leave out?

The short version

• Enterprise RAG is retrieval-augmented generation that respects each user's permissions, cites its sources, stays current and is measured against a test set.
• The gap between a RAG demo and enterprise RAG is content, permissions and evaluation, not a better model.
• Permissions must apply before retrieval, because filtering results afterwards means restricted content has already been read.
• RAG reduces invented answers but does not end them, so every answer needs a citation a person can check.
• A company whose documents sit in one suite should usually try that vendor's assistant before building anything.

What is enterprise RAG?

Enterprise RAG is retrieval-augmented generation run on a company's own documents, with the controls staff need before they can rely on it. A search step finds relevant passages. A language model writes an answer from them and says where it came from. The word 'enterprise' adds requirements, not a different technique.

There are four main requirements. It must respect who is asking. It must stay current as documents change. It must show its sources. And somebody must be able to measure whether it works. A system missing any of these is a prototype, however good the answers look.

The pattern is common. Menlo Ventures' 2024 survey of 600 US IT decision-makers reported RAG at 51% adoption. The page does not state the base of that figure, so read it as a sign of direction, not a share of companies. Every respondent worked at an enterprise with 50 or more employees, so it says little about smaller firms.

How is enterprise RAG different from a RAG demo?

A demo answers one person's questions from a tidy folder. Enterprise RAG answers many people's questions from messy, changing, restricted content. The retrieval technique can be identical. What differs is the material, and the cost of getting it wrong.

Demos flatter RAG because someone picked the documents. There is one version of each policy, nothing is restricted, and the questions came from a person who knew the answers. Production has three drafts of the same policy, a scanned contract with a table in it, and a member of staff asking about a folder they should not be able to open. Most teams discover the gap the first time a request fails in a way the demo never showed.

How do a RAG demo and enterprise RAG compare?

What are the parts of an enterprise RAG system?

An enterprise RAG system is a pipeline with eight parts: connectors, parsing, permissions, search, reranking, generation, evaluation and logging. Most of the engineering sits before the model is called. The model is the most replaceable part of the system.

The common mistake is choosing a model first. A better model cannot answer from a passage the search never found, and it will write a confident answer from the wrong one. If retrieval misses, nothing later in the pipeline can recover.

The eight parts, in order

1. Connectors that read from the systems of record, such as SharePoint, Google Drive, a wiki or a ticketing tool, not from a copy somebody exported once.
2. Parsing and chunking that keep tables, headings and document markings attached to the text they belong to.
3. A permission layer that resolves the user's identity and scopes the search before it runs.
4. Hybrid search, because vector search alone often misses exact terms such as part numbers and clause references.
5. Reranking, which reorders the candidates so the passages reaching the model are the ones that matter.
6. Generation that answers only from retrieved passages, cites them, and declines when they do not contain the answer.
7. An evaluation set, run on every change to the model, prompts, chunking or search.
8. Logging of the question, the passages retrieved and the answer, so a wrong answer can be traced.

Why do permissions decide whether enterprise RAG is safe to use?

Permissions decide it because copying content into a search index creates a second access control surface, and that surface does not inherit the first. A document in SharePoint has rules the host enforces on every read. Once extracted and indexed, it is text and a vector. Those rules travel only if somebody carries them across and keeps them current.

Enforce permissions before retrieval, not after. Resolve the user's identity, then search only what they may open. Filtering the results afterwards is a display control: the restricted content has already been retrieved, and every later code path has to remember to drop it.

Permissions also drift. People change roles, folders get reshared, and a document deleted in the source can survive in the index. So the design question is how often you reconcile, and how quickly a removed document stops being retrievable. 'We re-index weekly' is an awkward answer if someone was removed from a project on Monday.

Does RAG stop AI from making things up?

No. RAG reduces invented answers, but it does not end them. The model can still misread a passage, merge two sources, or cite a document that does not support the claim. A citation proves a source exists, not that it says what the answer claims.

A public benchmark from legal research shows the limit. Stanford RegLab and Stanford HAI tested commercial legal AI tools, which are retrieval-backed products, against a pre-registered dataset of over 200 open-ended legal queries. They reported in May 2024 that the tools hallucinated in 1 out of 6 or more queries. That is a 2024 result, specific to legal research, and the tools have since changed. Do not carry the number across to your own system.

The lesson does carry. Retrieval gives you a checkable answer, not a correct one. So design for checking. Link each claim to its passage, keep that passage one click away, and tell staff the citation is the part to read. For decisions with real consequences, a person signs off.

How do you measure whether enterprise RAG works?

Measure it with a fixed test set of real questions with known answers, and score retrieval and generation separately. Retrieval asks whether the right passage was found. Generation asks whether the answer used that passage faithfully. If you score only the final answer, you cannot tell which part to fix.

Build the set from what people actually ask, taken from support tickets, shared inboxes and the questions new staff put to colleagues. Include questions the system should refuse, questions whose answer changed recently, and questions from a user who is not allowed to see the answer. That last group is the one most test sets leave out.

Run the set on every change to the model, prompts, chunking or search. Small changes can quietly move results. Then sample live usage, because people ask differently once a system is real, and add the failures to the set.

Should a smaller company build enterprise RAG or buy it?

Buy first if your documents live in one suite and your permissions there are already correct. A vendor's assistant is usually the cheaper first test. Build when your knowledge spans several systems, when you need control over what is cited and how, or when the assistant has to work inside your own tools.

The price of the first option is public. Microsoft's pricing page, checked in November 2026, showed Microsoft 365 Copilot Business at $18.00 per user per month paid yearly, originally $21.00, or $25.20 paid monthly. That sits on top of a Microsoft 365 subscription. Recheck it before relying on it, because vendor prices move.

An off-the-shelf assistant inherits your existing mess. If everyone can open everything in SharePoint, the assistant can find everything too. Tidy the permissions and the duplicate documents first. That work pays off on either route. RAG is also not worth building at all for a few dozen stable documents that one person could keep in a handbook.

What goes wrong with enterprise RAG after launch?

Mostly, the content moves and nobody owns it. Documents get superseded while the old version stays retrievable. A folder gets renamed and a connector quietly stops picking up changes. Answers get worse slowly, and no alarm fires.

Treat it as a system with an owner. Someone needs to watch ingestion failures, review the questions that got no good answer, and decide which source wins when two documents disagree. That last decision is not an engineering question. Retrieval exposes contradictions the organisation already had, and the fix is usually to retire a document, not to tune a model.