What is a RAG bot, and what decides whether it works?

The short version

• A RAG bot answers questions by searching a set of documents first, then asking a language model to answer from the passages it found.
• Most wrong answers from a RAG bot are retrieval failures: the right passage was never found, so the model answered from the wrong one.
• Retrieval reduces invented answers but does not eliminate them, even in commercial tools from large vendors.
• Keyword search, embeddings and reranking used together find the right passage more often than any one of them alone.
• A RAG bot is not worth building over content that fits in one prompt, or for questions that need a database query.

What is a RAG bot?

A RAG bot is a chatbot that looks up the answer in your documents before it replies. RAG stands for retrieval-augmented generation: retrieve relevant passages first, then generate an answer from them. The model does not memorise your content. It reads a few passages, chosen fresh for every question, and answers from those.

That is the whole difference from a general chatbot. A general model answers from whatever it absorbed in training, which does not include your refund policy, your internal wiki or last week's price change. A RAG bot can answer about all three, because it reads them at the moment of asking, and it can show you which passage it used.

It is also the most common way companies put language models to work on their own information. Menlo Ventures' 2024 survey of 600 U.S. IT decision-makers found retrieval-augmented generation to be the dominant design pattern among the enterprises it surveyed, far more common than fine-tuning.

How is a RAG bot different from a plain chatbot or a fine-tuned model?

How does a RAG bot work?

A RAG bot works in two phases. Ahead of time, it turns your documents into a searchable index. At question time, it searches that index, hands the best passages to a language model, and the model writes an answer grounded in them, with references back to the source.

The indexing phase is where most of the engineering lives. Documents are extracted to text, split into chunks of a few paragraphs, and each chunk is stored twice: as an embedding, a list of numbers that captures meaning, and as words in a keyword index. Both matter. Embeddings find passages that say the same thing in different words. Keyword search finds exact terms, product codes and names that embeddings tend to blur.

The answering phase is shorter than people expect. The model is not doing the finding. It is given a handful of passages and an instruction: 'answer from these, cite them, and say so if the answer is not here'. If the right passage is not in that handful, no model can recover it.

What are the steps in a RAG pipeline?

1. Ingest: pull documents from where they live, such as a help centre, a shared drive or a ticket system, and extract clean text.
2. Chunk: split each document into passages small enough to be specific but large enough to make sense on their own.
3. Index: store each chunk as an embedding for meaning-based search and in a keyword index for exact terms, with metadata such as source, date and who may see it.
4. Retrieve: when a question arrives, check who is asking, then search only the content that person is allowed to see, using both indexes.
5. Rerank: score the candidate passages again with a slower, more precise model and keep the best few.
6. Generate: give those passages to the language model with instructions to answer only from them and to cite each one it uses.
7. Return: show the answer with links to its sources, and log the question, the passages and the answer so failures can be found later.

Why do RAG bots give wrong answers?

Most wrong answers from a RAG bot are retrieval failures. The passage that held the answer was never found, so the model answered from a nearby passage that looked relevant and was not. The model then does what models do: it writes a fluent, confident reply from what it was given.

The usual causes are mechanical. A chunk was cut in the middle of a table, so the price sits in one chunk and the product name in another. Two versions of the same policy are both indexed, and the old one ranks higher. A question uses the customer's word for something and the documents use the company's word. A scanned PDF was never converted to text, so search cannot see it.

Retrieval narrows the problem. It does not remove it. Stanford RegLab and HAI ran a pre-registered set of over 200 open-ended legal queries against commercial RAG-based legal research tools, including Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI, and checked the answers by hand. The tools hallucinated in 1 out of 6 benchmarking queries or more. These are products from large vendors, in a field where accuracy is the selling point.

What makes RAG retrieval better?

Three changes do most of the work. Search with keywords and embeddings together. Give each chunk the context it lost when it was cut out of its document. Rerank the results before the model sees them. Each one attacks a different way the right passage goes missing.

Anthropic measured this directly. In its contextual retrieval experiments it tracked how often the right chunk failed to appear in the top 20 results. Adding document context to each chunk before embedding reduced those failures. Adding a contextual keyword index on top reduced them further, and reranking reduced them most. The techniques stack.

None of that rescues bad source material. Before tuning search, look at the documents. Remove duplicates and superseded versions, fix extraction on scanned files, and decide which source wins when two disagree. A RAG bot repeats contradictions faithfully. It cannot tell you which version of a policy is current unless something in the data says so.

How do you test whether a RAG bot is good enough?

Build a set of real questions with known answers and the passage each answer should come from, then measure retrieval and answers separately. If the right passage was not retrieved, the fix is in search or in the data. If it was retrieved and the answer was still wrong, the fix is in the prompt or the model.

Take the questions from where people already ask them: support tickets, internal chat, the email the operations lead answers every week. Include questions whose answer is not in the documents. A good RAG bot says it does not know. A bad one invents a plausible answer, and that is the failure people remember.

Run the set every time something changes: a new document source, a new chunking rule, a new model. Without it, every change is a guess, judged by whoever happens to try the bot that afternoon.

Should you build a RAG bot or buy one?

Buy when the content is a public help centre and the job is answering customers. Build when the answers live across several internal systems, when different people may see different documents, or when you need control over where data goes and how retrieval works.

Hosted support agents are an established category, and the buy side of the comparison is easy to price because vendors publish it. Intercom lists Fin's price on its own pricing page. Set that against the cost of an engineer's time to build, test and maintain an equivalent. The U.S. Bureau of Labor Statistics publishes the median wage for software developers, and a RAG bot is not a one-off build.

Off-the-shelf tools get harder to justify when permissions matter. A support bot reading public articles has no access problem. An internal bot reading HR files, contracts or client folders does, and it has to check who is asking before it searches, not filter the answer afterwards.

When is a RAG bot not worth building?

A RAG bot is not worth building when the content is small and stable, when the questions need calculation rather than lookup, or when nobody owns the documents. In each case a simpler tool does the job better, or the bot would spread a problem that already exists.

If everything the bot needs fits comfortably in one prompt, such as a two-page policy or a short price list, skip retrieval and include it directly. If the questions look like 'how many orders shipped late last month', the answer is in a database, and a query or a dashboard is more reliable than a model reading passages. If the documents are out of date and nobody is responsible for them, a RAG bot will put that staleness in front of more people, faster, with more confidence.

A RAG bot earns its keep where the same questions are asked often, the answers sit in documents that change, and a person currently spends real time finding them. That is a narrower set of cases than the hype suggests, and a much more dependable one.