What should a RAG dashboard actually show?

The short version

• A dashboard that shows only latency, cost and request volume tells you the system is running, not that it is right.
• Most wrong answers in a retrieval system are retrieval failures, so retrieval is the layer worth instrumenting first.
• Every answer should be reconstructable: the question asked, the chunks returned, their scores, and the text the model received.
• The questions that returned nothing useful are the most valuable panel, because they name the content you do not have yet.
• A dashboard is not an evaluation. One describes what happened, the other judges whether it was correct.

What should a RAG dashboard actually show?

Four things, in the order they tend to break. What was retrieved for each question, whether the answer was grounded in what was retrieved, what the user did next, and which questions the corpus could not answer at all. Everything else on the page is supporting detail.

Retrieval comes first because that is where most wrong answers start. The model is usually doing a reasonable job with the passages it was handed. If those passages were the wrong ones, or the right one ranked fourth out of three shown, no amount of prompt work fixes the answer. So the first panel is retrieval: the similarity scores of the top results, how often the best result sits below whatever threshold you set, and how often nothing clears it at all.

Grounding comes second. For each answer you want to know whether its claims appear in the retrieved text. You can approximate that cheaply: check whether the answer cited anything, and whether the citations point at chunks that were genuinely in the context window. An answer with no citation, in a system designed to cite, is a defect whether or not it happens to be true.

User behaviour comes third, and it is the cheapest signal nobody collects. Did the reader open the cited source? Did they ask the same thing again in different words a minute later? Did they give up mid-session? A rephrase inside a minute is a failed answer, recorded by the user, for free.

Which panels to build first?

1. Retrieval: top result scores, the share of questions where nothing clears the threshold, and which documents win retrieval most often.
2. Grounding: share of answers carrying at least one citation, and whether cited chunks were actually in the context the model received.
3. Behaviour: citation click-through, rephrase-within-a-minute rate, abandoned sessions.
4. Coverage: the questions that found nothing useful, grouped by subject.
5. Cost and latency, last, because they are the easiest numbers to get and the least diagnostic.

Why does a dashboard built on latency and token spend mislead you?

Because a retrieval system fails without failing. Latency, token spend and error rate are the three numbers a model provider hands you by default, and all three can sit comfortably in range while the system returns a confident answer drawn from the wrong document. There is no stack trace for 'retrieved the superseded policy instead of the current one'.

This is the difference between a service dashboard and a product dashboard. Uptime panels answer whether the thing responded. A RAG dashboard has to answer whether it responded with something supported by the corpus. The second question is harder, and it is the only one your users care about.

Cost panels still earn a small place, for one narrow reason: they show when something has changed shape. A sudden rise in tokens per question usually means retrieval started returning more chunks or longer ones. That is a retrieval change arriving on the invoice before it arrives in a complaint.

What has to be logged before a dashboard can tell you anything?

The whole trace, per answer. The question as the user typed it, any rewritten or expanded query the system generated, the identifiers and scores of every chunk returned, the context the model actually received, the output, the citations displayed, and what the user did next. Miss any one of those and you can see that an answer was bad without being able to say why.

Log the identity that asked, and the permission scope the search ran under. When somebody reports an answer containing a line they should not have been able to see, that one field is the difference between an afternoon of investigation and a fortnight of it.

Then decide retention before you switch logging on. Storing question text and retrieved passages means you are now holding a second copy of sensitive content, in a system built for convenience rather than custody. The answer 'we keep everything' becomes awkward the first time somebody asks for their data to be deleted, or asks which systems hold it.

What does each panel tell you, and what does it not?

Why are the failed questions the most valuable thing on the page?

Because they are the only panel that tells you what to do next. Every other number describes the system you have. The list of questions that returned nothing above threshold describes the system you do not have yet, written in the users' own words and ranked by how many of them asked.

Group them and three shapes appear. The document exists but was never indexed. The document was indexed but its wording never matches how people ask. The answer does not exist anywhere and somebody has to write it. Only the third is a content problem. The first is a pipeline problem, the second is a retrieval problem, and all three get different fixes from different people.

This panel also settles an argument that otherwise returns every month, which is whether to keep adding content and what to add. You stop guessing at a content roadmap and start reading one off the failures.

How is a RAG dashboard different from an evaluation suite?

A dashboard is descriptive and continuous. An evaluation is judgemental and repeatable. The dashboard tells you what happened to real questions yesterday. The evaluation tells you whether a change you are about to ship makes answers better or worse across a fixed set of questions whose correct answers somebody wrote down in advance.

They feed each other. The failed questions and the rephrased questions on the dashboard are the raw material for the evaluation set, because they are real and they are hard. A set written by the team in a meeting room will be too easy, and it will miss the phrasings users actually type.

If you have to pick what to build first, build the log and the evaluation set. A dashboard over data you never stored is not possible at all, while an evaluation over a few dozen real questions can live in a spreadsheet and still catch a regression that a dashboard would only reveal a week later, through a drop in click-through that somebody has to notice.

When is a RAG dashboard not worth building?

When the volume is still low enough to read. A system answering a few dozen questions a day does not need panels. It needs a plain table of the last few hundred traces, sorted by time, that one person reads properly once a week. Dashboards are what you build when reading everything stops being possible, and building them earlier replaces attention with aggregation.

It is also not worth building twice. Hosted tracing and evaluation products are good at the request lifecycle, and reimplementing that is wasted effort. What they do not know is your corpus: which documents win retrieval, which parts of the shared drive are never retrieved at all, which questions came back empty. Buy the trace viewer, build the coverage panel.

And no dashboard replaces somebody whose job it is to look at it. The failure mode in every organisation is the same: good instrumentation, nobody named as the reader. A weekly half hour with the failed questions beats another row of panels.

What does a reasonable first version look like?

One page, six panels, built on a trace table you already have. Score distribution for the top retrieved chunk. No-result rate over time. Citation rate. Rephrase-within-a-minute rate. Top twenty documents by retrieval wins. A scrollable list of the questions that found nothing, newest first.

That is enough to answer the three questions anybody will ask you about a retrieval system in its first month: is it finding things, are the answers anchored to what it found, and what is missing. Add cost and latency because they are free. Resist adding anything that looks impressive without changing a decision, which in practice means most visualisations of the embedding space.

Instrumentation like this is cheaper to add at the start than to retrofit, because the hard part is not the charts. It is having kept the fields the charts need. If you are standing up retrieval this quarter, write the trace schema before the prompt.