Your chatbot is only as good as the knowledge base behind it

The short version

• A chatbot knowledge base is read in fragments, so an article that only makes sense as a whole page will be answered from badly.
• Most knowledge base chatbot failures are content failures: duplicate articles, stale articles, and answers that live only in somebody's head.
• Writing that points at the screen, 'click the button above', loses its meaning once a retriever lifts the paragraph out of the page.
• A bot that says it does not know and hands over the transcript is more useful than one that guesses.
• The questions your chatbot cannot answer are the most valuable thing it produces.

What is a chatbot knowledge base, and how is it different from a help centre?

A help centre is a website. A chatbot knowledge base is a retrieval corpus. The difference matters because of how the two get read. A person lands on a page and sees the title, the breadcrumb, the sidebar, the screenshots and the related links, then reconstructs the context before reading a word of the body. A retriever does none of that. It finds the passage closest to the question, hands that passage to the model, and the model answers from what it was given.

So the unit of quality changes. For a reader, the unit is the page. For a chatbot, the unit is the chunk, which is usually a few hundred words lifted out of the middle of something. If that chunk does not say which product it covers, which plan, which region and which version, the model will either guess or blend it with a chunk from a different article that was more specific.

This is why pointing a chatbot at an existing help centre tends to produce something that looks impressive in a demo and then goes wrong in ways nobody can explain. The prompt is rarely the problem. The corpus is.

Why do good help centre articles make bad chatbot answers?

Because good help centre writing leans on the page around it. Phrases like 'click the button above' and 'see the table below' are perfectly clear to someone looking at the screen and meaningless inside a retrieved fragment. The same goes for steps that continue across a screenshot, tabbed content where only the first tab exists in the markup, and the sentence 'as mentioned earlier' now pointing at nothing.

The second problem is implicit scope. An article written three years ago about a feature that has since changed does not announce 'this describes the old behaviour'. It just describes the old behaviour, confidently, in the present tense. A reader glances at the date in the corner and adjusts. A retriever has no reason to prefer the newer article, because the older one may match the question better word for word.

The third is duplication. Most knowledge bases hold three or four articles answering the same question slightly differently, written by different people at different times, none of them wrong when they were written. A person picks the one that looks freshest. A retriever picks the one with the closest wording, and the model presents it as the answer with no sign that a contradiction existed.

None of this is a model failure. It is the ordinary cost of writing for a reader who can see the whole screen.

What belongs in the knowledge base and what should stay out?

Put in what is true, current, and answers a question somebody actually asks. Keep out everything else, which in practice means much more than people expect. 'We keep everything' is a reasonable instinct for an archive and an expensive one for a retrieval corpus, because every stale document is a candidate answer.

Three categories cause most of the damage. Internal drafts and working documents that were never meant to face customers. Announcement posts, which are dated by nature and read as current fact once the date is stripped away. And anything that duplicates a definitive article with slightly different numbers, because the place a chatbot is most reliably wrong is where two sources disagree and nothing marks which one wins.

The other half of scoping is access. If the corpus mixes customer facing help articles with internal runbooks, the system has to know which audience it is serving before it retrieves, not after. A permission check applied to the output is a display control: the content already reached the model, and a model that has read something can be argued into repeating it.

What should you fix in an article before you index it?

• •Give the article one question as its title, phrased the way a customer would ask it.
• •State the product, plan, region and version in the body text, not only in the page title or breadcrumb.
• •Replace references to screen position, 'the button above', with the name of the thing.
• •Write out anything that exists only inside a screenshot, including error text and menu paths.
• •Keep a whole answer in one place and link out to related detail, rather than splitting one answer across three articles.
• •Delete, or clearly mark, anything that describes behaviour you no longer ship.
• •Make tabbed and accordion content retrievable as text, because a retriever only sees what is in the markup.

What should the bot do when the knowledge base has no answer?

Say so, and hand the conversation to a person with the transcript attached. This is the highest value behaviour in a knowledge base chatbot and the one most often skipped, because refusal looks like failure in a demo. It is not. A confident wrong answer costs a second contact plus the trust the customer brought to the first one.

Coverage gaps are the normal state, not a defect you can finish fixing. Pricing exceptions, account specific behaviour, anything decided case by case: none of that is in the articles, and no amount of retrieval will find it. The real question is whether the system can tell a thin match from a good one.

That decision needs a threshold you set, test and can move, not a number the model invents about itself. Ask a model whether it is confident and it will tell you it is. Check instead whether the retrieved passages actually contain the answer, and refuse when they do not. Then make the refusal useful: name what you could not confirm, say what happens next, and pass the handover across with everything the customer has already typed.

Escalation also protects the content. Every refusal is a logged question your knowledge base does not answer, which is a better content roadmap than any internal guess about what customers want to read.

Which questions should a knowledge base chatbot not try to answer?

The ones whose answers are not in documents. A knowledge base answers questions about how the product works. It cannot answer questions about one customer's account, because that information sits in a billing system or an order record and it is different for every person asking.

This sounds obvious, and it is still where most of the disappointment comes from. Inbound queues are dominated by account specific questions, and a retrieval bot meets them by producing the generic article about billing when the customer wanted to know about their own invoice. The fix is a tool call against the real system with an authenticated user, or a fast handover. It is not more articles.

Mapping the question types before you build is a short exercise, and it decides what you are actually building.

Where does the answer actually live?

Who keeps a chatbot for a knowledge base current?

Someone named, with time in their week. A chatbot for a knowledge base is not a project that finishes. It is a content operation with a thin layer of software on top, and the software is the part that stays still. The product ships, the pricing moves, a policy gets an exception, and the corpus drifts out of date while the bot keeps answering in the same even tone.

The practical version of maintenance is small. Tie article review to the events that invalidate articles: a release, a pricing change, a policy decision, a question that escalates twice in a month. Keep an owner per section of the corpus rather than one owner for all of it. Re index on a schedule you can state out loud, so that 'the bot is wrong' can be diagnosed as a stale index instead of argued about.

Deletion is part of this and it is the part people resist. An article nobody has opened in years is not harmless once it sits in a retrieval corpus. It is a competing answer, waiting for the one question it matches better than the article you would rather have been used.

How do you tell whether the chatbot is working?

Choose a measure before launch, and do not choose accuracy. Accuracy against a question set you wrote yourself tells you the bot can answer questions whose answers you already knew. The measures worth having come from live traffic: how often the retrieved passages genuinely contained the answer, how often the bot refused, what it refused about, and whether the conversations that reached a person arrived with enough context to close faster.

Comparison is what makes any of this credible, and it is the part teams skip. The most careful published work in this area treats access to the assistant as the variable rather than measuring the tool on its own. You can borrow that design at small scale: route a share of conversations without the bot, or hold one team back for a few weeks, and compare the two groups on the same queue.

Be sceptical about vendor figures, including the ones published by products you might be comparing yourself against. Intercom's numbers for its Fin agent come from its own aggregated product data across its customer base. That is real data, and it is data about Intercom's deployments, their content and their traffic mix. It is not a forecast for yours.

The same caution applies to survey work, which is useful for context and no use at all for prediction. Broad industry surveys tell you what other teams report doing. They cannot tell you whether your particular corpus will support the answers you want, and that is the only question that decides whether your build works.

When is a knowledge base chatbot not worth building?

When the knowledge base is thin, when the questions need account data more than documents, or when the volume does not justify the maintenance. All three are common and none of them is embarrassing.

A corpus of twenty articles does not need retrieval. It needs a better search box and, if the articles are good, a link in the right place. Building a bot on thin content produces a system whose main skill is rephrasing the same three articles in different words, which readers notice immediately.

If most of your inbound questions are 'where is my order' or 'why was I charged this', the work ahead is integration rather than retrieval. The answers sit in systems, not articles, and a bot that cannot reach those systems will route almost everything to a person anyway. That is a support agent with tool access, which is a different build with a different cost.

And if the queue is small and handled well by two people who know the product, a chatbot is a maintenance obligation with a thin return. It is fine to say that out loud, and better said before a build than after one.