What a chatbot demo proves, and what it hides

The short version

• Every current model can hold a conversation, so a demo that proves only that has told you nothing.
• A vendor demo runs on content the vendor chose and questions the vendor has already seen answered.
• The useful test is your own documents, your own worst questions, and one question that has no answer in the content.
• Most of the cost and most of the risk sit in identity, integration, escalation and review, none of which a demo touches.
• A bot that cannot say it does not know has no floor on how wrong it can be.

What does a chatbot demo actually prove?

That the system can talk. A chatbot demo puts a prepared question in front of prepared content and produces a fluent answer. Every current language model clears that bar, which means the demo is exercising the one part of the problem that is already solved. The interesting question is not whether it answers. It is what it does when it should not.

This matters because the demo is the artefact most buying decisions actually turn on. Somebody sits through a thirty minute call, the bot answers nine questions well, and the room concludes the hard part is done. The hard part has not been touched. It begins at the point where the content is yours, the questions are unscripted, and a wrong answer costs a customer money.

A cleaner way to frame it: a chat bot demo is a test of the vendor's content and the vendor's question list. Swap both for yours and you have an evaluation. Leave both as they are and you have a presentation.

Why does the demo chatbot always answer well?

Because it runs in a closed world, and a small one. The content behind a vendor's demo chatbot is usually a few dozen pages of marketing copy, written recently, by a handful of people, with the deliberate goal of being unambiguous. There is one answer per question and nothing in the corpus contradicts it.

Your content is not like that. It is thousands of pages written over years by people who have since left, with two live versions of the refund policy, a price list that was last updated in a spreadsheet nobody linked, and an internal note that must never reach a customer. Retrieval quality on that corpus is a different problem from retrieval quality on a brochure. No demo running on the vendor's own site will tell you anything about it.

The question list is curated as well. Demo questions are the ones somebody has already checked the answer to. Nobody volunteers the question where two documents disagree, because the honest behaviour there is to say they disagree, and saying that does not look impressive.

What should you bring to a chatbot demo?

Your own worst questions and your own messy documents. Those two inputs are what turn a demo into evidence. The rest of the meeting is theatre you can sit through politely.

Pull the questions from somewhere real: the support inbox, the live chat transcripts, the thread where the same thing gets asked every January. Take them verbatim, with the typing errors and the ones that trail off mid-sentence, because that is the shape of the traffic the system will actually receive. Twenty is enough to see the pattern, and few enough that you can run them in one sitting while somebody watches.

Bring these to the demo

• •Twenty real questions copied from your inbox or chat logs, unedited, including the badly spelled ones.
• •Two questions whose correct answer changed in the last three months.
• •One question where two of your own documents disagree with each other.
• •One question that has no answer anywhere in your content.
• •One question that is really a complaint, where the right move is a person rather than an answer.
• •One question about pricing or eligibility where being approximately right is worse than refusing.
• •One question in the second language your customers actually use.
• •One request to do something rather than answer something: cancel, reschedule, refund.

What happens when the demo meets your own documents?

It usually gets worse, and that is the useful part. The first run on real content is where you discover that a third of the corpus is out of date, that the answer to your most common question lives in a scanned PDF nothing can parse cleanly, and that retrieval picks the wrong of two similar documents more often than anyone expected.

None of that is a reason to stop. It is the actual work, and finding it in week one is cheap. Finding it after launch, through a customer who was quoted the wrong notice period, is not. So ask any vendor for a sandbox loaded with your own content and a week to poke at it unsupervised. A vendor who will not do that has told you something worth knowing.

Permissions need a separate look if the bot will ever see internal material. Content copied into a search index does not carry the access rules of the system it came from, so 'the bot can read it' and 'this user may see it' are two different statements. In a demo they look identical, because the demo has one user and nothing to protect.

What does a chatbot demo not show you at all?

The parts that cost money. A demo shows the conversation. It leaves out identity, integration, escalation, logging, evaluation, and behaviour under load. That is where the budget goes and where the incidents come from.

Identity first. A demo bot answers the same way for everyone, because there is nobody to distinguish. A production bot discussing an order has to be certain whose order it is, which means it is now part of your authentication story. Then integration. An answer is cheap, an action is not. 'Where is my order' is a retrieval problem. 'Cancel my order' is a write into a system with its own rules and its own consequences when the bot has misread the request.

Then the operational half. Who reads conversations each week, and what happens to what they find. What gets logged, how long it is kept, and who can see it. What the bot does when an upstream API is down, which is a design decision, not an accident. What a conversation costs, and what it costs at ten times the volume, because a pilot of a hundred chats a day and a launch of three thousand are not the same system economically. None of this appears in a demo unless you ask, so ask.

What the demo shows against what you still need to check

How should a chatbot demo handle a question it cannot answer?

By saying so quickly and offering a person. That single behaviour is the most informative thing in any demo, and it is the thing you are least likely to be shown unprompted. So ask for it directly. Put a question to the bot that has no answer in the content and watch what comes back.

There are three outcomes. It invents something plausible, which is the failure you must design against, and the one that survives review because it reads well. It produces a vague non-answer that avoids being technically wrong while telling the customer nothing, which is the same failure moving more slowly. Or it says it does not know, states what it can help with, and hands over. Only the third holds up against real traffic.

Ask how refusal is implemented, too. If the answer is that it is written in the prompt, you have a preference rather than a control. A preference degrades quietly when the model is updated, when the question is phrased unusually, or when somebody later appends an instruction to be more helpful. Ask what test proves the refusal still works next month.

What is the difference between a chatbot demo and a pilot?

A demo is a sales artefact. A pilot is a scoped deployment on real traffic with agreed numbers and a defined exit. If you are making a purchase decision of any size, you want the second one, and you should expect to pay for it.

A pilot worth running has four things fixed before the first conversation. A defined slice of traffic, so you know what it was tested on: one channel, one topic, one time window. Metrics written down in advance, including the one that would make you stop. A kill switch somebody on your side can pull without a support ticket. And clear ownership of content, so when the answer is wrong it is obvious whose job it is to fix the page.

Set the measurement before you start. Deciding afterwards what counted as success is how every pilot gets declared successful. Pick the numbers that would embarrass you: how often the bot answered when it should have escalated, how often a customer asked the same thing twice, how many conversations a human had to redo.

Is it worth building your own demo chatbot?

Often yes, and it is cheaper than people expect. A prototype on your own documents, built in a week, teaches you more than five vendor calls, because it runs on your mess instead of somebody else's brochure. It also gives you a baseline: if a paid product cannot clearly beat what two engineers managed in a week, that is a finding.

The trap is what you conclude from it. A working prototype has skipped authentication, logging, evaluation, rate limits, escalation, the second language and the day the model changes underneath you. The last tenth of the demo is the first tenth of the system. Teams who miss that distinction buy a prototype and then spend two quarters discovering what it was not.

And sometimes the honest answer is that you do not need a chatbot. If your top twenty questions are all answerable from a page that already exists, a better search box and a clearer pricing page will serve customers faster and cost less to run. That is not a conclusion anyone is incentivised to offer you during a demo, which is exactly why it is worth checking first.