Retrieval, tool use, guardrails and evaluation, assembled into something that answers from your content, acts on your systems, and can prove where every answer came from.
Most of what gets called an AI project is this: a language model, connected to your content and your systems, wrapped in rules about what it may and may not do, and measured against examples you already have. The model is the least interesting part and the part you have least control over. Everything that decides whether the thing works is around it.
These are the failures we see most often when we are called in to fix someone else's build.
Retrieval quality is a content quality problem wearing a technical costume. If your documentation contradicts itself, is three years stale, or exists mainly in the heads of two senior people, no amount of embedding tuning fixes it. We audit the corpus before we build, and sometimes the honest answer is that the first sprint is content work, not AI work.
Fixed size chunks split tables down the middle, separate a clause from the definition it depends on, and orphan headings from the paragraphs beneath them. Legal and technical content in particular needs structure aware splitting, and the right strategy differs per document type inside the same corpus.
Without a fixed set of questions with known good answers, every release is judged by whoever happened to try it that afternoon. You cannot tell whether a prompt change helped or hurt, and you cannot tell when a provider silently updates a model underneath you. We build the evaluation set from your own resolved tickets before we tune anything.
A model asked a question its context does not answer will produce something anyway. Confident invention is worse than silence, because it costs trust in every future answer too. Abstention has to be designed, rewarded in evaluation, and tested for, or it does not happen.
An agent that can issue a refund, cancel an order or send an email needs a defined blast radius: what it may do unattended, what needs a human, what it may never do. Most failures we are asked to review are not reasoning failures, they are permission failures.
When a customer disputes what the system told them, you need the retrieved context, the tools called, the version of the prompt and the model that produced it. If that was not logged at the time it cannot be reconstructed afterwards.
Hybrid search combining semantic and keyword matching, because pure vector search misses exact identifiers, product codes and clause numbers. Reranking on top, structure aware chunking per document type, and metadata filters so a query never crosses a boundary it should not.
Hybrid retrieval, reranking, chunking strategyEvery response links back to the passage it came from, so a human can verify in one click rather than trusting the system. This is the single change that most reliably moves an internal tool from distrusted to used.
Citation and attributionFiltering results after the model has already seen them is not access control. Identity is resolved before the search runs, so a user's query only ever touches documents they are entitled to, and the model cannot leak what it never received.
Permission aware retrievalInput and output classification, topic and action allowlists, spend ceilings per conversation, and refusal behaviour that is tested rather than hoped for. Prompt injection is treated as an expected input, not an edge case, because any system reading untrusted content will meet it.
Guardrails, injection defenceA regression suite drawn from real resolved tickets and real questions, scored on correctness, grounding and abstention. It runs on every change, so you find out that a prompt edit broke twelve cases before your customers do.
Evaluation harness, regression testingHandoff triggered by uncertainty, sentiment or account value rather than keyword matching, carrying the full context so the human does not restart the conversation. The measure of a good agent is partly how gracefully it gives up.
Human in the loopThe systems these techniques get assembled into, described in the language of the job rather than the technology.
A thirty minute call, no pitch. We map where this fits, what it would take to build, and whether it is worth doing at all.