An AI transformation playbook is a sequence, not a document

The short version

• An AI transformation playbook is a sequence of decisions, not a document you circulate.
• Nobody can say whether an AI system helped if nobody measured the workflow before it was built.
• The first project should be chosen for how cleanly it can be judged, not for how much it is worth.
• Platform and policy choices are cheaper to make after one system is live, because the live system tells you what they need to cover.
• Most AI programmes stall at handover, not at the model: nobody owns the thing once the pilot team moves on.

Why do most AI transformation plans stall after the pilot?

Because nothing in the pilot was built to survive contact with the rest of the business. A pilot proves that a model can produce a good answer when someone who understands it chooses the input. Production asks a different question: what happens on the input nobody anticipated, on a Sunday, when the person who wrote the prompt has left.

The second reason is that the pilot had no before. If nobody wrote down how long the task took, how often it came out wrong, and what the wrong cases cost, then the pilot's result is a story told by the people who built it. Finance cannot sign off on a story. The programme stops not because the technology failed but because nobody can state what changed.

The third reason is ownership. A pilot usually belongs to whoever was enthusiastic about it. A production system needs someone whose job includes it when it breaks, a place the errors go, and a budget line. If those three things do not exist by the time the pilot ends, the pilot is the end.

Where should an AI transformation playbook start?

With one workflow, chosen because it can be judged. Not the most valuable workflow in the company, and not the one with the most interesting technical problem. The first build exists to teach the organisation how it builds, ships, measures and fixes these systems. That lesson is much cheaper to learn on something small and legible.

The useful test is whether somebody can describe the failure. If a person can say how often the task goes wrong, who picks it up when it does, and roughly what the fix costs, you have a workflow you can instrument. If the only available description is that the work is slow and annoying, you have a conversation rather than a project, and the first thing to do is go and watch the work.

Scope it down harder than feels reasonable. One queue, one document type, one customer segment, one language. A narrow first slice reaches real data faster, and real data is the only thing that tells you whether the idea holds.

What makes a good first workflow?

• •It happens often enough that you have evidence in weeks rather than quarters.
• •Someone does it by hand today, so a baseline exists to compare against.
• •Being wrong is visible and recoverable, so early mistakes cost attention rather than money.
• •The inputs already sit in a system you can read, not in a format somebody has promised to clean up.
• •One team owns the outcome, so there is a single person who can say whether it improved.
• •You can describe what a good result looks like in a sentence a reviewer would agree with.

What do you need to measure before you build anything?

Five things, and all of them exist today. How often the task happens. How long it takes. How often the current process gets it wrong. What happens next when it does. Who notices. Write those down before the build starts, because once the system exists you will not be able to reconstruct them honestly.

This is the highest-return hour in the whole playbook and it is almost always skipped, because next to the interesting work it feels like admin. The cost of skipping it is permanent. Months later the only available comparison is memory, and memory favours whoever is arguing hardest.

The baseline also sets the bar. Manual processes have error rates, and the current one is not zero. Teams routinely hold a new AI system to a standard the process it replaced never met, then shut it down when it misses that standard. Knowing the real baseline turns that argument into arithmetic, which is the only form in which it can be settled.

What does each stage of the playbook produce?

When should you make the platform and policy decisions?

Mostly after the first system is running, with a few exceptions you cannot defer. The exceptions are the ones that are expensive to retrofit: whose identity the system acts under, what data is allowed to leave your network, and whether every request is logged. Decide those before the first build, because changing them later means rebuilding rather than editing.

Everything else is better decided late. Which model, which vector store, which orchestration framework, which monitoring vendor: these look like foundational choices and are mostly consequences. A team that has run one system in production for a month can settle them from experience in an afternoon. A team that has not will spend a quarter in comparison documents and still choose badly, because every requirement in the comparison was a guess.

The same holds for the AI policy. A policy written before anything has shipped describes imagined risks. A policy written after one system has been live describes the real ones: the staff member who pasted a confidential document into a consumer chatbot, the reply that quoted an internal note back to a customer, the search index quietly holding files half the company cannot open. The second kind of policy is shorter and people follow it.

How do you tell whether the system is actually working?

By reading a sample of real output every week against a list of failure classes you wrote down in advance. Not by the demo, and not by a score on a test set the build team assembled from cases it had already solved.

Build the evaluation set from real traffic, and deliberately include the awkward cases: the empty field, the duplicate, the request in the wrong language, the one where the correct answer is that the system does not know. Score it before launch and keep scoring it after. When a new model version arrives you will want exactly the same set, otherwise every upgrade is a coin toss dressed up as progress.

Alongside that, track the one business number from your baseline. Handling time, backlog, error rate, cost per item, whichever you chose. If the evaluation says the system is accurate and the business number has not moved after a fair run, the system is accurate at something nobody needed. That is a useful finding, and it is only available to teams that wrote the number down first.

What should the second workflow look like?

Faster and less interesting, and that is the point. The first build pays for the scaffolding: how you reach data, how you deploy, where logs go, how you evaluate, who gets paged at night. The second build reuses it and should cost a fraction of the attention.

If the second workflow is as slow and as dramatic as the first, the first one taught you nothing. The usual cause is that it was built as a throwaway prototype by people who knew it would be thrown away, so none of the scaffolding was real.

This is where the honest playbook diverges from the slide version. The slide version shows parallel workstreams starting in the same month. In practice, three first builds at once gives you three sets of unlearned lessons and no shared foundation. Sequence them until the third one is boring, then run things in parallel.

Do you need to hire an AI team to run this?

For the first one or two workflows, no. What you need is an engineer or a small outside team who will put something on real data quickly, and an internal owner with the authority to change how the work is done. Hiring a head of AI to decide what to do first is a reliable way to get a strategy document and no system.

A permanent team starts to make sense once enough systems are running that somebody is already doing the work full time: upgrades, evaluation, access reviews, the on-call rota. At that point you are hiring against known duties rather than against a hope, and the job description writes itself.

If you are weighing outside help, judge the shape of the plan rather than the badges. Ask what the first deliverable is and when, who owns the system afterwards, what happens when the model is wrong, and what the team is explicitly not going to build. For our part, we have no case studies to show you and no certifications to point at, and we say so plainly on the site. What you can inspect is the sequence someone proposes and whether it puts a measurable thing in front of you early.

What is not worth doing first?

• •Company-wide AI training, before anyone has a system worth being trained on.
• •A central AI committee that reviews proposals and owns no delivery.
• •Buying a platform licence in order to find out what you need it for.
• •A data warehouse rebuild as a prerequisite, when the first workflow needs three tables.
• •Fine-tuning a model before prompting and retrieval have been tried and found insufficient.
• •A twelve-month roadmap, which will be wrong the moment the first build teaches you anything.