The agentic AI tech stack: six layers, and the three that decide whether it works

The short version

• An agentic AI tech stack has six layers: a model, orchestration, tools, retrieval and memory, guardrails and permissions, and observability with evaluation.
• The model is the easiest layer to replace, so choose it last, by testing it against an evaluation set of real tasks.
• A framework is optional. A single agent with a few tools can run on a short loop in your own code.
• An agent should act with the permissions of the person it works for, never through a shared account with wider access.
• Without traces and an evaluation set, nobody can tell whether a change to a prompt, tool or model made the agent better or worse.

What are the layers of an agentic AI tech stack?

An agentic AI tech stack has six layers. A language model reasons about the task. An orchestration loop decides the next step and keeps track of state. Tools let the agent read from and write to other systems. Retrieval and memory supply context the model does not have. Guardrails and permissions limit what it can do. Observability and evaluation tell you whether any of it works.

Underneath sits ordinary software: hosting, a queue, a database, secrets management and deployment. None of that is specific to agents, and putting an agent on top does not make any of it optional.

The layers do not take equal effort. If the rest of the stack is built well, the model sits behind one interface and can be swapped quickly. The tools, the permissions and the evaluation set are where the engineering time goes, because they encode how your business actually works.

What does each layer of an agentic AI stack do?

Which model should an agentic AI stack use?

Use the model that passes your evaluation set at a cost per task you can accept. That sounds circular, and it is meant to. You cannot choose a model well until you have real tasks to test it against, so the model is the last layer to commit to, not the first.

Agentic work asks more of a model than chat does. It has to pick the right tool from several, fill in the arguments correctly, read the result and judge whether the task is finished. A model that writes fluent answers can still be poor at this, and the only way to find out is to run it on your own tasks.

A stack can use more than one model: a capable one for planning and difficult steps, and a cheaper, faster one for routing, classification or extraction. Keep every model behind a thin interface in your own code, so that changing one is a configuration change. Prices and rankings move quickly, and a stack tied to one provider pays for it later.

Do you need a framework to build an AI agent?

Not always. At its core an agent is a short loop. Send the conversation and the tool definitions to the model, run any tool it asks for, add the result, and repeat until it gives a final answer or hits a limit. For one agent with a handful of tools, that loop in your own code is often enough, and you can read every line of it.

A framework earns its place when runs get long or get interrupted. If a run takes minutes, waits for a person to approve a step, or has to survive a server restart halfway through, you need durable state, retries and a way to pause and resume. Frameworks such as LangGraph, and the agent SDKs that model providers publish, supply those pieces. You can write them yourself, but that is rarely the best use of the time.

Whatever you choose, three limits belong in this layer: a maximum number of steps, a timeout, and a spending cap for each run. An agent that misreads a tool result can call the same tool again and again, and without limits nothing stops it.

In a multi-agent design, several agents hand work to each other, which adds a coordination problem on top of everything else. Start with one agent. Split it only when one agent's instructions and tools no longer fit together cleanly.

How does an AI agent connect to tools and data?

Through tool definitions. Each tool has a name, a description and an input schema. The model reads them, decides to call one and writes the arguments. Your application runs the call and passes the result back, not the model. The model never touches your systems directly.

The Model Context Protocol (MCP) gives that connection a standard shape, so a tool written once as an MCP server works with any client that speaks the protocol. Anthropic created MCP. On 9 December 2025 it donated the protocol to the newly established Agentic AI Foundation, so a foundation now holds it, not a single vendor.

Anthropic also published its own count of active public MCP servers at the time, without saying how it counted them. Whatever the true number, a public MCP server is third-party code that runs with whatever credentials you give it. Review it as you would any dependency with access to production data, or write your own.

How good the tools are matters more than which protocol carries them. Narrow tools with clear descriptions get called correctly. Write operations should be idempotent, so that a retry does not send the same refund twice. Errors should tell the model what to do next. 'Customer not found, check the ID format' lets it recover. A bare 'error 500' leaves it guessing.

What do retrieval and memory add to an AI agent?

They give the model context it was not trained on: your documents, your records and what happened earlier in the task. Retrieval searches your content and returns the relevant passages. Memory carries state, either within a single run or across sessions.

In an agentic stack, retrieval can be a tool the agent calls when it needs it, instead of a search that runs before every answer. The agent can then search, read the results and search again with a better query. A single fixed search cannot do that.

Treat long-term memory with care. Facts an agent saves about a user go out of date. If they are stored under the wrong key they can leak into another user's session. They also become data you have to manage and delete on request. Store less than you think you need.

Both depend on permissions. If the index does not know who may read each document, the agent can retrieve content for a user who could never open it. Check permissions before the search, not after.

Where do guardrails and permissions fit in an agentic AI stack?

Around every tool call, not just the final answer. The agent should act with the identity and permissions of the person it is working for, so it can never see or change more than that person could.

In practice that means credentials scoped to each user instead of one service account, read tools kept separate from write tools, and a human approval step before anything irreversible, such as issuing a refund, emailing a customer or deleting a record. You can relax approval later, once the traces show the agent gets that action right.

Treat everything the agent reads as untrusted. A retrieved document, a web page or a tool result can contain instructions, and the model may follow them. This is called prompt injection, and you cannot rely on filtering it out of the input. The defence is to limit what a misled agent is able to do. That is a question of permissions, not of prompts.

How do you know whether an AI agent is working?

You trace every run and test every change against an evaluation set. A trace records each model call and tool call, with its inputs, outputs, latency and cost. An evaluation set is a collection of real tasks with known good outcomes. You run it again whenever a prompt, a tool or a model changes.

Judge the path as well as the result. An agent can reach the right answer by luck, or by calling five tools where two would do. Check that it chose the right tools, passed sensible arguments and stopped when it should have.

Track cost per completed task, not cost per call. An agent makes several model calls for each task, so a cheaper model that needs more steps can cost more overall.

Build this layer first, not last. Without it, every model upgrade or prompt edit is a guess, and you hear that it was a bad one from a user.

When is an agentic AI stack not worth building?

When the steps are known in advance. If you can draw the process as a fixed flowchart, a set workflow that uses a model for individual steps will be cheaper, faster and easier to test than an agent that chooses its own route.

A full agentic AI technology stack earns its complexity when each step depends on what the last one returned. Investigating a support ticket, reconciling records that disagree and researching across several sources all fit. Document extraction, routing and form filling usually do not.

In what order should you build an agentic AI stack?

1. Write down the task and what a correct outcome looks like.
2. Collect a few dozen real examples to use as the evaluation set.
3. Connect one model to one or two read-only tools with a plain loop, with tracing on from the first run.
4. Run the evaluation set and compare models on pass rate and cost per completed task.
5. Add permissions and human approval before any write action reaches production.
6. Add retrieval, long-term memory or a framework only when a failing test shows you need it.