AI agent architecture: four decisions that matter, and the ones you can defer

The short version

• An agent architecture is four decisions: the loop, the tool layer, the memory boundary and the stopping rule.
• If you can draw the flowchart, write the flowchart. A deterministic workflow with model calls inside it is cheaper and easier to debug than an agent.
• Permission checks belong in the tool, not the system prompt. A prompt is a suggestion to a probabilistic system.
• A multi-agent architecture earns its token cost when the work is parallel and the context is separable, not when the steps are sequential.
• One successful run is not evidence. Run the identical task repeatedly and look at the spread of outcomes.

What is an AI agent architecture?

It is the set of decisions about who holds control. An agent architecture has four moving parts: a model running in a loop, a set of tools the model can call, the state that survives between steps, and the code that decides when the loop stops. Frameworks, prompt formats and model vendors are all replaceable. Those four decisions are not.

The loop itself is simple enough to write on a napkin. The model receives a context, picks an action, your code runs the action, the result goes back into the context, and round it goes again. What makes this an agent rather than a pipeline is that the model chooses the next step. You are handing control flow to something probabilistic, and in exchange you get a system that can handle requests whose shape you did not know in advance.

So the interesting part of any architecture of AI agents is not the happy path. It is what happens when a tool returns an error, when the model calls the same tool four times in a row, when the context fills up halfway through a long run, and when a user asks for something the tool set cannot do. Design for those four and the happy path looks after itself.

When is an agent the wrong architecture?

When you already know the steps. If you can draw the flowchart, write the flowchart. A deterministic pipeline with model calls inside it is cheaper, faster, testable with ordinary unit tests, and debuggable by reading a log from top to bottom. Classify this document, extract these six fields, write them to that table, flag it for a human if confidence is low. There is no decision left for an agent to make.

The cost difference is not a rounding error. Anthropic's published telemetry for its own research feature puts agent token use at roughly four times a chat interaction, and multi-agent systems at roughly fifteen times. That multiplier buys flexibility. If the flexibility is not needed, you are paying for an open-ended loop to rediscover a sequence you could have written down once.

The honest test is whether the number of steps depends on what the system finds along the way. Checking forty suppliers where any one of them might need a follow-up call, researching a question where the next search depends on the last result, triaging a ticket that might need one lookup or nine: those are agent problems. A large share of back office automation is not, and saying so early saves a quarter.

What are the main AI agent architecture patterns?

There are five worth knowing, and they compose. A single agent with tools is the default and covers more ground than people expect. A router classifies the request and hands it to a specialist. A chain fixes the order of steps and uses the model inside each one. An orchestrator with workers splits a task, runs the parts in parallel and merges what comes back. An evaluator loop generates, critiques and revises until a check passes.

Start with one agent and a small tool set. Add structure only when a specific failure forces it. A router appears when one system prompt is trying to be three different things and doing all three badly. An orchestrator appears when the work is genuinely parallel. An evaluator loop appears when quality matters more than latency and you have a check you can actually express in code or in a rubric.

Most of what gets described as a multi-agent architecture is a single agent with a long prompt, or a chain with a model in the middle. That is not a criticism. It is usually the right build, and the diagram being boring is not a problem anybody has ever been paged about.

Which pattern holds which decision?

How do you design the tool layer?

Write each tool as an API for a caller who cannot ask you a clarifying question. The model sees a name, a description and a schema, and nothing else. If two tools overlap, it will sometimes pick the wrong one. If the description omits the units or the date format, it will guess. Tool design is where most agent reliability is won or lost, and it is ordinary interface design rather than prompt craft.

Four rules carry most of the weight. Keep tools narrow and verb-shaped, so one tool does one thing. Return errors as sentences the model can act on: 'no customer with that id, try searching by email' is worth more than a status code. Make every write idempotent and give it an explicit identifier, so a retry cannot double-charge anybody. Cap the size of every response, because a tool that returns a 200 kilobyte payload will eat the context the agent needed for reasoning.

Put the permission check in the tool, not the prompt. A system prompt saying the agent must never issue a refund above one hundred pounds is a request. A refund tool that rejects anything above one hundred pounds is a control. This matters more than it sounds, because prompt injection is not hypothetical: content the agent reads during a run can contain instructions, and nobody has a reliable way to filter all of them. Assume the instructions in the prompt can be overridden and the ones in your code cannot.

Where does memory belong in the architecture of AI agents?

Split it into four things, because they have four different lifetimes. The live context window is working memory and it is gone when the run ends. A scratchpad the agent writes to, a file or a row in a table, survives compaction and restarts. Retrieval over documents is not memory at all, it is a tool that happens to read. Durable application state, the order or the ticket or the case, is the record that has to be correct whatever the agent did.

'We keep everything in context' is an answer that works until the third hour of a long run, when the window fills and something has to be dropped. Decide in advance what gets dropped. A running summary of what has been done, plus the current objective, plus the last few tool results, will carry an agent further than an undifferentiated transcript of every message.

The failure mode to watch for is an agent reading its own stale notes and treating them as current. Write timestamps, write where each fact came from, and re-read from the system of record immediately before any write. Memory that cannot be invalidated is a bug with a long fuse, and it tends to go off in front of a customer.

When does a multi-agent architecture earn its cost?

When the work is parallel and the context is separable. Anthropic reported that a lead agent on Claude Opus 4 with Claude Sonnet 4 subagents outperformed a single Claude Opus 4 agent by 90.2% on its internal research eval. The shape of that task explains the gain: breadth-first research, where several independent searches can run at the same time, each subagent holding its own context window, results compressed back to the lead.

Invert those two conditions and the advantage disappears. Work with tight sequential dependencies gets nothing from parallelism, and anything that needs agents to share evolving context pays a translation cost at every handoff. Subagents cannot see each other's reasoning, so a brief has to be complete at the moment it is written. A lead agent that tells three workers to 'research the market' will get three overlapping reports and a bill for all of them.

Which means the useful question is not how many agents to run. It is whether the task splits into pieces that do not need to talk to each other. If it does not split, one agent with a well-designed tool layer will beat a committee and cost a fraction as much.

How do you know the architecture works?

Run the same task many times, not once. Sierra's τ-bench evaluated function-calling agents built on twelve proprietary and open models across retail and airline domains, and scored them with a pass^k metric over repeated trials of identical tasks. The result that matters for architecture is the gap: consistency across repeated attempts is far worse than single-run success, so an agent that completes a task today will often fail the same task tomorrow with the same inputs.

That gap changes what you build rather than just what you report. It argues for retries that are safe because writes are idempotent, a hard step limit enforced in code, and a human checkpoint in front of anything irreversible. It also argues against a demo-led rollout, because the demo is a single run.

Teams feel this in the roadmap. In LangChain's survey of more than 1,300 professionals, performance quality came ahead of cost as the biggest limitation on putting more agents into production. The respondents skew small, with 51% at companies of fewer than 100 people, and a small team is exactly the population that cannot absorb a system nobody trusts.

It is also still early ground. Stack Overflow put its agent question to 31,877 developers in 2025, and agent use is not yet the default way software gets written, which means fewer settled patterns than the marketing suggests. Instrument accordingly. Trace every tool call with its arguments and its result, record tokens and step count for each run, and alert on the distribution of step counts rather than the average. The run that loops thirty times is the one that will hurt.

What should you decide before writing the first prompt?

• •What the hard step limit is, and what the system returns when it is reached.
• •Which tools can write, and what confirms a write before it happens.
• •What survives a crash halfway through a run, and what is simply lost.
• •Who sees the trace when a run goes wrong, and how quickly.
• •Which part of this needs a human in front of it permanently, not just during the pilot.
• •What the fallback is when the agent cannot finish, stated to the user in plain words.

What should the first version look like?

One model, one loop, three to six tools, a step limit, and a trace you can read without tooling. No framework until you have hit a problem a framework solves. Then run it against twenty real requests pulled from what users actually send, not twenty you invented, and read every trace by hand. Those twenty traces will tell you more about the architecture than any design document.

What they reveal is usually boring and structural: a tool description that was ambiguous, a missing tool the agent kept trying to improvise around, an error message the model could not act on, a request type that should never have reached an agent at all. Fix those, then ask whether you still need the orchestrator you sketched on day one. Often the answer is no, and that is the cheapest finding available.