Automated design of agentic systems is a research tool, not a product

The short version

• Automated design of agentic systems replaces the engineer who chooses an agent's prompts, control flow and tool use with a search loop that proposes candidate designs and scores them.
• The search can only optimise what the scorer measures, so the work moves from designing the agent to designing the evaluation.
• Published methods search over code rather than over a menu of options, so the winning design is often odd in ways that only a score can justify.
• Cost multiplies: every candidate design is executed against every task in the evaluation set, and the open-ended runs in the literature take weeks apiece.
• For most teams the cheaper move is to build the evaluation harness first and tune prompts inside an architecture a person chose.

What does automated design of agentic systems actually automate?

Three things, and it is worth keeping them apart. The prompt text. The control flow, meaning which model gets called in what order, with which tools, and when the loop stops. And, in the most ambitious version, the agent's own source code. Most work published under this phrase is the middle one: a search over structure, scored by a benchmark.

The loop has the same shape in every method. A proposer generates a candidate design. A runner executes that design against a set of tasks. A scorer grades the runs. The scores decide which candidates get mutated next. Change the proposer and the search strategy and you have most of the literature.

AFlow is the clearest one to read if you want to see the machinery. It represents a workflow as code, searches the space with Monte Carlo Tree Search, and reports results across six benchmark datasets against both hand-designed and automated baselines. It also reports inference cost in dollars rather than accuracy alone, which is the habit worth copying whatever you decide about the method.

None of this removes the design question. It moves the design question one level up. You no longer choose the workflow. You choose the task distribution, the scorer and the budget, and those three choices determine the workflow you get.

Does a searched design beat one an engineer would write?

On the benchmarks, yes, and by margins that are real rather than dramatic. The published searches find workflow structures that score better than the hand-built baselines they are compared against, and they find them without a person spending a fortnight on trial and error. That is a genuine result and it is why the field exists.

The conditions matter. A benchmark hands the search a fixed task distribution, a graded answer for every item, and a scorer that costs almost nothing to run. Your production task has none of those on day one. The search does not degrade gracefully when the scorer is noisy. It optimises the noise.

'It beat the baseline' also means it beat the baseline on the distribution it was searched against. A design selected on graded maths problems is a design tuned for graded maths problems. If your traffic is messier, which it is, the honest expectation is that you rerun the search on your own data or accept a design fitted to somebody else's.

What does one search run cost?

More than teams expect, because the cost multiplies rather than adds. Every candidate design has to be executed against every task in the evaluation set, and every execution is a full multi-step agent run with tool calls and retries. Cost is candidates times tasks times tokens per run, and the first of those three is the number nobody estimates.

The open-ended end of the field makes this concrete. The Darwin Gödel Machine, an agent that edits its own code and keeps an archive of its descendants, takes about two weeks of wall-clock time for one 80-iteration run on SWE-bench, and the authors publish their own cost accounting for that run in an appendix. Read it before you put a figure in a proposal. It is a research expense and the paper does not pretend otherwise.

You can estimate your own version in an afternoon. Measure the token cost of one complete run of your current agent on one real task. Multiply by the size of the evaluation set you would need before you trusted a result. Multiply again by the number of candidates the search will try before it settles. If that number is uncomfortable, note that the usual response is to shrink the evaluation set, which is how teams end up selecting an architecture on twelve examples.

Why does the evaluation harness decide the outcome?

Because the search cannot see anything you did not measure. The scorer is not a test, it is the specification. Whatever it rewards is what you will be handed, including the parts you would have rejected on sight if a person had proposed them.

METR's work on task length shows how much one measurement choice carries. Human experts with the relevant domain expertise were timed on RE-Bench, HCAST and 66 novel shorter tasks, then agents were scored on the same tasks to derive a 50% task-completion time horizon. The metric is defined at the point where the agent succeeds half the time, which is an honest way to report a system that fails often, and it makes the dependence on task length visible instead of burying it in an average.

The consequence for a buyer is specific. If your evaluation set is built from short, clean requests, the search will hand you a design that is excellent at short, clean requests and brittle on the long ones where the money is. Build the set from real traffic. Keep the cases that currently fail. Keep the ones where the correct output is 'I cannot do this', because a searched design will learn to guess if guessing scores.

Which level of automation are you actually considering?

When is automated design the wrong tool?

When you do not yet know what good looks like. That is most teams at the point they first read about this, and it is not a criticism. If nobody has written down what a correct response is for a hundred real requests, there is nothing for a search to optimise towards, and the first candidate it scores will be graded by a function somebody invented that morning.

It is also wrong when the task moves. A searched design is fitted to a distribution. If your documents, policies or product change every month, the fit expires, and you pay for the search again and again to chase a target a person could have followed for free.

And it is wrong when the output has to be defended. Some work gets explained to a regulator, a client or a court, and 'the search found it' is a weak answer. In those settings a plainer workflow that a person chose, and can justify step by step, is worth more than a few points of benchmark accuracy.

Signals you are not ready to automate the design

• •You have no graded set of real tasks, only a demo script.
• •Your scorer is a language model asked whether the answer looks right, and nobody has checked how often it agrees with a human.
• •The failures are expensive, so a design that is correct most of the time is not acceptable.
• •The task definition changes faster than a search run finishes.
• •Nobody on the team can read the generated control flow well enough to debug it on a bad day.

Who maintains an agent that a machine designed?

Your engineers, and they did not write it. This is the cost that never appears in a benchmark table. A searched workflow is code with no author to ask, and the shape it arrives at is often strange in ways justified only by a score: a duplicated review step, a loop that runs three times for reasons nobody can reconstruct.

Stack Overflow's 2025 developer survey, an online survey of 49,009 developers across 177 countries fielded between 29 May and 23 June 2025, was published under the finding that developer trust in AI is at an all-time low. Survey sentiment is not evidence about the code. It is evidence about the room you will hand the code to, and that is the room that gets paged when the agent loops at 2am.

There is a workable compromise. Treat the search output as a proposal, not a deployment. Have a person read the winning design, rewrite it as ordinary code in your repository, delete the steps that cannot be explained, and keep the harness the search needed. The harness is the durable asset. If you rerun the search in six months, it is the only thing that makes the comparison mean anything.

What should you ask before committing to this?

Ask about the scorer first and the search second. Every vendor conversation and every repository README puts the search strategy at the front, because that is the interesting part. The part that decides whether you get anything useful is the evaluation.

Questions worth asking a vendor or a repository

1. What exactly is being searched: prompts, workflow structure, or the agent's own code?
2. What grades a candidate, who built that grader, and how was it checked against human judgement?
3. What does one full search run cost in money and wall-clock time on our data, not on a benchmark?
4. What artefact do we own at the end, and can we read and run it without your runtime?
5. When our task changes, do we rerun the search, and who pays for that?
6. What was the second-best design the search found, and why was it worse?