What are multimodal AI agents, and when are they worth building?

The short version

• A multimodal AI agent takes in more than one kind of input, such as text, images, audio or a screen, and acts through tools.
• A model that describes a photo is multimodal but is not an agent, because it takes no action.
• In a multimodal AI agent, a misread input becomes a wrong action, often with no error raised.
• Testing a multimodal AI agent means scoring what it read separately from what it did.
• A multimodal AI agent is not worth building if the data already exists as reliable text or through an API.

What is a multimodal AI agent?

A multimodal AI agent is a system that takes in more than one kind of input, decides what to do, and carries it out through tools. The inputs can be text, images, audio, video or the contents of a screen. The tools can be anything with an interface: a booking system, a database, a browser.

Two separate properties are being described. Multimodal is about perception: what the system can read, see or hear. Agent is about action: whether it chooses steps and carries them out, or only replies.

A model that describes a photo is multimodal but is not an agent. A text-only assistant that updates a customer record is an agent but is not multimodal. The term only applies when both are true.

How is a multimodal AI agent different from a multimodal model or a chatbot?

The difference is what happens after the input is understood. A multimodal model returns a response. A chatbot holds a conversation. A multimodal AI agent goes on to do something, then looks at the result and decides whether it needs another step.

That last part carries most of the risk and most of the value. A person reading a wrong reply can ignore it. A wrong action has already happened by the time anyone reads about it.

How does a multimodal AI agent work?

A multimodal AI agent works as a loop: it captures an input, interprets it, chooses a step, calls a tool, then checks the result. A text agent runs the same loop. The difference is that the first stage now produces an interpretation of the world, not an exact copy of it.

There are two common designs. In a conversion design, a separate component turns the input into text first, such as speech to text on a phone call, and a text agent takes it from there. In a native design, the model takes the audio or image directly.

Conversion gives you an intermediate transcript that you can log, read and test on its own, at the price of another stage that can add delay and errors. Native input removes that stage, but leaves less in the middle to inspect when something goes wrong. Neither is correct in general. For a voice agent, where delay is audible, the choice usually comes down to how much delay callers will tolerate against how much you need to inspect.

What can multimodal AI agents do in practice?

In practice, each kind of input suits a different kind of work and fails in its own way. Images and scans suit work that arrives on paper or by phone camera. Audio suits phone calls. Screen input suits software with no usable API. Video is rarely the place to start, because it carries the most data and is the hardest to test.

Why do multimodal AI agents fail differently from text agents?

They fail differently because the input itself can be wrong, and the agent has no way of knowing. Text reaches a text agent exactly as written. An image or a call reaches a multimodal agent as an interpretation, and the rest of the loop treats that interpretation as fact.

So a perception error becomes an action error. Say a scan reads 1,250 and the agent reads 7,250. Every later step is valid. The tool call succeeds, the log looks clean, and no exception fires.

There is a second difference. Every new input is a new way in for instructions the agent should not follow. Text in an image, a line on a web page in a screenshot, or a spoken sentence can all reach the model as content it may act on. Treat anything the agent perceives as untrusted, and keep the permission to act somewhere the input cannot reach.

How do you test a multimodal AI agent?

Test it on real, messy inputs, and score perception separately from the decision. If you only check the final outcome, you cannot tell whether the agent misread the input or reasoned badly from a correct reading, and those have different fixes.

1. Collect real inputs: poor scans, phone photos, regional accents, noisy lines, the screens your staff actually use.
2. Write down the correct reading for each one, then score the agent's reading on its own.
3. Score the action separately: given a correct reading, did the agent take the right step?
4. Check the result in the target system, not the agent's account of what it did.
5. Keep the original input with every run, so a bad outcome can be replayed.
6. Require a person to approve anything hard to undo, until the scores justify removing that check.

How should you read benchmarks for computer-use agents?

Read the task length and the scoring rule before the score. A computer-use agent works from what is on the screen, so it is a multimodal agent in the plain sense, and its benchmark results only mean something for tasks shaped like the ones tested.

OSWorld 2.0 is a useful example. It covers 108 long-horizon computer-use workflows across seven professional domains, scored on binary completion at 500 steps. The tasks are deliberately long, with a median of about 1.6 hours for skilled users. Binary scoring means a run that gets most of the way there still counts as a failure.

The best result the page reports came from Claude Opus 4.8 with maximum thinking and batched tool calls, and the authors appear to have run the evaluation themselves. Its scores are not comparable with scores on the shorter OSWorld-Verified benchmark. A figure lifted from one and set beside the other tells you nothing.

What does a multimodal AI agent cost to run?

It costs model fees plus everything around them, so a price list only gives you one line of the bill. For a voice agent, telephony, integration and hosting all sit on top of the model fee.

As one reference point, Google lists Gemini 3.8 Live on its paid tier at $0.75 per 1M tokens for text input, and at $12.00 per 1M tokens or $0.018 a minute for audio output. That was read in November 2026 and the page shows no last-updated date, so check it before budgeting. At that rate, ten minutes of spoken output comes to 18 cents in model fees, before anything else.

Two things follow. Cost a voice agent by the minute and by the call, because a per-token text estimate will not map onto how a phone conversation runs. And cost the build and upkeep, not only the running fees, because integration and testing do not appear on a model price list.

When is a multimodal AI agent not worth building?

It is not worth building when the information is already available as reliable text or structured data, or when nothing needs doing after the reading. Perception you do not need is cost and error you have chosen to add.

• •The software has a usable API. Call it. Driving the screen is slower and breaks when the layout changes.
• •The PDF has a text layer. Extract the text before paying a model to look at a picture of it.
• •The job ends at extraction. A fixed pipeline that reads a form and writes a row does not need an agent.
• •Customers would happily fill in a form. A form returns exact values, and no model has to guess them.
• •A wrong reading is costly and nobody can check it. Add the check first, or do not automate it yet.

How widely are multimodal AI agents used?

We do not have a reliable figure for multimodal AI agents specifically, and we would treat a confident one with suspicion. The surveys we can point to measure something nearby, and each has a limit you should know before quoting it.

The US Census Bureau's Business Trends and Outlook Survey asks whether a business used AI in the past two weeks, which covers AI generally, not agents. Menlo Ventures surveyed 495 US enterprise AI decision-makers at companies already using AI tools, and Menlo is an AI investor. Anthropic's report on AI agents draws on over 500 US technical leaders, and it is the vendor's own survey.

AssemblyAI is closest to the topic, with a survey of 455 voice agent builders run across Q4 2025 and Q1 2026. Of those respondents, 87.5% were actively building, which describes the sample, not the market. Its sampling method and margin of error are not stated, and AssemblyAI sells speech-to-text infrastructure.