AgentOps

What is AgentOps?

AgentOps is the practice of keeping AI agents working once they are live. It covers how agents are built, evaluated, released, watched, and corrected, treated as one loop rather than a pipeline, because an agent's behaviour is never finished changing.

The name follows a familiar pattern. DevOps gave software teams a way to ship and operate code continuously. MLOps did the same for trained models. LLMOps handled the specifics of large language models: prompts, context, evaluation, token cost. AgentOps extends that lineage to systems that do not just produce an output but choose an action.

The lineage is accurate and it is also where most explanations stop, which leaves out the part that actually changes how you work.

Why AgentOps has no release boundary

Every previous Ops discipline has a version you can pin. DevOps has a commit and a build. MLOps has a trained model artifact with a hash. You can point at the thing that is running and say what it is.

An agent's behaviour is determined by at least five things, and they change on different schedules:

• The model and its version, which the provider may update or deprecate on their own timetable.
• The prompt or instructions, which are often edited without a code deployment.
• The tools it can call, which can be added, removed, or changed behind a stable interface.
• The retrieved context, which changes every time the underlying data changes.
• The path it selects at runtime, which is chosen per request and is not fixed by any of the above.

Four of those five can change without anyone deploying anything. That is the structural difference, and it is why AgentOps looks like a loop rather than a pipeline. There is no moment at which an agent is finished and can be handed over. Asking "what version is in production" has no single answer, which is precisely why the answer has to be assembled from a record and from telemetry rather than from a release note.

How is AgentOps different from DevOps, MLOps, and LLMOps?

Each discipline manages a different kind of thing. DevOps manages code that behaves exactly as written. MLOps manages a model that behaves as trained. LLMOps manages a model that behaves as prompted. AgentOps manages a system that makes its own decisions, which is why supervision replaces one-time release as the main job.

These build on each other rather than compete. An organization doing AgentOps well is almost always doing the other three too, since agents still run on code, still call models, and still depend on prompts underneath.

The last column is the one worth dwelling on. Three of these disciplines have a terminal state that a team can reach and celebrate. AgentOps does not, and treating it as though it does is the most common way agent programmes get into trouble: an agent is launched, declared complete, and then quietly drifts as its model, its tools, and its data move underneath it.

What does the AI agent lifecycle include?

Agent lifecycle management runs through six stages: design, evaluate, release, observe, correct, and retire. It's drawn as a loop because observing an agent feed into correcting it, correcting it sends it back through evaluation, and nothing practically leaves the loop except by retiring, a step most organizations never take. That missing step is why agent estates only keep growing.

Agent lifecycle management and AgentOps are frequently used as synonyms. The distinction, if you want one, is that the lifecycle is the object and AgentOps is the practice of managing it. In everyday use they refer to the same work.

1. Design

   Decide what the agent is for, what it may reach, and where a human has to be involved. Scope decisions made here determine how much supervision the agent needs for the rest of its life, which makes design the cheapest place to reduce operational cost.

2. Evaluate

   Test how the agent behaves before it goes live, and keep testing it after. This stage is harder to do for agents than for models, for reasons the next section gets into, and it's where most AgentOps effort ends up going.

3. Release

   Move the agent through separate environments with a clear path into production, so there's an approval on record before it goes live. Because prompts and tools can change without a formal deployment, the release check has to cover configuration too, not just code.

4. Observe

   Track what the agent is up to: every step it takes, every tool it calls, every model call, the cost involved, and whether any guardrails kicked in. Observation isn't just for reporting here. It's the only real source of truth for how the agent is behaving right now.

5. Correct

   Act on what observation shows: adjust instructions, tighten permissions, narrow the tool set, change a threshold, or stop the agent. The distinguishing feature of a mature practice is that correction is routine rather than an incident.

6. Retire

   Decommission agents that no longer serve a purpose and withdraw their credentials and data access. This is the only exit from the loop and the stage most consistently absent, which is why agent estates grow monotonically.

Corrections loop back into evaluation, not into design, and that's what makes this a cycle rather than a straight line. A change to instructions or tools is a behavioral change, even if it looks small. Treating it as a minor config tweak instead of something that needs re-testing is exactly how agents quietly drift off course while still looking stable on the surface.

Why is evaluating an AI agent harder than evaluating a model?

A model can be scored against a labelled test set because a correct answer exists. An agent completing a multi-step task often has several acceptable paths and no single right one, so evaluation shifts from measuring accuracy to judging whether a trajectory was reasonable.

This is the part of AgentOps that has no clean equivalent in earlier disciplines, and it is where teams coming from MLOps are most often surprised.

Consider an agent asked to resolve a billing dispute. It might check the account, read the contract, look at payment history, and issue a credit. Or it might read the contract first, find a clause that settles the matter, and skip the payment history entirely. Both are defensible. A third path that issues the credit without reading the contract reaches the same outcome and is not acceptable. Outcome alone cannot distinguish them, so evaluation has to look at the path.

That drives three practical consequences.

• Accuracy stops being the primary metric. Useful measures become end-to-end task completion, the proportion of runs needing human intervention, and how often the agent took a path a reviewer would object to.
• Graded judgement replaces binary scoring. Rubrics, human review of sampled trajectories, and model-assisted grading all appear in practice. Each introduces its own reliability question, which is a real cost rather than a detail.
• Evaluation has to continue after release. A test suite passing before launch says little about behaviour three weeks later, because the model, the tools, and the data have moved. Continuous evaluation on live traffic is the norm rather than an advanced practice.

What should you trace in an agent's execution?

Seven artifacts make an agent's run reconstructable: the request, the plan, each tool call, retrieved context, each model call, guardrail decisions, and the outcome. Tracing fewer than these leaves gaps where a wrong answer cannot be attributed to a cause.

The academic treatment of this is worth knowing about. A 2024 taxonomy from CSIRO's Data61 set out the artifacts that should be traced across an agent's lifecycle to make observability possible, and the practical list below follows that logic. Source: Dong, Lu and Zhu, AgentOps: Enabling Observability of LLM Agents, 2024.

1. 01
   The request and its originWhat was asked, by whom or by what system, and under which identity and permissions. Without it: nothing downstream can be attributed to an accountable requester.
2. 02
   The planThe steps the agent decided to take, and any revision to that plan mid-run. Without it: you can see what happened but not what the agent was attempting.
3. 03
   Every tool callWhich tool, with what arguments, returning what, and how long it took. Without it: the reachable surface is unknown and a failing dependency looks like a failing agent.
4. 04
   Retrieved contextWhat was fetched and injected, and from which source. Without it: a wrong answer caused by a stale document is indistinguishable from a reasoning error.
5. 05
   Every model callModel, version, provider, tokens, latency, and cost per call. Without it: spend cannot be attributed and a provider change cannot be correlated with a behaviour change.
6. 06
   Guardrail decisionsWhich policies evaluated, which fired, and what they blocked or redacted. Without it: you cannot evidence that a control was applied, which an auditor treats as it not having been.
7. 07
   The outcome and any escalationWhat the agent produced or did, and whether it handed off to a person. Without it: completion rate cannot be measured, so nobody knows whether the agent is working.

Two notes on implementation. These are spans in a trace rather than log lines, because the useful questions are about a whole run and its structure. And the industry is converging on standard telemetry conventions for generative AI so that traces can move between tools, which matters if you would rather not commit your observability data to one vendor's schema.

How do you start doing AgentOps?

Start with tracing on one agent already in production, because you cannot improve behaviour you cannot see. Then add continuous evaluation, then a release gate covering configuration as well as code, then a retirement policy. Tooling choice matters less than this order.

Most teams arrive at AgentOps after a first agent reaches production and starts behaving in ways nobody can explain. That is a reasonable place to begin, and it suggests a specific sequence.

• Instrument before you organize. One agent, fully traced across the seven artifacts, teaches more than a programme designed around agents nobody is observing yet. It also produces the evidence needed to justify the rest.
• Define what good looks like, then measure it continuously. Write down what an acceptable run is for one agent, in enough detail that a reviewer could agree or disagree. Vague quality goals produce dashboards nobody acts on.
• Put a gate on configuration, not only on code. Because prompts and tool sets change without a deployment, a release process covering only code leaves the majority of behavioural changes ungoverned.
• Write the retirement policy early. It is trivial to write when you have three agents and politically difficult when you have ninety. Every quarter of delay makes it harder.
• Give each agent an owner on record. Operational practice needs somebody to route findings to. Where this record lives is covered in agent registry.

Which category of tooling supports AgentOps

No single category covers the loop. Agent observability and evaluation tools handle tracing, trajectory review, and graded scoring, and are usually the first purchase. Agent management platforms cover the registry, environment promotion, runtime policy, and cost attribution together, which suits estates needing governance alongside operations. Existing application performance monitoring vendors are adding generative AI tracing, which is attractive if your operations teams already live there. Open-source agent frameworks include some instrumentation, though it is generally development-grade rather than production-grade. The evaluation question is which stages of the loop a tool closes rather than which stages it reports on.

Two adjacent terms are worth separating. AgentOps is the operational practice. AI governance is the authority that decides what agents are permitted to do, and the two meet at the release gate and at runtime policy. An AI Agent Control Tower is where the observe and correct stages of the loop are actually operated.