OpenTelemetry for AI Agents Tracing: Spans That Help You Debug

OpenTelemetry for AI Agents Tracing: Spans That Help You Debug

To use OpenTelemetry for AI agents tracing, treat each run as one trace, and treat each model call, tool use and meaningful decision as a span with standardized attributes. Before anything goes to the collector, mask personal data, secrets and raw message content, and keep only what explains the agent's behavior. That way, when the agent gets something wrong, you can see the path it took without creating a new place for data to leak.

Key takeaways

• One agent run becomes one trace. Each model call, each tool and each routing decision becomes a child span with a start, an end and a result.
• Follow the OpenTelemetry semantic conventions for generative AI for attribute names, and use your own company-prefixed attributes for anything specific to your business.
• Record metadata by default (model, tool, duration, token counts, status) and message content only by exception, with masking.
• Apply masking as close to the source as possible, and use the collector as a second layer of protection, not the only one.
• Start with one real workflow and a defined debugging question, and only then widen the instrumentation.

Why scattered logs are not enough to debug an agent

Traditional software follows a predictable path: the same input leads to the same flow. An AI agent decides while it runs. It chooses which tool to call, with which arguments, how many times to retry and when to answer. The error is rarely in a single line of code. It is in the sequence: a query that came back empty, a model that ignored the empty result, and a confident answer at the end.

Scattered logs show events but lose the relationship between them. You find the tool's error message, but you do not know which model call triggered it or what the agent did next. A trace keeps that relationship of cause and timing. The OpenTelemetry documentation on observability concepts describes a trace as the path of a request through a system, made up of spans. That description fits an agent well, because the path is exactly the part that is hard to reconstruct.

There is also the problem of reproducing the failure. An agent rarely fails the same way twice. If all you have is the final output, you cannot reproduce it. With a trace, the run that failed is recorded as it happened, including the intermediate choices.

How to model an agent run as spans

The most useful design is usually a tree with few levels. The simpler it is, the easier it is to read in a hurry, when someone is investigating a complaint.

• Root span for the run: represents the whole task, from the incoming message to the reply or the handoff to a person. It carries the conversation identifier (already pseudonymized), the channel and the final outcome.
• Agent step span: each turn of the reasoning and action loop. This is what lets you count how many steps a run used.
• Model call span: one per request to the provider, with the model, relevant parameters, token counts and finish reason.
• Tool span: one per call to a function, API or MCP server, with the tool name, duration and status.
• Decision span: a short record of a choice that changes the course of the run, such as escalating to a human, refusing an action or asking for confirmation.

The OpenTelemetry semantic conventions for generative AI define attribute names for operations such as model calls, tool execution and agent invocation. They are still evolving, so check the documentation for the current names before you hard-code them into dashboards and alerts. In practice, that means isolating the names in a single module of your instrumentation so you can update them in one place.

One detail that helps: give spans names that make sense to people who did not write the code. Something like check_calendar tells the person running operations more than a generic function name does.

What to record in each span

The starting rule is to record what explains behavior, not everything that passed through the run.

On the model span, the attributes that earn their cost are: the requested model and the model actually used, input and output token counts, the finish reason and the error, if any. If cost is a concern, reading what tokens are in AI helps explain why that count deserves a place in every trace.

On the tool span, record the name, duration, status and a summary of the result as metadata: how many items came back, whether it was empty, whether it timed out. This summary is usually enough to find the cause without storing the returned content.

On the step span, record the step number within the run and the reason for the transition, for example tool requested, final answer or limit reached. This connects directly to the practice of step, time and budget limits for agents: if a limit is a system rule, the trace should show when it fired.

Finally, version whatever changes behavior: the version of the agent's instructions, the version of the toolset and a configuration identifier. Without these, you compare runs from before and after a change without knowing a change happened.

Decision spans: what makes a trace useful

The conventions cover models and tools well. The decision is where a trace stops being generic telemetry and becomes a debugging tool for your own process. Here you define your own attributes, with a prefix that marks them as yours.

A decision span can record which rule was evaluated, what the outcome was (allowed, blocked, asked for confirmation, handed off) and the reason as a closed category, not free text. Closed categories let you count and filter. Free text invites personal data to leak in and makes querying harder.

Examples of decisions that deserve a span: an answer issued without confirmation from a tool, an action with an external effect (sending a message, changing a record) that went through approval, an input flagged by a safety check, and a handoff to a human agent. The input, output and action checks described in guardrails for AI agents are natural candidates: each guardrail can emit a span with its verdict.

This layer also supports compliance. If the company needs to show how an automated decision was made and how it can be reviewed, a structured record of decisions is a concrete input. The topic is covered in LGPD automated decision-making. It is worth confirming with your legal team what has to be kept and for how long.

What to mask before sending to the collector

A trace is a new database holding a copy of what the agent saw. If you dump prompts and responses into it, you create a store of customer conversations with different access controls from the original. That is why masking is part of the design, not a later adjustment.

What should normally be masked or removed:

• Direct identifiers: name, tax ID (CPF), email, phone (including the WhatsApp number), address and payment data.
• Raw content of customer messages and agent replies, which can contain any personal data typed freely.
• Tool arguments and results that carry customer records, such as CRM or ERP responses.
• Secrets: API tokens, keys, authorization headers and connection strings that show up in arguments or errors.
• Passages from documents retrieved by search, when the document is internal or sensitive.
• Error messages from third parties that echo the original input.

The alternative to full masking is pseudonymization: replacing the identifier with a derived value, with the secret kept outside the telemetry, so you can correlate runs from the same conversation without exposing who the person is. Decide this together with whoever is responsible for privacy in your company.

On message content, the generative AI conventions treat capturing inputs and outputs as optional and sensitive. Keep it off by default, and turn it on temporarily, on a sample and in a controlled environment, when an investigation requires it. If the observability platform is outside Brazil, remember that sending the trace is a data transfer; the article on international data transfer under LGPD when using AI providers shows what to check. The ANPD, Brazil's data protection authority, is the official reference for guidance.

Think about the security of the content itself, too. The OWASP Top 10 for LLM applications lists risks such as sensitive information disclosure and manipulation through malicious instructions. A trace with raw text can carry a malicious instruction to whoever reads it with the help of another AI. Treat trace content as untrusted data.

Technically, apply masking in two layers. The first sits in the application itself, before export, using a span processor or a function that only allows attributes from an approved list. The second sits in the collector, with processors that remove, replace or hash known attributes. An allowlist is safer than a blocklist: whatever was not approved simply does not leave.

A step-by-step plan and an illustrative scenario

A lean path to get started from zero:

• Pick one real workflow and one debugging question, such as why the agent answered without confirming the calendar.
• List the attributes needed to answer that question and drop the rest.
• Define the allowed attribute list and the masking rules, approved by whoever owns privacy.
• Instrument the root, the steps, the model, the tools and the decisions.
• Configure the collector with the second layer of protection, and define retention and access.
• Test with synthetic conversations containing fake data and confirm that nothing sensitive shows up at the destination.
• Review after a period of use: what nobody queried comes out, what was missing goes in.

Illustrative scenario: a store uses a WhatsApp agent to book visits. A customer complains that the agent confirmed a time slot that did not exist. In the run's trace, the root span shows the channel and the final outcome. The check_calendar tool span shows a success status, but with an empty result. The next step span shows that the model generated the answer anyway, and the decision span records that the reply was issued without confirmation from the tool. The cause becomes clear in a few minutes: the date filter was wrong, and the agent's instructions did not forbid confirming on an empty result. The fix is a decision rule, and the same trace serves as a regression test. None of this required storing the text of the conversation.

How our method applies to this work

The method follows the order map, limit, pilot and hand over. Mapping is understanding the current process and choosing what needs to be explainable. Limiting is defining the data and actions the agent touches, which also defines what the trace may contain. Piloting is instrumenting one real workflow with an agreed success metric; the guide on how to measure an AI automation pilot helps define the criteria. Handing over is giving the team the dashboards, the masking rules and the investigation procedure, so the decision to expand, adjust or stop is informed by evidence.

Frequently asked questions

Do I need to record the full prompt to debug an agent? In most cases, no. Metadata such as model, tokens, tool called, status and decision category is usually enough to find the cause. When the text is indispensable, capture it temporarily, on a sample and already masked.

Does OpenTelemetry work with any model and agent framework? OpenTelemetry is vendor-neutral, and you can instrument any workflow manually, including ones that use local models. Some frameworks and libraries already ship with instrumentation, but check what it sends by default, because it may include message content.

Does a trace count as personal data under LGPD? If it contains information that identifies a person, directly or indirectly, yes, and it then requires a legal basis, defined retention and access control. That is why the ideal is to design the trace so it does not contain such data, and to validate the decision with the person responsible for privacy.

Where should a company start if it has never used tracing? Start with a single workflow, a few span types and a short list of allowed attributes. Make the trace answer one real debugging question, and only then add detail. Observability that nobody looks at becomes a cost and a risk with no return.

Sources

• OpenTelemetry: conceitos de observabilidade
• OpenTelemetry: convenções semânticas para IA generativa
• OWASP Top 10 para aplicações com LLMs
• ANPD: Autoridade Nacional de Proteção de Dados