How to Audit AI Agent Conversations: Sampling and a Simple Rubric

How to Audit AI Agent Conversations: Sampling and a Simple Rubric

To audit AI agent conversations, pull a fixed-size sample every week that mixes random conversations with risk-flagged ones, and score each with a short three-axis rubric: accuracy, tone and risk. An audit is only worth the time if it leads to change. Every recurring failure needs an owner, a fix to the instructions, the knowledge base or the agent's limits, and a re-check the following week.

Key takeaways

• Dashboards show volume and response time, but not whether the answer was right. Only reading conversations shows that.
• The weekly sample should be a size your team can really review, with one part random and one part chosen by risk signals.
• A three-axis rubric (accuracy, tone and risk) with three levels each is enough to start and keeps reviewers aligned.
• Every finding should be classified by cause (knowledge, instruction, limit, tool or flow) before it becomes a fix.
• Serious risk failures do not wait for the weekly routine. They become incidents the same day.

Why read conversations if you already have a metrics dashboard?

Aggregate metrics answer operational questions: how many conversations, how many were handed off, how long they took. They do not tell you whether the agent quoted a wrong deadline, was curt with an irritated customer or asked for personal data it did not need. Two conversations can have the same response time and opposite outcomes for the customer.

Anthropic's documentation on evaluating agents treats reading transcripts as part of the evaluation work, not an optional step. That holds up in practice: the real conversation is where you find the question nobody anticipated, the way customers actually write on WhatsApp, and the point where the agent gets lost.

Conversation audits are also a management control. The NIST AI Risk Management Framework organizes AI risk management into functions such as govern, map, measure and manage. The weekly routine described here is, in practice, the measure and manage part: look at what the system did and decide what to change.

How do you build the weekly sample?

The most common mistake is auditing only what showed up by chance or only what generated a complaint. The first approach misses silent failures. The second gives a distorted view, because it only sees what has already failed visibly.

A balanced sample combines two groups.

• Random sample: conversations drawn from all of the week's conversations, with no filter. It shows average quality and the failures nobody reported.
• Targeted sample: conversations chosen by risk signal. This includes conversations handed off to a human, those that ended with an unhappy customer, very long ones, those that triggered an action (booking, canceling, changing a record) and those touching topics that are sensitive for your business.

Set the sample size by review capacity, not ambition. Pick a number one person can read carefully in a single work session and keep it fixed for a few weeks so you can compare results. If the review turns into a marathon, it stops happening.

Two adjustments are worth making. After any change to the agent (instructions, knowledge base, tools), temporarily increase the targeted portion, because that is when regressions appear. And record where each conversation came from in the sample, so you know whether a failure shows up in the random draw or only in the cases you picked.

Which simple rubric should you use: accuracy, tone and risk?

A long rubric does not get used. Three axes, each with three levels (good, adjust, serious), are enough for a first phase and let two reviewers reach similar scores on the same conversation.

Accuracy: was the answer correct, and did it resolve the request?

• Good: the information matches the company's official source, and the request was resolved or routed correctly.
• Adjust: the answer was partly right, incomplete, or made the customer ask again.
• Serious: invented, outdated or policy-contradicting information.

Tone: did the reply sound the way the company wants to sound?

• Good: clear, short, and suited to the channel and the customer's situation.
• Adjust: correct, but too long, cold, repetitive or full of jargon.
• Serious: ignored the customer's frustration, was abrupt, or kept pushing a flow that was not helping.

Risk: did the conversation expose the company to any harm?

• Good: stayed in scope, made no improper promises, and handed off when it should have.
• Adjust: came close to a limit, for example offering opinions outside scope or asking for more personal data than necessary.
• Serious: promised a deadline, price, guarantee or condition without basis, ran a sensitive action without confirmation, exposed another person's data, or ignored a request to speak to a human.

Add a one-line free-text field for the probable cause. It is the bridge between the score and the fix. For decisions that significantly affect the customer, it is also worth reading about LGPD automated decision-making and the right to review, because auditing is part of your ability to review what the system decided.

What does the weekly routine look like, step by step?

The routine fits in a few hours spread across the week and needs a named owner. Without an owner, it disappears by the second busy week.

• Pull: at the start of the week, generate the list of last week's conversations and draw the random portion. Then add the targeted portion based on risk signals.
• Read and score: read each conversation in full, not just the last message. Give one score per axis and write the probable cause in one line.
• Handle serious items first: any serious risk score becomes an incident the same day. Notify the owner, fix what is urgent and, if needed, pause the affected flow.
• Group by cause: cluster the findings (knowledge, instruction, limit, tool, flow, should have been human). A repeating pattern weighs more than one dramatic isolated case.
• Pick a few fixes: prioritize a short list of changes for the week. Changing too much at once makes it impossible to know what worked.
• Apply and record: make the change, note the date and the reason, and keep the problem conversations as test cases.
• Re-check next week: see whether the pattern is gone from the new sample. If it is not, the fix did not work.

If you log agent telemetry, the OpenTelemetry semantic conventions for generative AI are a useful reference for how to structure what you store from each call. Even without that level of formality, the essentials are being able to reconstruct the conversation, the sources consulted and the actions taken.

Illustrative scenario: a WhatsApp agent at a services company

Illustrative scenario, not based on any real case. A home maintenance company uses a WhatsApp agent to answer questions and pre-book visits. In the weekly review, the owner finds three patterns.

First, in several conversations the agent says the visit will happen within a timeframe that appears in no company document. Accuracy: serious. Risk: serious. Probable cause: a knowledge gap, where the agent filled the void with an assumption. Second, customers who write in an irritated tone get a long reply full of caveats. Tone: adjust. Cause: the instructions have no guidance for complaints. Third, a customer asks twice to speak to a person and the agent keeps trying to resolve the issue itself. Risk: serious. Cause: a weak handoff rule.

The week's fixes look like this: record the real visit timeframe in the knowledge base and instruct the agent to say it will confirm when the information does not exist; add two example replies to the instructions for irritated customers, one good and one bad; and create a clear rule that an explicit request for a person ends the automation attempt and triggers the handoff. The three conversations become test cases for the next round.

How do you turn findings into fixes?

The risk with an audit is producing a polished report and no change. The way to avoid that is to decide by cause, not by symptom. Use this decision guide.

• The correct information is not in the knowledge base: fix or create the source. Tweaking the instructions so the agent answers better without having the data only hides the problem.
• The information exists, but the agent did not use it: review how the content is organized and retrieved, and only then the instructions.
• The tone is repeatedly off: adjust the instructions with concrete examples of good and bad replies. See how to write instructions for a support agent.
• The agent promised something it could not deliver: create an explicit limit and an output check before sending. The article on guardrails for AI agents explains how to separate input, output and action controls.
• The customer needed a person and never reached one: define handoff triggers, such as an explicit request, repeated dissatisfaction and a sensitive topic. For this kind of signal, see sentiment analysis and when to hand off to a human.
• The action taken was wrong: restrict the tool, require customer confirmation or human approval before actions that cannot be undone.
• The failure happened once, with no pattern: record it and watch the next samples. Reacting to every isolated case destabilizes the agent.

After each fix, run the problem conversations as test cases before you publish the change. OpenAI's evals documentation and Anthropic's writing on agent evaluations describe this logic: turn observed failures into repeatable tests, so the same error does not return unnoticed.

How do you handle privacy, reviewers and working method?

Support conversations contain personal data. Whoever audits needs justified access, and that access should be the minimum necessary. Define who can read, how long conversations stay available for review, and whether names, phone numbers and documents can be masked on the audit screen. Keep the purpose clear: the audit exists to improve the service, not to evaluate the person who handled it. For what the authority says about data processing, the ANPD (Brazil's data protection authority) website is the starting point. If the data is processed outside Brazil, also check the LGPD requirements for that case in international data transfer under LGPD when using AI providers.

In Juan Carlo's working method, auditing is a consequence of four steps. Map: understand the current process and record a baseline before automating. Limit: define which data and actions the agent may use, which gives the risk rubric concrete criteria. Pilot: run a real flow, with decision logs and an agreed success metric, which feeds the weekly sample. Transfer: hand the review routine to the client's team and decide, based on what the conversations showed, whether to expand, adjust or stop. To measure the pilot's result with decision criteria, see how to measure an AI automation pilot.

Frequently asked questions

How many conversations should I review per week? There is no universal number. Choose a quantity the responsible person can read carefully in one session and keep it fixed for a few weeks so you can compare. If conversation volume grows, increase the risk-targeted portion before the random one.

Who should run the audit, the support team or the technical team? Ideally a pair. People who know the support work score accuracy and tone more precisely, and people who know the system can tell whether the cause is in the instructions, the knowledge base or the integration. Name one person as the owner of the routine so it does not depend on goodwill.

Can I use another AI to audit the conversations? It can help with triage, for example by flagging conversations with a possible improper promise or an unhappy customer, to feed the targeted sample. But the final score on serious cases and the decision on what to change should be human, and it is worth checking periodically that the automated triage agrees with a person's review.

Does the audit require telling the customer? It depends on what your privacy notices say and the legal basis used for the processing. Customers should know they are talking to an AI agent and that conversations may be analyzed to improve the service. Align the wording with whoever handles LGPD in your company and revisit it when the purpose changes.

Sources

• Anthropic: Demystifying evals for AI agents
• NIST: AI Risk Management Framework
• OpenTelemetry: semantic conventions for generative AI
• OpenAI: Evals guide