How to Build an Evaluation Dataset for AI Agents From Real Chats

How to Build an Evaluation Dataset for AI Agents From Real Chats

To build an evaluation dataset for AI agents, start from real support conversations: pick a sample that represents the daily routine and the hard cases, remove personal data, and label each case with the expected outcome and a pass criterion. The set becomes the agent's regression test, and every change to a prompt, model or tool runs against it before going to production. Start small, but only with situations that actually happened.

Key takeaways

• Real conversations show how customers write, make mistakes, change the subject and send voice notes. Invented cases are almost always too polite.
• The sample must mix common, edge and risky cases, not just the examples that are easy to get right.
• Anonymizing means deciding what the test needs to keep (intent, context, tone) and removing the rest (name, phone, ID number, address).
• Each conversation gets a label with the expected outcome, what the agent must not do, and when it should hand off to a person.
• The set is not a fixed file: it grows with every failure found in production and is reviewed by people who know the process.

Why use real conversations instead of invented cases?

When a team writes its own test examples, it writes the way it imagines customers talk. Real customers send three questions in one message, reply with something unrelated to the previous question, abbreviate, use regional slang, send a blurry photo and disappear for two hours before coming back. An agent that passes clean examples can fail in exactly these situations.

There is also a bias effect. Whoever creates the test knows the prompt and tends, without noticing, to write cases the agent already knows how to solve. Real conversations do not have that habit. They bring the questions nobody anticipated when the instructions were written.

The agent evaluation documentation published by model vendors follows the same logic: begin with real tasks, define clear success criteria, and expand the set as failures appear. It is worth reading Anthropic's guidance on evaluating agents and OpenAI's evals guide to see how each one structures tasks, runs and graders.

How to select conversations: common, edge and risky

The most frequent mistake is taking the latest conversations from the history and calling that a sample. The latest conversations reflect one specific week, one specific campaign, maybe a temporary problem. Build the selection by category instead, looking at a longer period.

Use this decision guide to classify each candidate conversation:

• Common case: a frequent question with a clear answer (opening hours, order status, duplicate invoice, simple booking). These cases protect the basics and should be most of the set.
• Edge case: the request is legitimate but leaves the script (two intents in one message, a customer who corrects information midway, a request outside the service area, a message with only audio or an image).
• Ambiguous case: data is missing and the agent should ask, not guess.
• Risky case: a complaint with a threatening tone, a refund request, sensitive data, a legal question, health or safety. The expected outcome is almost always a handoff to a person.
• Manipulation attempt: the customer asks the agent to ignore rules, reveal its instructions or promise something outside policy. These cases connect to the topic of guardrails, covered in guardrails for AI agents.
• Case with no answer in the material: the information does not exist in the agent's knowledge base. The correct behavior is to admit it and hand off.

If volume allows, ask the people who handle support every day to flag the conversations that gave them the most trouble. That list tends to reveal edge cases no automatic search would find. Also include conversations the human team handled poorly, because they show where the process itself is confusing and where the expected label needs a business decision.

How to anonymize without destroying context

The starting question is: what does the test need to be useful? In general, it needs the customer's intent, the sequence of messages, the tone and the facts relevant to the decision (for example, that the order was placed a few days ago, or that the product is out of warranty). It almost never needs the real name, phone number, CPF (Brazilian individual taxpayer ID), or address.

A simple anonymization routine:

• Remove or replace direct identifiers: name, phone, email, CPF, CNPJ (Brazilian company ID), address, order number and license plate.
• Swap them for consistent placeholders within the conversation, such as CUSTOMER_A, PHONE_1 and ORDER_X, so the thread still makes sense.
• Review the free text, where personal data tends to hide: signatures, children's names, workplace, health details, photos of documents.
• Handle media separately. Audio and images carry voice, faces and documents. If the test needs them, use the already cleaned transcript. For care with this kind of content, see transcribing WhatsApp voice messages with AI.
• Have a second person, different from the one who anonymized, read everything before the set enters the repository.

One legal point to watch: under LGPD (Brazil's data protection law), data that is truly anonymized, meaning the person cannot be re-identified by reasonable means, falls outside the law. Data that is only pseudonymized (replacing the name with a code while keeping a lookup table) is still personal data. In practice, treat the test set as personal data until your privacy team confirms the anonymization is sufficient. If the set is sent to a model provider outside Brazil to run the evaluations, check what we covered in international data transfer when using AI providers. The ANPD, Brazil's data protection authority, publishes official guidance and materials on the subject.

Anonymizing also means limiting access. Who can read the raw set, who can read the anonymized set and how long the raw version is kept are decisions that should be written down before the first file is created.

How to label: what does a correct answer mean in each conversation?

Labeling is not writing a perfect answer to compare word for word. An agent can answer well in many ways. The label should describe what must be true in the answer and what must never happen.

A lean label sheet for each conversation:

• Context: what the agent knows at the start (account data, current policy, service hours).
• Customer intent: what the person actually wants, in one sentence.
• Expected outcome: answer, ask for missing data, perform an action or hand off to a person.
• Required information: facts that must appear in the reply.
• Prohibitions: promises, amounts, deadlines or diagnoses the agent must not invent.
• Handoff condition: the moment a human takes over and with what summary.
• Case category: common, edge, ambiguous, risky or manipulation.

The most valuable part is usually the list of prohibitions. It forces the business to say out loud what the agent cannot do, and that often reveals rules that lived only in the head of an experienced support person. If two people in operations disagree on the label for a conversation, do not hide the disagreement: it is a sign the policy needs to be defined, and the case becomes a good test item once the decision is made.

The pass criterion can be checked in three ways: an automatic rule (the reply contains or does not contain certain information, the right action was called), evaluation by another model following a rubric, or human review. Use automatic rules where you can and keep human review for judgment cases. If you use a model as the grader, sample the results and check them by hand, because the grader also makes mistakes.

Step by step to build the first set

A plan that fits in a few weeks of part-time work:

• Define the flow to be tested. Pick just one (for example, booking or order status), not the entire support operation.
• Gather the base: export conversations from the period, only from the chosen channels and flows, with restricted access.
• Classify by category using the guide from the previous section and cut duplicates, keeping real variations in wording.
• Anonymize and do the second read.
• Label with the sheet, with input from the people who run the process.
• Set aside a small part as a holdout set that nobody uses to tune the prompt. It checks whether the agent really improved or just memorized the examples.
• Run the current agent and record the baseline: where it succeeds, where it fails, where it hands off too often.
• Fix the labels that turned out to be ambiguous, and only then start changing the agent.

The baseline step is the one most often skipped and the one most missed. Without it, any improvement is an impression. With it, the conversation with leadership changes from we think it got better to this group of cases now works and this one still fails.

Illustrative scenario: a clinic that reschedules appointments on WhatsApp

Illustrative scenario, not a real customer. A small clinic wants an agent that reschedules appointments on WhatsApp. The team exports rescheduling conversations from several months and sorts them by category.

The common cases are direct requests to change a time. Among the edge cases, one patient asks to change the day and, in the same message, asks whether the exam requires fasting, and another says to reschedule for next week without naming a day. There is an ambiguous case where the message says only I need to change, with no context. Among the risky ones, a patient reports a strong symptom and asks for an earlier appointment, and the expected outcome is an immediate handoff to a person, with the agent offering no opinion on the symptom. There is also a manipulation case, in which someone asks the agent to share another patient's appointment times.

During anonymization, names become PATIENT_A, and health details are generalized whenever the test does not depend on them. During labeling, the prohibitions list includes giving clinical advice, confirming a time that is not on the schedule and revealing third-party data. When the team runs the agent, it finds that the agent handles common cases well but tries to advise on fasting without having that information. The finding becomes an instruction fix and a permanent case in the set.

How to keep the set alive and tie it to operations

A static test set ages. Policies change, products change, customers start writing differently. Define a simple ritual: every relevant production failure, reviewed by a person, becomes a new labeled case; every change to a prompt, model, tool or knowledge base runs the set before release; and from time to time, obsolete cases are reviewed or removed.

The set also needs to connect to business indicators. It measures whether the agent behaves well in known situations, but it does not replace tracking resolution, handling time and satisfaction in production, the subject of customer support KPIs with AI. The two views complement each other: the dataset warns you before release, the indicators confirm afterward.

In the method we use, this fits into four moves. Map: understand the current process and record the baseline with real conversations. Limit: decide which data enters the set and which actions the agent never performs alone. Pilot: run one real flow with logs and an agreed success metric, using the set as the gate for each change. Transfer: hand the set, the label sheet and the update ritual to the team, which then decides whether to expand, adjust or stop. The dataset is part of what gets delivered, not backstage material.

Frequently asked questions

How many conversations do I need to start? Start with enough to cover each case category in the chosen flow, so you can read and discuss every one with the team. A small, well-labeled set is worth more than a large, confusing one. Expand as new failures appear.

Can I use customer conversations to test the agent without asking them? It depends on the stated purpose, the legal basis you adopt and the level of anonymization applied. Treat the material as personal data until your privacy lead validates the process, and keep your privacy notice consistent with this use. When in doubt, consult ANPD guidance and your legal counsel.

Do I need a specific evaluation tool? Not to begin with. A spreadsheet or structured file with the label sheets and a script that runs the agent on each case is enough for the first cycle. Evaluation tools help later, when the volume of cases and runs justifies automation.

Does the test set guarantee the agent will not make mistakes in production? No. It reduces the chance of known regressions and gives you evidence to decide, but customers will always bring new situations. That is why the set should be combined with logs, human review of samples and clear rules for handing off to a person.

Sources

• Anthropic: Demystifying evals for AI agents
• OpenAI: Evals guide
• ANPD: Autoridade Nacional de Proteção de Dados