Incident Response Plan When an AI Agent Makes a Mistake

Incident Response Plan When an AI Agent Makes a Mistake

When an AI agent makes a mistake with a customer, the right order is: contain the error, fix what the customer experienced, communicate clearly, and only then investigate the cause and adjust the system. Teams that start with technical diagnosis leave the customer waiting while the error keeps repeating. This article describes an incident response plan your company can write before it needs one.

Key takeaways

• Contain first: pause the action or flow that is failing, not your entire support operation.
• Fix the effect on the customer before investigating the cause: undo the action, deliver what was owed and apologize plainly.
• Communicate in simple language: what happened, what has already been done and who owns the case.
• Log everything. Without a record of the conversation, the decision and the tools called, the review turns into opinion.
• After the error, change the system (limit, rule, test, permission), not just the prompt text.

Why write the plan before the error happens?

Every AI agent in production will get something wrong eventually. It might invent an answer, quote an outdated policy, run an action in the wrong system or send a message to the wrong person. The difference between a small incident and a reputation problem usually comes down to what the team does in the first few hours, and improvising under pressure produces bad decisions.

An incident response plan does not need to be long. It needs to answer a few questions in writing, with names and channels:

• Who owns the incident, and who covers when that person is unavailable?
• Who has authority to pause the agent or a specific tool, and how is it done?
• Who decides on compensation, refunds or exceptions for the affected customer?
• Where are the records of the agent's conversations and actions stored?
• Who talks to the customer, and with what base text?

Also define a simple severity scale based on consequence, not technical cause. A minor error is an imprecise answer with no harm. A moderate error creates rework or real frustration. A severe error involves money, personal data, health, safety or a promise the company cannot keep. Severity determines who joins the response and how urgently.

What the agent may or may not do alone should already be decided. If that boundary is still vague in your company, read what not to delegate to AI in business before writing the plan.

How do you contain the error without shutting down all support?

Containment means stopping the error from continuing. The error is almost always local: one topic, one tool, one type of message. Switching everything off punishes customers who were being served well, and switching nothing off lets the problem grow. That is why the plan needs small switches, built before the incident.

Use this decision guide as a starting point:

• Wrong information with no action taken in a system: block the topic or the affected part of the knowledge base, and route that subject to a person until it is corrected.
• Wrong action in a system (cancelled, changed, refunded, sent): revoke write permission for that tool and leave the agent in read-only mode.
• Message sent to the wrong person or personal data exposed: pause automatic sends and alert whoever is responsible for personal data at the company.
• Same error appearing for several customers: move the whole flow to human handoff until you understand the pattern.
• Promise outside policy, inappropriate tone or offensive behavior: pause the agent on that topic and review the instructions and output limits.

After you flip the switch, preserve the evidence before changing anything. Export the conversation, the tool calls, and the versions of the instructions and knowledge base that were active. If you fix first and collect later, you lose the chance to understand what really happened.

Whoever operates the agent must be able to pause it without opening a ticket with the technical team. For WhatsApp support, that usually means a simple switch that diverts the conversation to the human queue. For the input, output and action validation logic that lowers the chance of an error reaching the customer, see guardrails for AI agents.

How do you fix what the customer went through?

The fix is about the customer, not the model. While the technical cause is still unknown, the company can already solve the person's real problem. A practical sequence:

• Identify everyone affected. Search the records for other conversations with the same situation, topic or tool during the error window. The customer who complained is rarely the only one.
• Undo what can be undone: cancel the wrongful action, resend the correct information, restore the record, reopen the order.
• Decide what the company will honor. If the agent promised something, a person with authority decides whether the promise will be kept, adjusted or declined. In many cases, what support tells a consumer can be treated as a commitment by the company, so align with legal counsel before setting a general rule.
• Set compensation with human judgment. A gift, a discount, priority handling or a simple correction is a decision for people, not for the agent that made the mistake.
• Assign the case to a named person. The customer needs to know who to talk to.

A common mistake is asking the same agent to repair the situation. If it failed for lack of context or limits, it is likely to fail again in the remedy. During the incident, handle the case through a human channel.

What do you tell the customer, and how?

The customer wants three things: to know what happened, to know it is solved or being solved, and to have a person to turn to. Avoid jargon about models and hallucinations. Also avoid hiding that an automated system was involved when it matters to the case. Plain transparency builds more trust than a vague explanation.

A structure that works:

• Acknowledge the error directly.
• State the effect on the customer in one sentence.
• Say what has already been done and what happens next.
• Name the responsible person and the channel.
• Only then apologize and, if applicable, mention compensation.

Example of a short WhatsApp message:

Hi, this is Marina from the support team. Our automated assistant gave you incorrect information about your order. I have already corrected the record and I will follow your case personally until it is resolved. I apologize for the inconvenience.

Illustrative scenario: a field service company uses a WhatsApp agent to confirm visits. Because of a failure reading the calendar, the agent confirms a time slot that does not exist. The customer arranges to be home and nobody shows up. When the complaint arrives, the team pauses automatic time confirmations and keeps the rest of support running. It then searches the records for other confirmations from the same period and finds customers in the same situation. A support agent calls each one, reschedules with priority and explains what happened. Only afterward does the technical team investigate the calendar reading. The customer got the solution before the technical explanation, and that is the order your plan should guarantee. For how to design this kind of flow, see automatic appointment scheduling on WhatsApp with AI.

Two traps: blaming the technology (to the customer, the company is one entity) and promising it will never happen again. Promise what the company controls: fixing the case, following up and reviewing the process.

What if the error involved personal data?

If the failure exposed personal data, for example a message sent to another person, a reply containing a third party's information or improper access through a tool, treat it as a possible security incident as well as a support incident. This is where Brazil's data protection law (LGPD) enters the picture.

The practical path is to alert your data protection officer (or whoever performs that role), record which data, how many people and through which channel were affected, and assess the risk or harm to the data subjects. The LGPD provides for notifying Brazil's data protection authority (ANPD) and the data subjects when an incident may cause relevant risk or harm, and the analysis of each case belongs to legal counsel and the data protection officer, not to the support team. For the broader set of obligations, read LGPD and artificial intelligence for companies: what to do.

The records you need to investigate also contain personal data. Define retention periods, masking and who can access them before the incident, not during it. The article on AI conversation log retention and data masking covers this balance between evidence and data minimization.

How do you run the review after the error?

Once the customer is taken care of and the flow is stable, hold a review that does not hunt for someone to blame. The goal is to find which part of the system allowed the error. Bring together operations, support and technology, and rebuild the sequence from the records: what the customer asked, what context the agent had, what it decided and what it executed.

Causes usually fall into a few families:

• Outdated or conflicting information in the knowledge base the agent consults.
• Ambiguous instructions, or no explicit limit on what the agent must not answer.
• A tool with broader permission than the flow required.
• No handoff point to a human in situations of doubt or sensitive value.
• A change of model, vendor or integration without retesting.
• Malicious or unexpected input, such as an attempt to make the agent ignore its rules. The OWASP Top 10 for LLM applications describes these risks in an organized way.

For each cause, choose a structural fix. Prefer deterministic rules and permission limits over extra sentences in the prompt, because instruction text lowers the chance of error but does not eliminate it. Turn the incident into a test case: the conversation that failed goes into the evaluation set that runs before every change. Anthropic describes this practice in its engineering article on evals for AI agents, and the NIST AI Risk Management Framework offers a general structure for governing and monitoring AI system risks. For content errors themselves, the guide on how to reduce AI hallucinations in customer service complements this step.

Close the review with a short record: what happened, who was affected, what was done, the probable cause, the change applied and who confirms it works. Here it helps to separate what is fact in the records, what is the team's conclusion and what is still a hypothesis. An unconfirmed hypothesis should become a test, not a statement in the report.

How does mapping, limiting, piloting and handing off reduce the damage of an error?

Most of what determines an error's severity is decided before it happens. That is why the working method we use also covers the response plan.

Map: before putting the agent live, describe the current process and identify where an error costs the most, whether in money, personal data or a promise to a customer. Those points define severity in your plan.

Limit: restrict the data the agent can see and the actions it can perform. Starting in read mode and releasing write access step by step makes containment much simpler, because there is less to undo.

Pilot: run a real flow with complete records and an agreed success criterion. The pilot is the cheapest moment to make mistakes and rehearse the plan. The article on how to measure an AI automation pilot shows how to define those criteria.

Hand off: make clear the point where the conversation passes to a person, with the history attached. During an incident, that same path becomes your main containment tool.

Frequently asked questions

What should we do immediately when we find out the AI made a mistake with a customer?

Pause the flow, topic or tool responsible, without shutting down all support, and preserve the records of the conversation and actions. Then a person takes over the customer's case and contacts them. Investigating the cause comes after the customer is looked after.

Should we tell the customer that an automated assistant made the mistake?

If the error had a concrete effect on them, transparency is usually the best path: say simply that an automated piece of information or action was incorrect and that a person is handling the case. There is no need to explain technical details. What matters is what happened, what was done and who is accountable.

Is the company responsible for the AI's mistake, or the tool vendor?

To the customer, the company is the one that served them, and it is the one usually held to account. How responsibility is divided between company and vendor depends on the contract and each party's role in data processing, a topic explained in the article on data controller vs processor in an AI vendor contract. For a specific concern, consult a lawyer.

How do we keep the same error from happening again?

Turn the incident into a structural change: reduce permissions, add a rule or handoff point, fix the knowledge base and include the failed conversation in the tests that run before every change. Changing only the instruction text helps, but on its own it rarely suffices. Then confirm, using data from the flow itself, that the fix worked.

Sources

• NIST: AI Risk Management Framework
• OWASP: Top 10 for LLM Applications
• ANPD: Autoridade Nacional de Proteção de Dados
• Anthropic: Demystifying evals for AI agents