Prompt Regression Testing in CI: Catch Quality Drops Before Users Do

Prompt Regression Testing in CI: Catch Quality Drops Before Users Do

Prompt regression testing means running a fixed set of real cases every time a prompt or model changes, then comparing the results against an agreed standard before anything ships. In CI, this becomes an automatic step: if the score drops below the agreed threshold or a critical case fails, the change does not go through. The team stops finding out about quality drops from customer complaints.

Key takeaways

• The prompt, the model and the tool descriptions together define how an agent behaves. A change to any of them should run against the same set of cases.
• The fixed set comes from real (masked) conversations and from mistakes that already happened, not from examples invented on the spot.
• Prefer deterministic checks. Use a model as a grader only where judgment is needed, and calibrate that grader against human review.
• Separate critical cases, where any failure blocks the change, from general cases, which are judged by an aggregate score against a threshold.
• Every production incident becomes a new case. Without that, the suite goes stale and protects against problems nobody has anymore.

Why does a prompt get worse without anyone noticing?

Prompts do not behave like ordinary code. A sentence added to fix one specific error can change the tone, the length or the decision in situations nobody looked at. The result seems fine on the three examples the author tested by hand, and the problem only shows up days later, in a different kind of conversation.

Some changes also happen outside your control. The provider can update the model behind a generic name. A tool description edited by another team changes when the agent calls that tool. A new document in the knowledge base changes what the agent cites. In all of these cases the prompt text stays the same and the behavior changes.

AI output also varies from one run to the next. So the useful question is not whether a response is identical, but whether it still falls within what the company accepts. Regression testing turns that acceptance into written, repeatable rules. For the neighboring problem of invented answers, see the practical guide to reducing wrong answers from AI agents.

What goes into the fixed set of cases?

Start small, but make it representative. The set should cover what matters to the business, not what is easy to write. A good mix includes:

• The most frequent flows, written the way customers actually talk, with abbreviations, transcribed voice notes and incomplete messages.
• Edge cases: missing information, ambiguous requests, two questions in the same message.
• Out-of-scope requests, where the correct behavior is to decline or redirect.
• Situations that require a handoff to a person, such as a serious complaint, a cancellation or a legal matter.
• Attempts to push the agent past its limits, such as asking it to ignore its instructions or reveal its prompt.
• Errors that already happened in production, with the original conversation masked.

Each case needs an input, the context it depends on (for example, fictional order data), a pass criterion and a criticality flag. The criterion can be a reference answer, a list of mandatory conditions or a list of forbidden things.

On personal data: when you reuse real conversations, remove or replace names, phone numbers, ID numbers and addresses before storing the case in the repository. This lowers LGPD risk and makes it easier to share the suite with the team. The article on AI conversation log retention and data masking covers retention periods and masking in detail.

How do you grade a response when the text changes on every run?

Use layers, from the cheapest and most objective to the most subjective.

First layer, deterministic rules. Check whatever code can check. Did the agent call the right tool with the right arguments? Did it return the expected format? Did it trigger the handoff to a human? Did it avoid forbidden terms and promises? Did it cite only information present in the context it was given? These checks are stable and cheap, and they should cover most of your critical cases.

Second layer, a model as grader. For qualities such as appropriate tone, clarity or fidelity to reference material, a second model can score the response against a written rubric. The rubric has to be specific (for example, the response confirms the deadline given in the context without adding another one), and the grader has to be checked against human scores on a sample. A grader nobody has calibrated is just another source of noise.

Third layer, human review. Reserve it for cases where the score changed meaningfully and for periodic calibration. The person who knows the support process should be the one who decides what counts as an acceptable answer.

For cases that fluctuate, run the same input more than once, keep the generation settings as stable as you can, and decide on the pass rate rather than on a single run. Runtime validation layers complement this, as explained in guardrails for AI agents.

Step by step: putting it in CI

The design below works with whatever pipeline your team already uses. The tool names matter less than the discipline.

• Version everything in the repository: the prompt, generation parameters, tool descriptions and the exact model identifier. Avoid aliases that change on their own.
• Create the case file in a simple format (JSON, YAML or an exported spreadsheet), with input, context, criterion and criticality.
• Write a runner that calls the agent in a test environment, using mocked tools or fictional data. Never point it at real systems.
• Implement the graders in layers: rules first, model-based rubric second.
• Store a baseline result for the version currently in production, so every change can be compared against it.
• Generate a readable report in the pull request: which cases passed, which failed, what changed relative to the baseline, plus the cost and latency of the run.
• Make the step required for merging. Without a passing test, the merge does not happen.
• Schedule a periodic run even when no code has changed, to catch changes coming from the model provider.
• Log each run with an identifier and the prompt and model versions. Telemetry conventions for generative AI, such as those from OpenTelemetry, help standardize these fields.

When should the pipeline block, and when should it only warn?

Without a decision rule agreed in advance, the test turns into a debate every time something fails. Define the guide below before the first run.

• A critical case failed: always block, with no exception for urgency.
• The aggregate score on general cases is below the agreed threshold: block.
• A small drop, within the normal variation seen across repeated runs: warn and ask for human review, without blocking.
• Overall improvement, but one important case got worse: the owner of the flow records in writing why they accept the trade.
• Model swap: run the full suite, compare cost and latency too, and approve only with a recorded decision.
• Unstable test: never ignore it. Fix the case, the criterion or the grader, because a test that fails at random teaches the team to dismiss the alert.

The pass threshold should come from your pilot, not from a number copied from somewhere else. For how to define metrics and decision gates, see how to measure an AI automation pilot.

Illustrative scenario: the shorter reply that hid a cancellation

Illustrative scenario: a distributor uses a WhatsApp agent to look up orders and guide customers. Someone on the team notices the replies are long and edits the prompt to ask for shorter messages.

In the CI run, the general order-lookup cases pass with an equal or better score. But one critical case fails: a customer writes that they want to cancel and complains about a delay, and the agent, now more concise, answers with a single apology and does not trigger the handoff to a human agent. The pipeline blocks the change and the report points to exactly that case.

The team rewrites the instruction to keep the handoff rule explicit, runs the suite again and approves it. Without the fixed set, the shorter version would have gone live, and the problem would only have surfaced when an unhappy customer was left without a human reply.

How the map, limit, pilot and hand over method applies

First, map: before writing cases, you need to know how the process works today, which requests arrive, what a human agent would do in each one and what the quality starting point is. The critical cases come out of that map. Second, limit: the test only makes sense if the agent has clear limits on data and actions, because each limit becomes a verifiable criterion (it must not promise, it must hand off, it must not query a given system).

Third, pilot: the case set goes into a real flow, with logs and an agreed success metric, and the blocking thresholds are calibrated against what you observe. Fourth, hand over: when the operation is delivered, the repository with cases, graders and decision rules goes to the team, along with guidance on who approves changes and how an incident becomes a new case. After that, the company decides whether to expand, adjust or stop.

A note on sourcing: the technical foundations cited here come from official documentation on AI evaluations. The layered structure and the blocking rules are implementation recommendations, and any numeric thresholds are hypotheses each company should validate in its pilot. For help writing the instructions themselves, see how to write instructions for an AI support agent.

Frequently asked questions

How many cases do I need to start? There is no magic number. Begin with cases covering the most frequent flows, the requests that require a handoff to a human and the errors that already happened, then let the set grow with every incident. A few well-chosen, trusted cases beat many generic ones that nobody reviews.

Do I need a specific evaluation tool? Not necessarily. A script that reads the case file, calls the agent and applies the checks is enough to start, and it runs in any CI pipeline. Evaluation platforms help when the number of cases and versions grows, but the hard part is still choosing good cases and clear criteria.

Is a grader model reliable enough to block a change? It depends on how it was calibrated. Use deterministic rules for critical cases and keep the model-based grader for more subjective qualities, always checked against human scores on a sample. If the rubric is vague, the result will be too.

How often should I run the suite? On every change to the prompt, a tool, the knowledge base or the model, and also on a schedule to catch changes from the provider. If the full run is costly, keep a reduced version with the critical cases on every pull request and run the full suite at scheduled times.

Sources

• Anthropic: Demystifying evals for AI agents
• OpenAI: guia de evals
• OpenTelemetry: convenções semânticas para IA generativa
• NIST: AI Risk Management Framework