How to Debug Production Bugs With AI: Masked Logs and Hypotheses

How to Debug Production Bugs With AI: Masked Logs and Hypotheses

To debug production bugs with AI, mask your logs before sending them, ask for ranked hypotheses that state what each one predicts you will find, and leave validation to the developer: reproduce the error, confirm the cause, and fix it with a test. AI is good at reading a lot of text quickly and proposing plausible explanations. It does not know what actually happened in your system, so the gain is a shorter search, not skipping the proof.

Key takeaways

• Production logs contain personal data and secrets. Mask them before sending anything to a model, and treat masking as a fixed step of the workflow.
• Ask for hypotheses, not verdicts. Each hypothesis should come with the evidence behind it and the test that would refute it.
• AI proposes, the developer decides: no fix goes in without a reproduced error and a test that failed before and passes after.
• A well-trimmed context beats a huge one. Send the slice from the incident time window, not the whole file.
• Record what was a hypothesis, what was proven and what was ruled out. That becomes team memory for the next incident.

Why start with logs instead of code?

In production, the code is rarely the first signal. The first signal is a behavior: orders that never complete, messages that do not arrive, a screen that freezes for one type of customer. The record of that behavior lives in logs, metrics and traces. The OpenTelemetry observability primer describes the idea well: observability means being able to ask questions about a system from the signals it emits, without changing the code for every new question.

This is where AI helps. One incident can produce thousands of lines, and a tired developer tends to fixate on the first anomaly they see. A model reads the whole set without fatigue and can point out patterns: an error that always precedes another, a field that is empty only on certain requests, a time interval that keeps repeating. Those patterns are starting points for investigation, nothing more.

If your system emits few useful signals, the first step is not AI. It is improving the logs: request identifier, operation name, outcome and duration. Without those, any analysis, human or automated, becomes guessing.

What should you mask before pasting anything?

Production logs often carry more than people expect: emails, phone numbers, national ID numbers, addresses, API tokens, session cookies, snippets of customer messages and sometimes entire request bodies. Sending that to an external service is a data processing decision, and Brazil's data protection authority (ANPD) is the official reference for understanding LGPD obligations on this topic. If the provider processes data outside Brazil, also read about international data transfer under LGPD when using AI providers.

A practical rule: mask by substitution that preserves format and the relationship between values. If the same customer appears on five lines, it should become the same placeholder on all five, so the model can still see the pattern.

• Personal identifiers (name, ID number, email, phone): replace with stable placeholders such as USER_A, EMAIL_A, PHONE_A.
• Secrets (keys, tokens, passwords, cookies, authorization headers): remove completely, with no reversible placeholder.
• Free text typed by people (messages, notes, attachments): remove or summarize, since this is where data leaks most and also where malicious text can appear.
• Technical identifiers (request ID, internal order ID): keep them, since they are the thread that links events.
• Host names, databases and internal paths: decide case by case. They can usually be generalized.

Automate masking with a versioned, reviewed and tested script instead of relying on each person to remember. Then spot-check whether anything slipped through. Masking that fails silently is worse than none, because it gives false confidence.

There is also a security concern: logs contain text that comes from outside, and outside text can try to instruct the model. The OWASP Top 10 for LLM applications treats this as a real risk. Tell the model explicitly that the log is data to analyze, never instructions to follow, and do not give it tools that perform actions. For more on this risk, see prompt injection protection for AI agents.

How do you ask for hypotheses instead of a ready answer?

The most common mistake is pasting the log and asking what the bug is. The model will answer with confidence even without enough evidence. A better request forces structure and honesty about uncertainty. Always include these elements:

• Observed symptom: what users or monitoring saw, and when it started.
• Time window and scope: only the relevant slice, with a few lines before the first error.
• What changed: recent deploys, configuration, dependencies, usage volume.
• What has already been ruled out: this keeps the model from repeating the obvious.
• Answer format: ranked hypotheses, each with the log evidence that supports it, what would contradict it, and the cheapest test to check it.
• Explicit limit: ask the model to say when there is not enough evidence, instead of filling the gap.

On context size: more text does not mean better analysis. Anthropic discusses this in its material on context engineering for agents, and the logic applies here: what goes into the conversation should be the smallest set of high-signal information. To understand the limit in practice, read about the context window in AI.

Treat the answer as a list of suspects, not a sentence. A good output has hypotheses that contradict each other, and that is useful: it shows exactly which observation separates one from the other.

How do you validate and reproduce before touching the code?

This step belongs to the developer, and it is what stops AI from turning a guess into a wrong fix that goes unnoticed. A simple flow:

• Step one: pick the hypothesis that is cheapest to test, not necessarily the most likely. If a database query confirms or rules it out in minutes, start there.
• Step two: check the evidence yourself in the original, unmasked logs, inside your own environment. Do not trust the model's quotation of the log excerpt.
• Step three: try to reproduce it in a test or staging environment, with synthetic data that mimics the shape of the real case.
• Step four: write a test that fails for the suspected reason. If you cannot make it fail, the hypothesis is not confirmed yet.
• Step five: only then change the code, and run the test again to see it go from failing to passing.
• Step six: watch production using the same signal that revealed the problem, for an agreed period, before declaring the incident closed.

If no hypothesis is confirmed, go back to the model with what was ruled out and the new observations. Each round should narrow the search space, not restart it.

What does this look like in practice? An illustrative scenario

Illustrative scenario: an online store notices that some orders are stuck in a pending payment status even though the payment was approved. The team extracts the logs from the incident window, runs the masking script and asks the model for hypotheses, mentioning that a deploy happened the day before.

The model returns three hypotheses. The first: receiving notifications from the payment gateway is failing on signature validation, and the log shows rejections concentrated after the deploy. The second: notifications are being processed twice, causing a database conflict. The third: a timeout that is too short is dropping the response to the gateway, which then resends later. For each one, it says what it would expect to see.

The developer picks the cheapest check: counting signature rejections in the original logs. They exist, but they are few and do not explain the volume of stuck orders. She moves on to the third hypothesis and sees in the duration records that responses exceeded the configured limit after a new query was added to the notification handler. She reproduces it in staging with a simulated slow query, writes a test that fails on timeout, and moves the query to asynchronous processing. The test passes, and she watches production to confirm the status stops getting stuck. The second hypothesis becomes a note to investigate later, because idempotency in the receiving step deserves review anyway.

AI's role was organizing the search space. The proof, the decision and the fix were human.

How do you fix and close the incident without repeating it?

After the fix, three records make the difference. The first is the regression test, which protects against the same error coming back. If AI helped write that test, apply the same care described in AI-generated tests: can you trust them and how to validate: the test must fail without the fix and check behavior, not implementation.

The second is a short incident note with four fields: symptom, confirmed cause, what was hypothesized and ruled out, and what changed. Separating fact from hypothesis prevents someone, months later, from treating an old guess as established knowledge.

The third is improving the signal itself. If a request identifier or a duration metric was missing, add it now. The next incident will be faster to investigate with or without AI.

Whoever leads the team should avoid turning this workflow into individual monitoring. Measure whether incidents are being resolved with more clarity and less rework, not how much each person used the assistant. For a fuller approach, see how to measure AI productivity without surveillance on your team.

When should you not send logs to an external AI?

Not every log should leave the company, even masked. A simple decision guide:

• If the log contains free-form customer content that you cannot mask with confidence, do not send it. Analyze it manually or use a model running on your own infrastructure.
• If contractual confidentiality or a regulated sector is involved, consult legal and your data protection officer before defining the workflow.
• If the incident involves compromised credentials, treat it as a security incident first and investigate in a controlled environment.
• If the problem is structurally simple, such as a configuration error visible in a few lines, you may not need AI at all.

For sensitive cases, a local model may be the way to go. See running Ollama local models in a company, keeping in mind that smaller models have reasoning limits and require more verification.

The method we apply in AI projects also fits this internal workflow. Map: understand how the team investigates incidents today and where time is lost. Limit: define which data can leave, with what masking, and make clear that AI does not execute actions in production. Pilot: test it on one type of incident, recording the hypotheses and an agreed success criterion. Transfer: document the workflow and the masking script for the team, and decide whether to expand, adjust or stop.

Frequently asked questions

Is it safe to paste production logs into ChatGPT or another AI? Only after masking personal data and removing secrets, and within a company policy that defines which tool and which plan may be used. Without masking, you are handing customer data to a third party, which requires a legal basis and care under LGPD. When in doubt, use a local model or ask your data protection officer.

Can AI find the root cause on its own? In simple cases it sometimes gets it right the first time, but there is no guarantee, and it cannot see what is not in the log. Treat the answer as a hypothesis and confirm it with reproduction and a test. The root cause is only established when you can trigger the error and eliminate it with the fix.

How much log should I send to the model? The minimum that covers the incident window and the moment just before the first sign of error, along with what changed recently. Huge files dilute the signal and raise the cost. If the first round is not enough, widen the slice deliberately.

Can I let AI apply the fix directly in production? It is not recommended. Let it suggest the code, but the change should go through testing, human review and your normal deploy process. Tools with production access increase the risk, and any permission should be minimal, audited and approved by a person.

Sources

• OpenTelemetry: conceitos de observabilidade
• Anthropic: Effective context engineering for AI agents
• OWASP Top 10 for LLM Applications
• ANPD: Autoridade Nacional de Proteção de Dados