Handling LLM API Rate Limits and Outages: Keep Your Agent Running

Handling LLM API Rate Limits and Outages: Keep Your Agent Running

To keep an AI agent running when your provider throttles or goes down, combine four layers: a queue that absorbs spikes, retries with exponential backoff and jitter, a backup model for the tasks that tolerate a swap, and an honest degraded-mode message with a handoff to a person. The most common mistake is treating every failure the same and repeating the call without criteria, which makes the limit worse and can duplicate actions. The design below works for support agents, internal operations and AI automations.

Key takeaways

• Rate limits and provider outages are different problems and need different responses: the first is solved with pacing, the second with an alternative.
• Every model call should go through a queue, so spikes become controlled waiting instead of lost messages.
• Retries are only safe with backoff, jitter, a cap on attempts and idempotency for any action that changes a system.
• Model fallback is a business decision: some tasks accept a smaller model, others should wait or go to a person.
• The customer needs a clear degraded-mode message, and the team needs metrics and alerts to know the agent is running outside normal conditions.

What is the difference between a rate limit and a provider outage?

A rate limit means the provider is working but refused the call because your account exceeded the volume allowed in a time window. These limits are usually measured in requests and in tokens, and they vary by plan and account, so the starting point is to read your provider's documentation and confirm the values in your account dashboard. The typical response is HTTP status 429, often with a header telling you when to try again.

An outage or degradation is a different situation: the service returns a server error, takes too long or does not respond at all. Here, insisting on the same model for a long time does not help. The outage may be partial, limited to one model or one region, and in that case an alternative exists inside the same provider or at another one.

Separating the two cases avoids two opposite disasters. The first is switching models because of a simple spike, which changes answer quality for no reason. The second is waiting on a model that is down while customers sit without a reply.

How to classify the error before deciding what to do

The first component is a single layer that receives the provider's error and classifies it. Without it, each flow reacts in its own way and the agent's behavior becomes unpredictable. A simple decision guide covers most cases:

• Rate limit (429): do not switch models right away. Respect the wait time the provider reports, if any, and put the task back in the queue with a delay.
• Temporary server error or timeout: retry with backoff, with few attempts. If it persists, trigger the fallback.
• Provider overload: treat it as temporary, but reduce your own throughput instead of adding pressure.
• Invalid request error (context too large, wrong parameter): do not repeat. This is a bug or an input limit, and retrying only burns capacity. Log it and fix it.
• Authentication or permission error: do not repeat and do not fall back blindly. Alert the team, because it may be an expired or revoked key.
• Response received but rejected by output validation: this is not an infrastructure failure. Follow your rules for guardrails that validate agent outputs and choose between a controlled retry and a handoff.

Exact codes and header formats change from provider to provider. That is why classification should live in a single module, easy to adjust when you switch or add a vendor.

Queues: how to absorb spikes without losing messages

In a WhatsApp agent, the customer's message arrives through a webhook and the model is called right after. If the model call happens inside the webhook itself, any slowness or limit turns into a lost or duplicated message. The fix is to separate the steps: the webhook only validates the message, stores it and places it in a queue. A separate process consumes the queue at the pace the provider allows.

A few details make a difference in practice:

• Concurrency limit: define how many model calls can run at the same time and keep that number below what the account supports.
• Priority: messages from customers waiting for a live reply go ahead of batch tasks such as overnight summaries or bulk classifications.
• Order per conversation: messages from the same conversation should be processed in sequence, so the agent does not reply out of order.
• Expiry: a reply that sat in the queue too long may no longer make sense. Define how long it is worth trying and what happens afterward (handoff or notice).
• Dead-letter queue: tasks that exhausted their attempts go to a visible place, with the context a person needs to take over.

The queue is also the natural place to control cost, because you see the volume before it reaches the provider. To reduce consumption, combine it with techniques such as prompt caching, routing and compaction, which ease pressure on the limit.

Backoff with jitter and safe retries

Exponential backoff means waiting longer after each new attempt: a short interval the first time, longer the second, longer still the third. Jitter is a small amount of randomness in that interval. Without it, every task that failed together retries together, and the spike repeats in waves. With jitter, the attempts spread out over time.

A responsible retry policy has these elements:

• Respect the wait time the provider reports, when it exists, instead of relying only on your local calculation.
• Set a maximum number of attempts and a maximum total time per task.
• Retry only errors classified as temporary.
• Use a limiter on your side, so you do not depend only on the provider's errors to slow down.
• Stop retries collectively when many errors happen in a row (a circuit breaker), giving the provider time to recover.

The most neglected point is idempotency. If the agent called the model, decided to create an order in the ERP and the response got lost on the way, a new attempt may create the order twice. The rule is that actions with an external effect carry an idempotency key, so repeating the same operation does not produce a second effect. Stripe's documentation on idempotent requests is a clear example of how a provider defines this contract, and the same reasoning applies to your internal systems. Also separate the model call (which can be repeated) from the execution of the action (which needs a key and, in sensitive cases, human approval).

Model fallback: when to switch and when to warn

Fallback means having a plan B when the primary model does not respond. It can be another model from the same provider, a smaller model, a second provider or, in some cases, a model you host yourself. Each option has a maintenance cost: prompts that work well on one model may behave differently on another, and data starts flowing through one more vendor, which matters for LGPD and for your contract.

So the right question is not which model to use as fallback, but which tasks accept a fallback. A practical way to decide:

• Low-risk tasks with a rigid format, such as classifying the topic of a message or extracting an order number: they accept a smaller or alternative model, as long as the output goes through the same validation.
• Free-form customer replies: they accept fallback only if the backup model was tested against the same evaluation cases as the primary. Otherwise, prefer waiting in the queue or handing off.
• Tasks that execute actions (cancel, refund, change a record): in degraded mode, they should go to human approval or stay paused, not be delegated to an untested model.
• Tasks with sensitive data: use a second provider only if it has already been assessed and is covered by your records of processing and your contract. When in doubt, do not.

Before enabling any backup, run the same set of test cases on both models and compare. That set also shows what changes in behavior, such as tone, answer length and tool use. If the decision involves comparing models, the article on how to compare Kimi K2.5 and Claude Opus 4.5 shows a testing method you can adapt to your case.

To structure the agent so that swapping is simple, Anthropic's engineering write-up on building effective agents argues for starting with simple, composable patterns. That makes it easier to isolate the model call behind a single interface and replace it when needed.

What to tell the customer when the agent is degraded

Even with a queue and fallback, there will be moments when the answer is slow or does not arrive. What the customer sees at that point decides whether they wait or give up. A good degraded-mode message is short, does not invent a technical cause, says what happens next and offers a path.

An example text for WhatsApp support:

Our automatic support is running slowly right now. Your message was received and a team member will reply during business hours, or you can keep writing here and we will pick it up as soon as we can.

Good practices for these messages:

• Send the notice once per conversation, not after every customer message.
• Never promise a deadline the operation cannot meet.
• Do not pretend the reply is normal when it came from a backup model on a sensitive task.
• Record that the conversation was marked as degraded, so the team can prioritize it later.
• For urgent requests (cancellation, payment problems, health), trigger the handoff immediately, following the WhatsApp AI agent human handoff playbook.

The message also needs a defined operating mode: normal, degraded and paused. In degraded mode, the agent answers only what is safe and routes the rest. In paused mode, it only receives and organizes. Who decides the mode change should be an explicit rule (for example, consecutive errors above a threshold), with an alert to the team, not improvisation during the incident.

Illustrative scenario and rollout plan

Illustrative scenario: an online store has an agent that answers order and exchange questions on WhatsApp. After a campaign, message volume suddenly grows and the provider starts returning rate limit errors. Without a queue, the webhook fails, WhatsApp resends messages and the agent answers some customers twice. With the design described here, messages enter the queue, the agent processes them at the allowed pace, tracking lookups (low risk) use a smaller model when the wait grows, exchange requests wait for the primary model or go to the team, and customers receive a single slowness notice. At the end of the day, the team sees how many conversations were degraded and how many went to handoff.

To get there, a staged plan works well:

• Step one: list every point in the flow that calls the model and classify each task by risk (read, reply, action).
• Step two: centralize the model call in a single module, with error classification and a log of every attempt.
• Step three: place the queue between the channel and the model, with a concurrency limit, priority and expiry.
• Step four: implement backoff with jitter, a cap on attempts, a circuit breaker and idempotency keys for actions.
• Step five: define which tasks accept fallback, test the backup model against the same evaluation cases and document the differences.
• Step six: write the degraded-mode messages, the operating modes and the handoff rules.
• Step seven: instrument the flow with metrics for errors, queue wait time, fallback use and degraded conversations, and set up alerts.
• Step eight: simulate failures on purpose (an artificial limit, an unavailable provider) before going to production.

On observability, the OpenTelemetry semantic conventions for generative AI help standardize what to record on each call, such as the model used, the error and the duration, without inventing a format of your own.

In the method we use, this fits into map, limit, pilot and transfer. Mapping is discovering which flows depend on the model and what happens to the business if each one stops. Limiting is deciding what the agent may do in degraded mode. Piloting is testing a real flow with logs and an agreed metric, as described in the guide to measuring an AI automation pilot with decision gates. Transferring is handing the team the runbook, the alerts and the decision to expand, adjust or stop.

Frequently asked questions

Should I have a second LLM provider from the start? Not necessarily. Start with a queue, backoff and handoff, which resolve most short incidents. A second provider makes sense when downtime is costly for the business, and it should come with testing, a contract review and a check on how personal data will be handled.

How many times can I retry a failed call? There is no universal number. Set a low cap on attempts and a maximum total time per task, based on how long your customer can reasonably wait. After that, the task should go to fallback, to a person or to the dead-letter queue, not keep repeating indefinitely.

Can falling back to a smaller model make the customer's answer worse? It can, which is why fallback should be limited to tested, low-risk tasks. Run the same evaluation cases on both models, validate the output with the same rules and, for sensitive replies or actions, prefer waiting or handing off to a person.

How do I know the agent is running in degraded mode? Log every attempt, error and model switch, and set alerts for streaks of failures, a growing queue and frequent fallback use. That way the team is warned before the customer complains, and can later review which conversations were affected.

Sources

• Stripe: requisições idempotentes
• Anthropic: Building effective agents
• OpenTelemetry: convenções semânticas para IA generativa