Reduce LLM Token Costs With Prompt Caching, Routing and Compaction

Reduce LLM Token Costs With Prompt Caching, Routing and Compaction

To reduce LLM token costs with prompt caching and related techniques, start by measuring how much each workflow spends per completed task. Then apply three levers, in this order: reuse what repeats with prompt caching, send simple tasks to smaller models, and shorten the context that does not need to travel in full. Every cut is only worth keeping if a fixed set of tests shows quality held.

Key takeaways

• Measure cost per resolved task, not cost per call. A cheap answer that causes rework is expensive.
• Prompt caching works best when the fixed part of the prompt (instructions, rules, catalog) comes first and the variable part (the customer message) comes last.
• Smaller models handle classification, extraction and standardized replies well. Keep the larger model for work that requires judgment.
• Compacting context means summarizing history and retrieving only the relevant passages, without deleting anything the decision depends on.
• Every cost cut should pass through a set of test cases with an approval criterion defined before the change.

Where token costs actually come from

Before optimizing, you need to know what you are paying for. The cost of a call has two parts: input tokens (everything you send to the model) and output tokens (what it writes back). In many business workflows, input dominates, because every call repeats the same instructions, the same business rules and a conversation history that only grows.

A support agent on WhatsApp, for example, usually sends the full system prompt, the returns policy, the tone of voice and the latest messages with every single message. The customer wrote one short sentence, but the call carries a large block that barely changes.

That is why the first task is measurement. For each workflow, record how many input and output tokens each call uses, how many calls it takes to resolve a case, and how many cases need human intervention. The unit economics practice described by the FinOps Foundation helps tie spend to a unit of value, such as a "conversation resolved" or an "order classified". Without that unit, there is no way to tell whether a change improved or worsened the bill.

How does prompt caching reduce input costs?

Prompt caching lets the provider reuse the processing of an opening section of the prompt that repeats across calls. In practice, the stable part of the prompt is processed once, and later calls pay less to read it. Details vary by provider, but the general logic is the same. Anthropic's documentation on prompt caching describes the mechanism, the minimum size of an eligible section, how long the cache lasts, and the cost difference between writing and reading. Check those values at the source, since they change over time and by model.

What is up to you is how the prompt is organized. The cache usually applies to the prefix: if any part of the beginning changes, reuse is lost from that point on. That leads to simple assembly rules:

• Put what never changes first: the agent's role, rules, company policy, fixed examples and tool definitions.
• Put what changes rarely next: customer or account data that holds for the whole session.
• Put what changes on every call last: the current message and tool results.
• Do not insert the date, time, request ID or any dynamic value at the start of the prompt. A single variable field at the beginning cancels the cache.
• Keep the order and text of tool definitions stable between calls. Reordering the list already changes the prefix.

One caution: the cache has a short lifespan. Workflows with steady traffic benefit well. Workflows with spaced-out calls may pay the write cost repeatedly without ever getting the read benefit. Before investing, check the provider's logs to see how many tokens are being read from the cache and how many are being written. If the read rate is low, the problem is almost always the prompt order or the gap between calls.

If you are still writing the support prompt, it is worth reading how to write instructions for an AI support agent. A prompt split cleanly into fixed and variable blocks is both easier to maintain and cheaper to run.

When should you use a smaller model?

Model routing is the practice of choosing the model according to the difficulty of the task. Not every step in a workflow needs the most capable model. Many steps are mechanical: identifying the intent of a message, extracting a name and order number, classifying the topic, formatting a reply from data that has already been validated.

A simple decision guide:

• Intent classification with a few clear categories: candidate for a smaller model.
• Extracting fields from a short text (name, date, order number): candidate for a smaller model.
• Rewriting or formatting a reply already decided by a rule: candidate for a smaller model.
• A reply that combines several policies, exceptions and a delicate tone (complaints, collections): keep the larger model or escalate to a human.
• A decision with financial or legal consequences: larger model with validation, and human approval when policy requires it.
• A new task with no test cases: start with the larger model, measure, and only then try moving down.

There are two common ways to route. The first is static: each step of the workflow has a model assigned in advance. It is the most predictable and the easiest to audit. The second is dynamic: a lightweight classifier decides, message by message, whether it goes to the small model or the large one. It saves more at high volume but adds a new point of failure, because a classifier that underestimates difficulty sends complex cases to the wrong model.

A prudent rule is to start static, step by step, and adopt dynamic routing only when you have enough data to validate it. Also define a safety exit: if the smaller model returns low confidence, an invalid format or an answer that fails a check, the call moves up to the larger model or goes to a person. This design resembles the chaining and routing patterns described in Anthropic's article on building effective agents.

If the question is how to compare models from different providers for a specific task, the testing method in Kimi K2.5 vs Claude Opus 4.5 shows how to build a comparison with clear criteria instead of impressions.

How do you compact context without losing what matters?

The third path is to send less text. More context does not mean a better answer: irrelevant information spends tokens and can distract the model. Anthropic discusses this in its writing on context engineering for agents, with the idea of keeping in context only the leanest set of information that supports the next decision.

Techniques that tend to work:

• Summarize the history: instead of resending the whole conversation, keep the latest messages verbatim and a structured summary of what came before (the customer's request, data already confirmed, open items).
• Retrieve, do not paste: instead of including the entire manual, fetch the passages relevant to that question. This is the retrieval logic described in materials such as AWS's overview of retrieval-augmented generation.
• Clean tool results: a query to an ERP or CRM may return dozens of fields. Pass the model only the fields the decision uses.
• Limit the output: ask for short replies in a defined format when the workflow does not need long text. Output tokens usually cost more than input tokens.
• Remove redundant examples: a few good examples are worth more than many similar ones.

The risk of compaction is discarding a detail that would have changed the decision. A summary that forgets "the customer already asked to cancel" makes the agent offer the product again. So define which facts can never be lost (identification, promises made, restrictions, consent) and keep them in structured fields outside the free-text summary. Where data protection law applies, such as LGPD, use compaction to avoid carrying unnecessary personal data in the call: sending less is both cheaper and more aligned with data minimization.

How do you prove quality did not drop?

Saving money without measuring is a bet. Before any change, build a set of test cases from anonymized real situations: the common cases, the hard ones and the ones that have gone wrong before. Define what an acceptable answer is for each. That set becomes the ruler for comparing the current setup with the cheaper one.

Anthropic explains how to structure this kind of evaluation in its piece on evals for AI agents, and OpenAI maintains an evaluations guide built on the same logic. When correctness requires judgment and not just exact matching, another model can act as the grader, as long as it is calibrated by human review. The article on LLM as a judge covers that method in detail.

Set three numbers before you start, without borrowing market targets: the acceptable quality floor in the tests, the limit of cases allowed to go to a human, and the cost per resolved task in the current setup. A change goes to production only if it keeps quality above the floor and lowers the cost per task. If it lowers cost but increases human rework, the gain is an illusion. To structure the decision criteria for a pilot, the guide to metrics and decision gates for an AI automation pilot is a good base.

Step-by-step plan

Order matters because each step creates the measurement base for the next:

• Step 1: pick a single workflow with recurring volume and record input tokens, output tokens, number of calls and outcome per case.
• Step 2: build the test set and fix the quality floor with the person who owns the process.
• Step 3: reorganize the prompt with fixed blocks first and variable ones last, remove dynamic values from the start, and turn on the cache.
• Step 4: check the logs to confirm cache reads are actually happening. If not, fix the order or the gap between calls.
• Step 5: identify the mechanical steps in the workflow and test a smaller model on each one, one at a time, with a fallback to the larger model.
• Step 6: compact the context (structured summary, passage retrieval, cleaned tool outputs) and run the tests again.
• Step 7: compare cost per resolved task and quality against the baseline. Decide whether to expand, adjust or stop.
• Step 8: keep monitoring. Models and prices change, and a new prompt can break the cache without warning.

Illustrative scenario: an online store has an agent that answers order and return questions on WhatsApp. Every call sends the full returns policy, the tone of voice and the entire history. The team reorganizes the prompt to put policy and tone at the start, moves the current time to the end, uses a smaller model only to identify intent and extract the order number, and sends a structured summary instead of the whole conversation. Then it runs the same test cases before and after. If the delicate complaint cases got worse, it keeps the larger model on that route only. The gain shows up in cost per resolved order, and quality is demonstrated by the tests, not by impression. To see what to keep human in this kind of workflow, read AI customer support: what to automate and what to keep human.

How this work fits into an implementation method

The method we use for AI projects follows four moves, and cost optimization fits into all of them. Map: document the current process and the baseline for spend and quality. Limit: restrict data and actions, which also reduces the context sent. Pilot: apply caching, routing and compaction to a real workflow, with logs and an agreed success metric. Transfer: hand over the cost-per-task dashboard and the re-evaluation routine, so the team can decide whether to expand, adjust or stop.

Optimization should come after the workflow works and is measured. Cutting the cost of an agent that still makes mistakes only makes the mistakes cheaper.

Frequently asked questions

Does prompt caching change the model's answer? In principle no, because the cache reuses the processing of the same input section and the content sent stays identical. What can change is latency, usually for the better. Still, run your tests after enabling it, because reorganizing the prompt to make the cache work can alter behavior.

Is a smaller model always cheaper in the end? Not always. If it makes more mistakes and causes repeated calls, manual correction or escalation to a human, the cost per resolved task can go up. Always compare the total cost to resolution, not the price per token in isolation.

Where do you start if spend is already high and nothing is measured? Start by recording tokens and outcomes for a single workflow over a short period, without changing anything. With that baseline, the first action is usually reorganizing the prompt for the cache, which changes behavior less than switching models. Only then test smaller models and compaction.

Does compacting context affect LGPD compliance? It can help, as long as the summary does not create a new, uncontrolled store of data. Sending the model only the data it needs is aligned with minimization, but summaries that contain personal data need retention and access rules. For the general framework, see LGPD and artificial intelligence for companies.

Sources

• Anthropic: prompt caching
• Anthropic: Effective context engineering for AI agents
• FinOps Foundation: unit economics
• Anthropic: Demystifying evals for AI agents