LLM Batch API for Large Volume Processing: A Practical Guide

LLM Batch API for Large Volume Processing: A Practical Guide

Use an LLM batch API when the task does not need an immediate answer and the volume is large. Bulk classification, summarization and extraction are the typical cases. You send a file with many requests, the provider processes them asynchronously and returns the results later, with an identifier for each item. What decides whether a batch works is how you handle partial failures: every item must be traceable, validated and resendable on its own.

Key takeaways

• Batch is for work that can wait: if nobody is waiting for the answer on a screen or in WhatsApp, consider the batch API.
• Providers generally offer a lower cost per token in batch mode in exchange for a more flexible completion window. Confirm the current terms in your provider's documentation.
• A batch almost never fails or succeeds as a whole. Plan for items with errors, invalid items and items that came back with no result.
• Each request needs its own stable identifier tied to the source record, so you can reconcile results and resend only what failed.
• Before running the full volume, test a small sample and define what counts as an acceptable answer.

What is batch processing and how does it differ from a normal API call?

In a normal (synchronous) call, your system sends a request and waits for the response. This is the model for a virtual assistant, an assistant inside an internal system, or any flow where a person is waiting. Latency matters, so you pay for responsiveness.

With a batch API, you assemble a set of requests, send them all together and check the status later. The provider processes them when it has capacity, within a completion window that it defines. When it finishes, you download a results file. In general, the order of results is not guaranteed, which is why each item carries an identifier that you assign yourself.

In practice, this changes three things in your system design:

• You trade time for cost and operational slack: the result arrives later, but the processing does not compete for capacity with real-time traffic.
• You need state: which items were sent, in which batch, with which version of the instructions and with what result.
• You need a reconciliation step that compares what was requested with what came back.

Field names, batch size limits and the completion window vary between providers and change over time. Treat the provider's official documentation as the source for these details, and do not copy limits from articles, including this one.

Which tasks should go to batch and which should not?

The question that organizes the decision is simple: does anyone need this right now? If the answer is no, and there are many similar items, batch is a strong candidate. Use this decision guide as a checklist:

• Goes to batch: classifying a backlog of messages, tickets or reviews by topic, urgency or sentiment, when the result feeds a report and not an immediate reply.
• Goes to batch: summarizing a large history of conversations, meeting notes or documents for later reference.
• Goes to batch: extracting fields from many documents you already received, such as invoices, contracts or reports. For field design and human checking, see extract data from PDFs with AI.
• Goes to batch: enriching records, standardizing product descriptions or reviewing the quality of a knowledge base.
• Goes to batch: running offline evaluations, comparing instruction versions on the same set of examples.
• Stays real-time: customer conversations, where the person is waiting for the reply.
• Stays real-time: any step that blocks an immediate human action, such as validating data at the moment of registration.
• Stays real-time: flows where the result for one item decides the next question to the same user.

There is also a middle ground: tasks the business calls urgent but that tolerate delay. An end-of-day summary, for example, does not need to be generated after every message. Ask the responsible team what delay is actually acceptable before choosing synchronous mode out of habit.

How do you build a batch, step by step?

A lean plan that works for most classification, summarization and extraction cases:

• Define the task and the output format. Write the instruction, the expected response schema (for example, a category from a closed list) and what to do when the data does not exist in the text.
• Select the records. List where they come from, which fields go to the model and which stay out. Unnecessary personal data should not go into the request.
• Assign one identifier per item. Use a stable value tied to the source record and the instruction version, so that resending the same item does not create duplicates.
• Run a small sample in synchronous mode or in a test batch. Compare the output with what a person from the business area would consider correct.
• Submit the full batch and record: batch identifier, submission date, instruction version, model used and number of items sent.
• Check the status at reasonable intervals. Do not poll in a tight loop; processing is asynchronous by definition.
• Download the results and reconcile: every identifier sent must have a destination, whether success, error or missing.
• Validate the outputs before writing to the destination system. Invalid schema, categories outside the list and empty fields go to a review or resend queue.
• Resend only the items that failed, in a new batch, keeping the original identifiers linked to the attempt.
• Close the loop with a short report: sent, completed, rejected by validation, resent and pending human review.

How do you handle partial failures without reprocessing everything?

Partial failure is the normal behavior of a large batch, not the exception. Some items return a provider error, others expire within the window, and others come back with an answer that does not respect the requested format. Each situation calls for a different action.

Separate failures into categories:

• Request error: the item was malformed, too large or had content that was refused. Resending without changing anything repeats the error. Fix the item or send it to human review.
• Transient provider error: unavailability or overload. Resending usually works, with increasing wait times between attempts and a cap on the number of attempts.
• Expired or canceled item: the batch did not finish in time for that item. Resend it in a new batch.
• Invalid response: the item was processed, but the output fails validation. Here the problem is usually in the instruction or the data, not the infrastructure.
• Valid but doubtful response: it passes the format check but contradicts the source text or is inconsistent. This only shows up with sampling and human review.

A few rules make resending safe:

• Idempotency: resending the same item must not write twice to the destination. The idea is the same as in payment APIs that accept an idempotency key per operation, as Stripe's documentation on idempotent requests describes. You can apply the principle by using the item identifier as the write key.
• Attempt limit: after a few rounds, the item leaves automation and goes to a person. A resend loop with no ceiling becomes endless cost.
• Explicit state per item: pending, sent, completed, rejected, in review. Without this, nobody knows what is missing.
• Separate resend batch: do not mix new and resent items in the same batch without recording their origin. It makes auditing easier.

If your orchestrator is a workflow tool, apply the same practices from error handling and retries in n8n: errors separated by type, waiting between attempts and an exit path to human review.

Illustrative scenario: classifying a backlog of messages

This is an illustrative scenario, unrelated to any real company. A store has accumulated a large history of customer messages and wants to know the main reasons for contact each quarter, to adjust its catalog and support. Nobody needs the answer for each message right now.

The team defines a closed list of categories, plus an "other" category and a rule: if a message is ambiguous, classify it as "other" instead of guessing. Each message gets an identifier tied to the original record. Before the full batch, a person reviews a sample of the classifications and adjusts the instruction where categories were being confused.

The batch is submitted. When the results arrive, reconciliation shows three groups: completed and valid items, items with a provider error, and items whose category came back outside the list. The first group is written to the system. The second goes into a resend batch. The third goes to a small review queue, and the patterns that appear there feed a new version of the instruction. The final report states how many items were left unclassified, so nobody confuses "no error" with "complete".

The gain here is not speed. It is being able to process a large backlog at a lower cost, without taking capacity away from real-time support, while keeping every model decision traceable.

How do you manage cost, privacy and quality?

Cost. Batch mode is usually cheaper per token, but the total cost depends on the size of each item and on the instruction repeated in all of them. Understanding what a token is helps you estimate volume: see what tokens are in AI. Long, identical instructions across all items are candidates for prompt caching, when the provider offers that combination with batch mode. Anthropic's prompt caching documentation explains the mechanism, and our guide to reducing token costs with prompt caching, routing and compaction covers the practical side. Check whether and how it applies to your scenario. Measure the cost per completed and valid item, not just the cost of the batch, because resent and reviewed items also count.

Privacy. A batch tends to concentrate a lot of personal data in a single submission, which raises the impact of a mistake. Send only the necessary fields, anonymize what you can and record the purpose of the processing. If the provider processes data outside Brazil, LGPD has specific requirements; the article on international data transfer under LGPD when using AI providers details what to check. For official guidance, the ANPD website is the starting point.

Quality. Because nobody follows the batch item by item, evaluation must be planned beforehand. Build a set of examples with the expected answer, run the instruction on it and only then scale. OpenAI's evals guide shows a structured way to think about this, and the logic applies to any provider. After the batch, sample a portion of the valid results for human review; a correct format does not guarantee correct content.

How does this work fit into Juan Carlo's method?

Batch processing follows the same sequence as any AI automation. Map: understand where the data comes from, who uses the result and what delay is acceptable. Limit: define which fields the model receives, which categories are allowed and what happens when there is doubt. Pilot: run a real flow at reduced scale, with a record of each item and a success metric agreed in advance. Transfer: hand over the operation with documentation, the reconciliation report and a clear decision to expand, adjust or stop. To structure the pilot decision, see how to measure an AI automation pilot.

On what is official documentation, what is implementation experience and what is hypothesis: the existence of batch APIs and the idempotency principle are documented by providers. The split into failure categories and the reconciliation flow described above are implementation practices. The idea that your specific task tolerates delay is a hypothesis, and only the pilot confirms it.

Frequently asked questions

When is a batch API a better choice than normal calls?

When there are many similar items, nobody is waiting for the answer in real time and the result feeds a report, a database or a later step. If a person is waiting on a screen or in a conversation, keep the call synchronous. When in doubt, ask the responsible team what delay is acceptable.

What should I do when part of the batch comes back with errors?

Classify the error before resending. Transient failures can be resent with waiting and an attempt limit; malformed items need correction or human review. Resend only the items that failed, keeping the same identifier to avoid duplicate writes.

Is the order of results the same as the order of submission?

Do not count on it. Providers generally return results without a guaranteed order and expect you to match each response to its request using the identifier you defined. Confirm this behavior in the documentation of the provider you use.

Can I send customer data in a batch?

Yes, as long as there is a legal basis, a clear purpose and only the necessary fields. Because a batch concentrates many records, a configuration mistake has a larger impact. Evaluate anonymization, the provider's processing location and LGPD requirements before sending.

Sources

• Stripe: requisições idempotentes
• OpenAI: guia de avaliações (evals)
• Anthropic: prompt caching
• ANPD: Autoridade Nacional de Proteção de Dados