Small Language Models for Business Tasks: When They Are Enough

Small Language Models for Business Tasks: When They Are Enough

Small language models (SLMs) are often enough for closed, repetitive tasks such as classifying messages, extracting fields from documents and routing requests to the right queue. To find out whether they are enough in your company, do not rely on opinion. Build a set of real examples, run a small model and a larger one side by side, and compare errors, cost and response time. If the quality gap does not change the outcome of the process, the smaller model is the simpler one to operate.

Key takeaways

• Small models work best when the output is short and defined: one category, one field, one destination.
• Long reasoning, sensitive free text and ambiguous cases still call for a larger model or human review.
• The test is a comparison: the same real examples, the same correctness criteria, small model against larger model.
• Measure errors by type, not just overall accuracy, and decide in advance which errors the company can accept.
• A common architecture is routing: the small model handles the simple cases and passes the hard ones up.

What small language models are and why they matter to a business

A small language model has far less total capacity than the large frontier models. In practice, that means faster responses, lower resource use and, in many cases, the option of running on your own infrastructure. There is no official size that separates small from large, and the names change often. So decide by the task, not by the label.

For anyone running an operation, the interest is practical. Tasks that happen many times a day, like reading a message and deciding who should get it, do not need the most expensive intelligence available. Using an oversized model for that adds token cost and latency without improving the result. If you want to understand where that cost comes from, it is worth reading what tokens are in AI and why they set the cost, but the central point here is different: size the model to the work.

Anthropic's documentation on agent patterns describes routing as a pattern in which an input is classified and directed to a specialized follow-up task, and it mentions the idea of sending simple questions to smaller models and hard ones to larger models. That is architectural guidance, not a promise of results. You only learn the results by testing with your own data.

Which tasks usually fit a small model

The most useful criterion is the shape of the output. The more closed the expected answer, the better the chance that a small model can handle it. These are the task families that fit best:

• Classification: sorting messages into categories such as question, request for a duplicate invoice, complaint, cancellation or out-of-scope topic.
• Routing: deciding which queue, person or workflow should receive the request, based on rules you can write down.
• Extraction: pulling a name, order number, date, address or amount from a short text, a message or a form.
• Normalization: standardizing formats such as dates, phone numbers and product names so they enter the system without rework.
• Signal detection: flagging urgency, a request to speak to a human, or personal data that should not be circulating.
• Short summaries in a fixed format: turning a conversation into a few fields so the agent can pick up where it left off.

All of these share something: there is a right answer that someone on the team can check quickly. That is what makes testing possible. With longer documents, such as PDFs with tables, the difficulty changes in nature, and the file format matters more than the model size. The article on extracting data from PDFs with AI, with fields, rules and human review covers that care.

Which tasks still call for a larger model or a person

Not everything fits a small model, and pretending otherwise creates rework. Watch for these signs that the task demands more:

• Multi-step reasoning, such as cross-checking a contract, a history and a policy to decide on an exception.
• Long, ambiguous free text, where context changes the meaning of the answer.
• Writing that needs tone, nuance and care for the brand, such as a reply to a delicate complaint.
• Tool use with consequences, such as changing a record, issuing something or touching money.
• Questions that depend on a lot of supporting material in the same query, which fills up the context window.

In these cases, the usual path is not to swap everything for a large model but to combine them. The small model does the triage, and whatever falls outside its comfort zone moves up to a larger model or to a person. Decisions that affect someone's rights call for even more care, and it is worth knowing what LGPD says about automated decisions and the right to review. Some tasks also simply should not be delegated, which is the subject of the article on what not to delegate to AI in your business.

How to test a small model against a larger one

The test does not need to be sophisticated, but it needs to be honest. The goal is to take the choice out of the realm of intuition. OpenAI's evaluation documentation and Anthropic's material on agent evaluations start from the same principle: define what success is, gather representative cases and measure in a repeatable way. A lean routine:

• Step one: write the task in one sentence and list the allowed outputs. If you cannot list them, the task is still too vague.
• Step two: collect real, recent examples, including the ugly cases, such as messages with typos, transcribed audio, slang and double requests. Remove or mask personal data in line with LGPD before sharing.
• Step three: ask someone on the team to label the correct answer for each example. That is your answer key. Set aside a group of examples that you do not use to tune instructions.
• Step four: run the larger model and the small model with the same instructions, the same input and the same output format requirement.
• Step five: compare by error type, not just total correct answers. Getting the category wrong on a simple question is different from missing a cancellation request or a serious complaint.
• Step six: improve the small model's instructions with examples and clearer category definitions, then run it again on the held-out group. Much of the initial gap disappears with better instructions.
• Step seven: also record response time, format failures and cost per task, using the same unit for both models.
• Step eight: decide with a rule written before the test, for example: the small model is accepted if it makes no errors on the critical error types and the remaining errors can be recovered through review.

One frequent mistake is using the larger model as the judge of the smaller one without checking the judge. That can work as support, but the reference answer key should come from people who know the process. For thinking about metrics and decision criteria for a pilot, the article on how to measure an AI automation pilot complements this routine.

Illustrative scenario: message triage at a building supply store

Imagine a building supply store that receives WhatsApp messages about quotes, delivery, exchanges and billing. Today one person reads everything and forwards it. The company wants an AI to indicate the category and extract the order number when there is one. This is an illustrative scenario, not a real case.

The team gathers a set of old conversations, masks names and phone numbers, and an experienced agent marks the correct category and order number for each one. Then they run the small model and a larger model with the same instructions. When they compare, they find that the small model gets the common categories right, but confuses an exchange with a complaint when the message is long and emotional, and sometimes picks up a street number instead of the order number.

Instead of abandoning the small model, the team refines the definitions, adds examples of exchanges and complaints, and creates a rule: when the small model is not confident or the message is long, the conversation goes up to the larger model or to a person. Order number extraction goes through a simple check against the expected format. The result of this design is a flow in which the small model handles the volume and the rest gets more careful treatment, with a record of every decision.

Cost, latency, privacy and operations: what weighs most in the choice

Answer quality is only one criterion. In operations, four others tend to decide:

• Cost per task: smaller models tend to cost less per call, which matters when volume is high. Techniques such as instruction caching also help, as shown in the article on reducing token costs with caching, routing and compaction.
• Latency: in message-based support, a fast classification response improves the whole flow, because the following steps start earlier.
• Privacy: running a model on your own infrastructure can reduce the amount of data sent outside, which helps with LGPD concerns, although it does not remove the need for access control and a legal basis. For that path, see running Ollama local models in a company: a practical guide.
• Maintenance: your own model requires someone to handle updates, monitoring and capacity. A model bought as a service transfers part of that, but creates vendor dependence.

None of these criteria is decisive alone. A cheaper local model that errs on critical types does not help, and an excellent model that the team cannot operate does not help either. That is why model outputs, small or large, should pass through validation. The article on guardrails for AI agents, covering input, output and action controls explains how to do it.

How the map, limit, pilot and transfer method applies

Choosing the model size fits naturally into the method we use. In map, we describe the current process and measure the baseline: how many messages, which categories, how much time and which errors happen today. In limit, we define which data the model sees and which actions it can take, and classifying or extracting is usually a low-risk start because the AI suggests and the system validates.

In pilot, we choose a real workflow, with logs and an agreed success criterion, and that is where the comparative test between the small and larger model takes place. In transfer, we hand over the workflow with the decision log and the review rule, and the company decides whether to expand, adjust or stop. If the small model does not pass the test, that is also a useful result.

Frequently asked questions

Are small language models suitable for WhatsApp customer support?

They work well for closed steps of support, such as identifying the intent of a message, extracting the order number and choosing the queue. For long replies, delicate tone or answers that require consulting several documents, use a larger model or have a human review. The ideal is to test with real messages before deciding.

How many examples do I need to test a small model against a larger one?

There is no magic number. You need enough examples to cover every category, including rare and difficult cases, plus a separate group that you do not use to tune the instructions. Start with what the team can label carefully and add more if the results look unstable.

Can I use the larger model only when the small one is unsure?

Yes, and it is a common routing design. The small model resolves the clear case and escalates the ambiguous one, either by rule or through a low-confidence signal. Test that escalation rule too, because it can fail, and keep a path for a person to take over.

Does running a small model in-house solve LGPD?

Not by itself. Keeping data inside your infrastructure can reduce transfer risks, but the company still needs a legal basis, a clear purpose, access control, defined retention and a record of what the system does. For specific questions, consult a legal professional and the ANPD guidance.

Sources

• Anthropic: Building effective agents
• Anthropic: Demystifying evals for AI agents
• OpenAI: Evals guide
• ANPD: Autoridade Nacional de Proteção de Dados