Portuguese Embeddings Model Choice: How to Select and Test One

Portuguese Embeddings Model Choice: How to Select and Test One

For Portuguese embeddings model choice, do not start from a public ranking. Build a set of real queries from your own company, mark the correct answer for each one, and compare candidate models by retrieval accuracy on your own documents. The best model is the one that finds the right passage for the questions your customers and your team actually ask, even if it does not lead any table.

Key takeaways

• Public rankings measure generic tasks and generic text. Your documents, your vocabulary and your internal acronyms decide the result in practice.
• The most valuable asset in the test is the set of real queries, with the correct source marked by someone who knows the business.
• Compare models with simple retrieval measures (does the right passage appear among the first results?) before looking at any AI-generated answer.
• Cost, latency, where data is processed and LGPD implications belong in the decision alongside accuracy.
• Treat the choice as reversible: keep the test set and repeat the comparison when you change models or change your document base.

Why can't a public ranking decide for you?

An embeddings model turns a piece of text into a list of numbers that represents its meaning. Texts with similar meaning end up close together in that space, which is what lets a search find "second copy of the invoice" when the document says "billing reissue". This mechanism is the foundation of the retrieval step in retrieval-augmented generation (RAG) systems, as described in the AWS documentation on RAG and in Google Cloud's reference architecture for RAG-capable applications.

The problem with choosing by ranking alone is that a ranking answers a different question. Public benchmarks bundle diverse tasks, usually with open-domain text, and many have limited coverage of Brazilian Portuguese. Even when they do cover it, your case has particularities that no table captures:

• Your own vocabulary: product codes, acronyms, plan names, industry terms.
• Informal register: WhatsApp messages with abbreviations, typos and transcribed audio.
• Regional variation and jargon: the same item has different names depending on the region or the team.
• Document format: tables, badly extracted PDFs, long contracts, manuals with numbered sections.

A model can do very well in a ranking and still stumble on the acronyms in your catalog. Another, less celebrated model may do better because your documents are short and direct. Only measuring with your own data answers that.

This is a working hypothesis, not a rule: rankings remain useful for building an initial list of candidates. The mistake is using them as the final decision.

How do you build a set of real queries?

The query set is a list of questions people actually ask, each tied to the document or passage that should be found. It is the central asset of the test, and it is worth more than any individual model. Follow these steps:

• Define the workflow. Pick one concrete use, such as customer questions in support, internal policy lookup or contract search. Keep one set per workflow, because mixing uses hides differences.
• Collect queries from real sources. Use support history, questions the team repeats, emails and messages, always with authorization and within the original purpose of the data.
• Anonymize. Remove names, CPF tax IDs, phone numbers, addresses and any personal data that is not needed for the test. If that is not possible, run the comparison in a controlled environment.
• Keep the mess. Preserve abbreviations, spelling mistakes, incomplete sentences and vague questions. That is how they arrive in practice.
• Mark the correct answer. For each query, someone who knows the business indicates which document or passage answers it. Some queries may have more than one valid passage.
• Include questions with no answer. Set aside queries whose information does not exist in the base. A good system needs to know when it found nothing relevant.
• Freeze part of the set. Reserve queries that nobody uses to tune parameters, so you do not end up optimizing the test instead of the system.

The right size is whatever your team can review carefully. A modest set of well-labeled queries already shows meaningful differences between models. If the difference between two candidates shows up in only a few cases, that is a sign they are equivalent for your use, and the decision should shift to cost and operations.

One detail that is often forgotten: the labeling should be done by the people who answer these questions day to day, not by the people building the system. Builders tend to mark what the system already gets right.

Which metrics should you use and how do you run the comparison?

Measure retrieval separately from the final answer. If you only evaluate the text the AI generates, you cannot tell whether the error came from the search or from the generation. Three simple measures cover most cases:

• Hit rate in the top k results: for what share of queries does the correct passage appear among the first k returned. Choose k according to how many passages your system actually sends to the language model.
• Position of the first hit: does the right passage come first or is it buried at the bottom? This matters when the context you send is small.
• False-find rate: on queries with no answer in the base, how often does the system return something that looks relevant but is not.

The mechanics of the comparison are straightforward. Index exactly the same documents with each candidate model, using the same chunk size and the same splitting strategy. Run all the queries, record the results and calculate the measures. Change one variable at a time: if you change the model and the chunk size together, you will not know what caused the difference.

Then read the errors one by one. Spreadsheets with averages hide patterns that reading reveals: the model fails on acronyms, on negative questions ("when can't I exchange"), on numbers, on very short queries. These failures tell you whether the problem can be solved with a different search strategy, for example combining keyword search with semantic search, or whether it requires changing the model.

To turn evaluation into a routine, the OpenAI guides on evals and Anthropic's writing on evaluations for AI agents help with thinking about test cases, clear criteria and repetition. They do not address embeddings specifically, but the logic of a fixed set, a defined metric and review of failures applies.

Illustrative scenario: policy search at a services company

Illustrative scenario: a services company with WhatsApp support wants an agent to consult its policy base (exchanges, deadlines, warranties, rescheduling rules) to support its agents. Three embeddings models enter the comparison: a widely used multilingual model, a model from a cloud provider and an open model that could run on the company's own infrastructure.

The operations team gathers real questions from the support history, already stripped of personal data. Phrases like "can I reschedule last minute?", "what's the refund deadline" and "double charge" go in. The support lead marks, for each question, the policy passage that answers it. She also sets aside a few questions the base does not cover.

The three models index the same base. Reading the errors, the team notices that one of them confuses "refund" with "chargeback reversal", two terms the company treats as different processes, while another gets very short questions wrong. The third performs similarly to the first but requires running its own infrastructure.

In this scenario the decision does not come from a table. It comes from a combination of accuracy on the most frequent queries, behavior on questions with no answer and the effort of operating the model. If the two best candidates tie on accuracy, the tiebreakers become cost, latency and where the data is processed.

What criteria beyond accuracy belong in the decision?

After retrieval accuracy, several practical factors determine which model goes to the pilot. Use this list as a decision guide:

• Where data is processed. Sending text to an external API means transferring company content, sometimes including personal data, to a third party. Check the provider's terms on retention and data use, and compare with the alternative of running an open model internally. The article on self-hosted LLM vs API follows similar reasoning.
• Roles under LGPD. If the provider processes personal data on the company's behalf, the contract needs to reflect that. It is worth reviewing who is the controller and who is the processor and assessing whether the project calls for a data protection impact assessment. Specific regulatory questions should go through your legal team, and the ANPD (Brazil's data protection authority) is the official reference.
• Cost to index and to query. Consider the volume of documents, how often they are updated and how many queries per day the system will receive. Reindexing the whole base when you switch models also costs time.
• Latency. In live support, the search needs to be fast enough not to delay the reply.
• Maximum text length. Models accept inputs of different sizes, which affects how you split long documents.
• Dependency and continuity. If the provider discontinues the model, you will need to reindex. Keeping the test set turns that switch into a comparison, not a surprise.

None of these items appear in a public ranking, and every one of them can flip the choice between two models with similar accuracy.

How do you apply this with a method and keep the choice sound?

The reasoning follows the four steps of the method we use in AI projects: map, limit, pilot and transfer.

• Map: describe the current workflow, who asks what, where the answer lives today and how errors happen. This map produces the query set and the baseline for later comparisons.
• Limit: restrict the data that enters the test and the indexed base. Start with documents that contain no personal data and expand only if there is a need and a legal basis.
• Pilot: put the chosen model into a real workflow, with logging of queries and retrieved passages, and a success metric agreed in advance. The article on how to measure an AI automation pilot details decision criteria for continuing, adjusting or stopping.
• Transfer: hand the test set, the comparison procedure and the switching criteria to the responsible team, so the company does not depend on whoever built it.

After the pilot, treat the query set as a regression test. Whenever you change the model, the chunk size or the document base, run it again and compare. The logic is the same one described in prompt regression testing in CI: without a fixed set, a drop in quality goes unnoticed. Also use queries that failed in production, with human review, to grow the set over time.

Frequently asked questions

What is the best embeddings model for Portuguese? There is no single best model in general. Multilingual models are usually a good starting point, but performance depends on your vocabulary, your document format and the type of question. The reliable way to know is to compare candidates using real queries from your company.

Do I need a model trained specifically on Portuguese? Not always. A multilingual model can serve well if the test with your queries shows good accuracy, and a Portuguese-specific model may not be worth it if it performs worse on your documents. Include both types among the candidates and let the test result decide.

How many queries do I need for the test to be reliable? There is no magic number, and it depends on the variety of your use. Start with what your team can label carefully, covering the most frequent topics and a few hard cases, and expand over time. If two models end up close, growing the set on the critical topics helps separate the candidates.

Do embeddings replace keyword search? Not always. Semantic search helps when the words in the question differ from those in the document, but codes, acronyms and exact names are usually better served by text search. Often the best solution combines both, and your query set shows whether that combination improves accuracy.

Sources

• AWS: What is retrieval-augmented generation
• Google Cloud: retrieval-augmented generation architecture
• OpenAI: evals guide
• Anthropic: Demystifying evals for AI agents