RAG in Practice: Chunking, Embeddings and Reranking Explained

RAG in Practice: Chunking, Embeddings and Reranking Explained

To build RAG in practice, you prepare the documents, split them into chunks, turn each chunk into an embedding, retrieve the chunks closest to the question, reorder the candidates with reranking, and only then hand the context to the model to answer. Quality is rarely lost at answer generation. It is lost earlier: badly extracted text, badly cut chunks and a search nobody tested. That is why the safest path is to test each stage separately, using real questions with known answers.

Key takeaways

• RAG is a chain of stages. If the right chunk never reaches the model, no prompt will fix the answer.
• Build a reference set first: real questions from your operation, each paired with the document and passage that answer it.
• How you split the text matters more than which embedding model you pick. Cutting an exception away from its rule produces wrong answers that look right.
• Reranking is a second screening pass. Add it when the right chunk shows up among the candidates but not at the top.
• Make the system admit when it found nothing. Testing the "I don't know" answer matters as much as testing the correct ones.

What happens in a RAG pipeline, from document to answer?

RAG (retrieval-augmented generation) means the model answers based on passages from your own documents, retrieved at the moment of the question, instead of relying only on what it learned in training. AWS explains the idea in its introduction to retrieval-augmented generation, and Google Cloud publishes a reference architecture with the same building blocks. In practice there are six stages:

• Ingestion: extract the text from the files and attach metadata (source, version, department, permissions).
• Chunking: cut the text into pieces that make sense on their own.
• Embeddings: turn each chunk into a vector that represents its meaning.
• Retrieval: find the chunks closest to the question.
• Reranking: reorder the candidates using a finer criterion.
• Generation: the model answers using the selected chunks and cites the source.

The order matters because errors propagate. A PDF extracted with a scrambled table produces bad chunks, which produce bad vectors, which produce bad retrieval, and in the end the model answers with confidence about something wrong. If you only evaluate the final answer, you cannot tell which stage failed, and you end up editing the prompt with no result.

The first artifact, before any code, is the reference set. Ask the people who handle customers or run the process to list real questions, written the way people actually ask them (including abbreviations and typos, as on WhatsApp). For each question, record which document and which passage answer it. This set becomes the yardstick for every stage below.

Stage 1: do your documents reach the pipeline clean?

The cheapest quality loss to prevent happens at ingestion. The most common problems:

• Scanned PDFs with no recognized text, which become empty pages.
• Tables that, once extracted, turn into a sequence of unrelated cells.
• Headers and footers repeated on every page, polluting each chunk.
• Several versions of the same document, where the old policy keeps answering.
• Documents that certain people should not see, indexed for everyone.

The test is simple: take a few representative documents, open the extracted text (not the original file) and read it as if you were the model. If you cannot make sense of a table or a list in that text, the system will not either.

Treat metadata as part of the content. Store at least the origin, the version or effective date, and the access permission. This lets you filter retrieval (only current documents, only those the user may see) and cite the source in the answer. If the documents contain personal data, decide beforehand what goes into the index and who is accountable for it. The article on a data protection impact assessment for AI projects helps you decide when that analysis is needed.

How do you split text into chunks without breaking its meaning?

Chunking is the stage where the most quality is lost without anyone noticing. Chunks that are too large mix topics and dilute the signal in retrieval. Chunks that are too small lose context: a sentence like "in that case, the deadline is longer" means nothing without what came before it.

The most common approaches, from simplest to most careful:

• Fixed size with overlap: cuts by amount of text and repeats a piece between neighboring chunks. It is fast, but it ignores structure.
• By structure: respects headings, sections, numbered items and clauses. Usually the best starting point for manuals, contracts and policies.
• By unit of meaning: a FAQ question with its answer, a product with its spec sheet, a clause with its exceptions all stay together.

Two practices improve results a lot. The first is to prefix each chunk with the document title and section title, so it carries its own context. The second is to keep the rule and the exception in the same chunk, or in neighboring chunks that retrieval can bring back together.

The test for this stage involves no model at all. For each question in the reference set, find by hand the chunk that answers it and check three things: does the whole answer fit there? Does the chunk make sense on its own? Is the important exception included? Every "no" is a chunking problem to fix before you spend time on retrieval.

Embeddings: how do you choose a model and test retrieval?

An embedding turns text into numbers so that chunks with similar meaning end up close together. For most companies, the main criterion is performance on your own language and vocabulary, not the fame of the model. Two practical cautions: use the same model to index the documents and to embed the questions, and reindex everything if you switch models.

Search by meaning has a known weak spot: codes, acronyms, order numbers and proper names. A question about a specific product code may return chunks that are similar in topic but belong to the wrong product. In those cases, combine vector search with keyword search (hybrid search) and use metadata filters.

The test for this stage is about retrieval. For each question in the reference set, run the search and check whether the correct chunk appears among the top results. Record a hit or a miss, question by question. Then:

• Rewrite each question two or three ways (formal, informal, with a typo) and repeat.
• Include questions with codes and acronyms to see whether hybrid search is needed.
• Look at the misses one by one. Often the problem is in the chunking or in duplicated documents, not in the embedding model.

Only change models after ruling those causes out. Swapping the model is the most expensive change and the one least likely to solve the problem.

Reranking: when is it worth adding this stage?

Vector search is fast and returns a broad list of candidates, but it orders them only approximately. Reranking is a second pass: a model reads the question and each candidate together and reorders them by relevance. It is more precise and slower, so it is applied only to a small group of candidates.

A simple decision guide:

• If the right chunk almost always appears in the first results, reranking adds little. Do not add it.
• If the right chunk appears among the candidates but sits in low positions, reranking tends to help.
• If the right chunk does not appear among the candidates at all, the problem is earlier: chunking, embeddings or documents. Reranking cannot create what retrieval did not bring back.
• If your support flow needs fast replies, measure the extra latency before deciding.

To test it, use the same reference set and compare the position of the correct chunk before and after reranking. If the position improves in most cases and latency stays acceptable, the stage pays for itself. If not, remove it and keep things simple.

Also set a relevance threshold. When no candidate is good enough, the system should say it found nothing instead of handing the least bad option to the model.

How should the final answer use the retrieved context?

After retrieval, the challenge is to give the model little context, and good context. More chunks do not mean a better answer; irrelevant chunks compete for the model's attention. Anthropic discusses this in its engineering write-up on context engineering for agents. In practice, send a few chunks with the source identified, and instruct the model to answer only from them and to cite where each claim came from.

Three tests for this stage:

• A question with no answer in the documents: the system should admit it does not know and, where appropriate, hand off to a person.
• Conflicting documents, such as two versions of a policy: the system should prefer the current one or flag the doubt.
• A document with an embedded instruction, such as text telling the model to ignore its rules: the system must not obey. This risk is described in the OWASP Top 10 for LLM applications and is handled with output validation and action limits, as covered in the article on guardrails for AI agents.

Keep logs of every answer: the question, the retrieved chunks, the document version and the answer given. Without them, when someone complains about a wrong answer, nobody will know which stage failed.

A step-by-step plan, and how we apply the method

A lean roadmap for your first RAG:

• Choose a single set of documents with a clear owner and some stability (a policy manual, for example).
• Build the reference set with real questions and the chunk that answers each one.
• Review the extracted text and fix the problem documents.
• Define chunking by structure, with title and section in each chunk, and validate it by hand.
• Choose the embedding model, index and measure retrieval. Add hybrid search if codes and acronyms fail.
• Add reranking only if the diagnosis shows the right chunk in a low position.
• Test answers, including questions with no answer and conflicting documents.
• Rerun the reference set after every change, as in a prompt regression test.

Illustrative scenario: a distributor wants its support team to consult the exchange and returns policy. In the first test, retrieval brings back the general rule but not the exception for discounted products, which sat on another page of the PDF. The error is not in the model or in reranking: chunking separated the rule from its exception. After reorganizing the chunks by section and keeping the exception next to the rule, the same question is answered correctly, and the reference set confirms that the other questions did not get worse.

The method we use follows the same reasoning. Map: understand the current process, which questions come up and where the answer lives today. Limit: choose a few documents and define what the system may answer and when it must pass to a person. Pilot: run a real flow with logs and an agreed success metric, as described in how to measure an AI automation pilot. Hand over: deliver the documented operation, including the reference set, and decide with the team whether to expand, adjust or stop.

Frequently asked questions

Do I need a vector database to start with RAG? Not necessarily. For a small set of documents, a vector extension on the database you already use or a simple in-memory index is often enough. What you cannot skip is the reference set to measure retrieval, because it works with whatever technology you choose later.

RAG or fine-tuning: which should I choose to answer from my own documents? For content that changes and needs a citable source, such as policies, manuals and catalogs, RAG is usually the most direct path, because you only need to update the document. Fine-tuning is better for adjusting style or behavior than for memorizing facts. The two can coexist, but start with RAG and measure.

How do I know if my RAG is good enough to use with customers? Define what "good enough" means before the pilot: for example, the correct chunk appearing among the first results for the reference set questions, and the system correctly declining questions with no answer. Run the pilot with human review and logs, and expand only when the remaining errors are known and manageable.

Can I index documents that contain personal data? It depends on the purpose, the legal basis and who may access what. Avoid indexing personal data the use case does not require, apply permissions at retrieval and record who queried what. If in doubt, consult your legal team and follow the guidance of Brazil's data protection authority (ANPD).

Sources

• AWS: What is retrieval-augmented generation
• Google Cloud: retrieval-augmented generation architecture
• Anthropic: Effective context engineering for AI agents
• OWASP Top 10 for LLM Applications