OCR for Brazilian Documents With AI: CNH, RG and Proof of Address

OCR for Brazilian Documents With AI: CNH, RG and Proof of Address

To run OCR on Brazilian documents with AI, send the image to a vision model with a fixed field schema, ask only for what the process actually needs, validate every field with deterministic rules (CPF check digits, date format and consistency), and delete the image as soon as the check is done. The model reads, the code validates, and a person decides the doubtful cases. This design produces a useful result and keeps exposure of personal data to the minimum necessary.

Key takeaways

• Vision models handle documents photographed on a phone with more tolerance for glare, angle and layout than traditional OCR, but they can still get a digit wrong.
• Extraction should use a closed schema of fields, with a null value when the text is unreadable, never a guess.
• Validation is not the model's job: CPF, dates, postal codes and consistency between fields are solved with simple, testable code.
• Minimization is part of the design: extract only what is needed, mask values in logs, set a retention period and do not keep the image without a clear reason.
• OCR does not prove authenticity. Reading a document is different from confirming that it is genuine and belongs to the person who sent it.

Why use a vision model instead of traditional OCR?

Traditional OCR converts pixels into text and leaves the interpretation to you. With Brazilian documents that is usually laborious: the CNH (driver's license) comes in several versions, the RG (national ID) varies by state and by year of issue, and proof-of-address documents change layout with every utility company. Each variation needs position rules or template models.

A vision model receives the image and understands the context. It recognizes that the number next to an expiry label is a date, even when the layout is new. That reduces template maintenance and tolerates phone photos taken at an angle or with reflections.

The trade is predictability. Traditional OCR fails visibly (truncated or missing text). A vision model can return a well-formatted field that is wrong, for example an eight read as a three. That is why the design needs a validation layer after the reading, rather than direct trust in what the model returns. The same logic applies to PDFs, as we show in extracting data from PDFs with AI: fields, rules and human review.

Which fields should you extract from a CNH, RG and proof of address?

Start from the business question: what does the process need to know? Someone registering a customer rarely needs every field on the document. A lean list per document type helps you decide.

On the CNH, the most common fields are name, CPF (taxpayer number), date of birth, registration number, category, expiry date, date of first license and place of issue. If the goal is to check that the license is currently valid, for example in an equipment rental, name, category and expiry are enough.

On the RG, you usually find name, parents' names, date of birth, place of birth, registration number and issue date. The number format varies between states, and newer versions of the identity card may use the CPF as the registration number. Do not assume a single pattern: treat the RG number as free text with light validation.

On proof of address (electricity, water, phone or internet bill), the useful fields are account holder name, full address, postal code (CEP), issuing company name and issue or due date. The date matters because many processes ask for a recent document, and the recency rule is yours, not the model's.

A practical rule: if you cannot explain what a field will be used for, do not extract it.

How to ask the model for the extraction

The instruction should be short, with structured output and no room for creativity. The points that matter most:

• Define the document type up front, or ask the model to classify it as CNH, RG, proof of address or other, with the option to decline.
• Provide a fixed output schema, with the name, type and expected format of each field.
• Instruct the model to return unreadable or missing fields as null, with an explicit ban on filling in or deducing values.
• Ask for dates in ISO format and for the CPF as digits only, to make the later validation easier.
• Ask for an image quality indicator (readable, partially readable, unreadable) so you can decide whether to request a new photo.

There is also a security risk that is rarely mentioned: the image is untrusted input. Text written on a photo can try to give instructions to the model. Always treat the document content as data, never as a command, and limit what the system can do with the response. The OWASP Top 10 for LLM applications describes this kind of risk, and our piece on guardrails for AI agents explains how to validate input, output and action.

How to validate CPF, dates and other formats

Validation is ordinary code: fast and easy to test. It turns text read by a model into data you can trust a little more.

• CPF: check that it has eleven digits, discard sequences where all digits are the same, and calculate the two check digits with the official algorithm (modulo eleven). If the calculation does not match, the field goes to review or triggers a request for a new photo.
• Dates: confirm the day, month and year format, reject impossible dates and check consistency. A birth date cannot be in the future, the CNH expiry must be later than the first license date, and the resulting age must be plausible for the process.
• Validity: compare the document's expiry with today's date. An expired document is not a reading error, it is a business rule that deserves its own response.
• CEP (postal code): eight digits. If you query an address service, compare the city and state with what the proof of address says.
• Name: compare with the name the customer provided, normalizing accents, capitalization and abbreviations, and use a small, defined tolerance. Never make it an automatic rejection decision.
• Consistency across documents: if the customer sends a CNH and a proof of address, the names should be compatible. A mismatch is not fraud, it is a reason for human review.

A CPF with correct check digits is well formed, but that does not prove it exists, is in good standing or belongs to the person. That kind of confirmation requires an official source or a contracted verification service, with a clear legal basis and purpose.

How to apply data minimization in the flow

Brazil's data protection law (LGPD) requires processing to use only what is necessary for the purpose. In document OCR, that translates into concrete architecture decisions, not a generic policy text. The website of the ANPD, the Brazilian data protection authority, is the starting point for following official guidance.

• Defined purpose: write in one sentence why the document is read (for example, confirming ownership of an account) and do not reuse it for something else without reassessing.
• Minimum fields: extract only what the purpose requires. If the expiry date is enough, do not store parents' names or place of birth.
• Ephemeral image: process the image and discard it. If you must keep it for audit, set a retention period, a restricted-access location and an owner.
• Masked logs: record that the CPF was validated, not the full CPF. Show only the last digits when a check is needed.
• Model provider: read the retention and data use terms of the API you contract, confirm that submitted data stays out of training, and assess where it is processed.
• Role-based access: only the people who handle exceptions see the document, and every access is logged.
• Data subject rights: have a path to handle requests for access, correction and deletion, including in copies and logs.

If the flow handles many documents or involves decisions that affect the person, consider an impact assessment. The article on data protection impact assessments for AI projects explains when to do one and what to include. And if reading the document feeds an automatic approval or rejection, also read about LGPD automated decision-making.

Illustrative scenario: onboarding over WhatsApp

Imagine an equipment rental company that registers customers over WhatsApp. This is an illustrative scenario, not a real case. Today, an agent receives the photo of the CNH and the proof of address and types the data into the system.

In the redesigned flow, the customer sends the two photos. The system classifies each image and extracts only the name, CPF, category and expiry from the CNH, and the name, address and date from the proof of address. The code validates the CPF, the license validity and the proof-of-address date. The agent then sends the customer a short summary and asks for confirmation: are the name and the last digits of the CPF correct?

If any field fails validation or the image is poor, the agent asks for a new photo, at most once, and then hands over to a human agent. When the registration is complete, the images are removed from temporary storage, and the system keeps the necessary fields, with the CPF masked in logs. The human agent remains responsible for the final approval.

The conversation side depends on good channel practices. For the technical foundation, see how to integrate AI into your company's WhatsApp.

Putting it into practice: map, limit, pilot and hand over

The method we use in AI projects fits this case well, with no shortcuts.

• Map: describe how the document arrives, who reads it today, which decision depends on it and which errors already happen. This survey is the baseline for comparison later.
• Limit: choose the minimum fields, define what the agent never does (approve, reject, query external databases without authorization) and restrict data and actions.
• Pilot: run a single real flow, with logs, a set of test documents and an agreed success metric, such as the rate of fields correct without human edits. The guide on how to measure an AI automation pilot helps you define decision criteria.
• Hand over: deliver the operation with documentation, owners and a review routine, and decide from the data whether the pilot expands, changes or ends.

For the test set, build varied examples: dark photos, cropped documents, old and new versions, proofs of address from several companies. Anthropic's material on evaluating AI agents helps structure this kind of test, and it is worth repeating whenever you change the model or the instructions.

Reading the document is automatic. Trusting it is a process decision.

Frequently asked questions

Does a vision model replace traditional OCR? In many cases yes, especially when documents vary in layout and arrive as phone photos. For large volumes of a single, well-controlled format, traditional OCR can be cheaper and more predictable. The criterion is to measure both on your own set of documents.

Can I keep the photo of the document? It depends on the purpose and legal basis of your process, and the starting rule is to keep it only if there is a clear need, for a defined period and with restricted access. If the image only serves to extract fields, discarding it after the check is the safest option. When in doubt, consult your data protection officer or a data protection lawyer.

Does CPF validation confirm that the person is who they say they are? No. It confirms that the number is well formed according to the check digit calculation. Verifying registration status or identity requires another source and another data processing activity, with its own purpose and legal basis.

What should happen when the model cannot read a field? Return the field as null and follow the exception flow: ask for a new photo once and, if the problem persists, hand over to a person. Never fill in the field through the model's deduction, because a silent error in personal data costs more than a manual check.

Sources

• ANPD: Autoridade Nacional de Proteção de Dados
• OWASP Top 10 for LLM Applications
• Anthropic: Demystifying evals for AI agents