Hardware to Run Local LLMs: GPU, Memory and What to Buy

Hardware to Run Local LLMs: GPU, Memory and What to Buy

The hardware to run a local LLM comes down to one thing above all: the memory on the graphics card (VRAM), or unified memory on a Mac. The model has to fit in that memory with room left over for context. The safe path is to pick the smallest model that does the job, estimate how many people will use it at the same time, and only then choose the computer. Buying for GPU power without checking model size is the most common mistake.

Key takeaways

• VRAM decides which models run well. GPU speed decides how fast the answers come, but it only matters once the model fits.
• Model size and quantization level determine the memory required. Long context and several simultaneous users increase that demand.
• A well-defined task, such as classifying messages or summarizing documents, often works with small models. That cuts the hardware you need considerably.
• Test the candidate model on a borrowed or rented machine before buying. The decision should come from a pilot with real use, not from a spec sheet.
• Local hardware gives you privacy and control, but it also brings maintenance, power needs and security responsibility. Not every case needs it.

Why VRAM matters more than raw GPU power

A language model is, in practice, a large set of numbers called parameters. To generate text, those numbers must sit in the fastest memory available. On a dedicated graphics card, that memory is VRAM. If the model does not fit, the system shifts part of the work to ordinary RAM and the processor, and responses become much slower.

That is why two cards with similar power can feel very different. The one with more VRAM runs larger models or longer context. The one with less forces you to choose a smaller model or accept slowness. For a business owner, the useful question is not which GPU is the strongest, but what is the largest model the task really requires.

The documentation for each local runtime tool usually lists memory requirements per model. Treat those figures as a starting point and confirm them with your own use, because consumption changes with context size and configuration.

How to match model size, memory and expected use

Three variables add up to the memory you need.

• Model size: the more parameters, the more memory the weights take.
• Quantization: a technique that stores the weights with less precision to take up less space. It generally reduces memory at a cost in quality that depends on the model and the task, so it needs to be tested.
• Context and concurrency: the text the model reads with each request creates additional memory use (the context cache). Long context and several conversations at once increase it. To understand this point, read about the context window in AI.

The sizing logic goes like this: add the memory for the weights, add the memory for the expected context, and leave a margin for the operating system and for peaks. If the total exceeds your VRAM, you have three options: a smaller model, more aggressive quantization, or more hardware.

Expected use helps you decide. An internal assistant that summarizes documents for one person at a time has one kind of load. An agent that serves several customers at once on WhatsApp has another, because each conversation takes up context and competes for the same GPU. Before thinking about a purchase, write down how many people or workflows will use the model at peak hours and how long the texts are.

Is a small model enough? Test before paying for big hardware

Many business tasks do not need the largest model. Sorting a message by topic, extracting fields from a document, rewriting a text in a standard tone or answering questions about a short knowledge base often works with smaller models, as long as the context is well prepared. The article on small language models for business tasks explains when they are enough and how to set up the test.

The economic reasoning is simple: each step up in model size pushes the hardware requirement higher, and the jump does not always produce a visible gain for the task. Build a small set of real examples from your operation, with the expected answer for each, and compare models of different sizes. If the smallest one delivers what you need, the computer can be smaller too.

Tasks that need long reasoning, complex code or many instructions at once tend to call for larger models. In those cases it may be wiser to use a cloud service than to build a local server, a topic covered further below.

Dedicated GPU, Mac with unified memory or CPU only?

Three families of machine usually come up in conversations about local AI. Each has its own profile.

• PC or server with a dedicated GPU: the most common path for anyone who wants speed and several simultaneous conversations. NVIDIA has the most widely supported ecosystem across runtime tools, while support for other brands varies by tool and version and should be checked in the documentation before you buy. The limit is the card's VRAM.
• Mac with an Apple Silicon chip: uses unified memory, shared between the processor and the GPU. This lets you load larger models than would fit on a consumer card with little VRAM, generally with slower generation speed than a high-performance dedicated GPU. It is an attractive option for individual use or a small team with few simultaneous users.
• CPU-only machine: works for small models and for testing, with slower responses. It is good for validating an idea, but usually not for customer service where people expect fast answers.

The right choice depends on what you need to prioritize: speed, the size of the model that fits, the number of users or simplicity of operation. There is no single answer, and any performance comparison should be run with your model and your workload.

What matters besides the GPU

The budget for a local AI computer does not end at the graphics card. Some items get forgotten until they turn into problems.

• RAM: it has to hold the system, the tools and sometimes part of the model. Too little RAM makes the machine stall even with a good GPU.
• Fast storage: models take up a lot of disk space and load faster from an SSD. If you test several models, the disk fills up quickly.
• Power supply and cooling: powerful cards draw a lot of power and produce heat. In an office without proper ventilation, the machine throttles its performance or fails.
• Electrical stability: a UPS or voltage stabilizer prevents data corruption and restarts during power drops.
• Network and access: if more than one person will use it, you need to define who gets access, from where and with what authentication. A local model exposed without controls becomes a security risk.
• Maintenance: someone has to update the software, watch for failures and swap the model when a better version appears.

On the operations side, the guide to running Ollama in a company shows how to run a local model without improvising, with attention to access, updates and usage logs.

A step-by-step plan to choose without oversizing

Follow this order. It keeps you from buying before you know what you need.

• Describe the task in one sentence, along with the expected result. Example: summarize internal meeting minutes and return action items.
• Define the data involved and the reason to keep it local. If personal data is involved, check the guidance from Brazil's data protection authority (ANPD) and assess the need with your legal team.
• Gather a few real examples, each with the answer you consider good. They will be your test.
• Compare a small model, a medium one and, if necessary, a large one, using a borrowed machine or a server rented for a few days.
• Record the memory used, the perceived speed and the quality of the answer in each test.
• Estimate peak use: how many people at the same time and the typical text size.
• Choose the smallest hardware that handles the peak with some margin. If you are torn between two tiers, start with the smaller one and leave room to expand.
• Compare against the cloud alternative using cost per completed task, a concept explained under unit economics in the FinOps Foundation framework.

Illustrative scenario

Imagine a services company with a small administrative team that wants to summarize contracts and internal memos without sending the documents outside. This is an illustrative scenario, not a real case.

The person in charge lists the task: read a document and return a summary with deadlines and obligations. They set aside a few real documents, write the expected summary for each and test a small model and a medium model on a borrowed machine. The medium model produces slightly better organized summaries, but the small one gets the deadlines right in the examples tested. Expected use is one person at a time, during business hours.

From this, the company learns it does not need a server with several cards. A machine with a moderate-VRAM GPU, a good amount of RAM and a fast SSD covers the pilot. If use grows to several people at once, the decision is revisited with usage data, not by guesswork.

When local hardware is not the best answer

Running locally makes sense when there is a privacy requirement, a need to operate without internet, a need to control the model version, or a steady volume that justifies the investment. It does not make sense when the load is irregular, when the task needs the largest model available or when nobody has time to maintain the machine.

In those cases a cloud service may be a better fit, but it brings its own contract and data questions. If the provider processes data outside Brazil, check what LGPD requires in international data transfer when using AI providers. A mixed setup is also possible: a local model for sensitive data and an external service for tasks that involve no personal data.

The method we apply has four steps: map the process and the current baseline, limit the system's data and actions, pilot a real workflow with logging and an agreed success metric, and hand over operations with a decision to expand, adjust or stop. Applied to hardware, this means the purchase comes after the pilot, not before. For broader risk questions, the NIST AI Risk Management Framework offers a governance reference.

Frequently asked questions

What is the minimum computer setup to run a local LLM?

There is no single minimum, because it depends on the model. Small models run on modest machines, even CPU-only ones, while larger models need a lot of VRAM or unified memory. Check the documentation for the tool and model you plan to use, and confirm in a test.

Is it better to buy a powerful GPU or one with more VRAM?

To run larger models, more VRAM usually takes priority, because the model has to fit in memory. Extra power improves speed, but only once the model fits. Define the model and the context first, then choose the card.

Is a Mac good enough to run local AI in a company?

It works for many individual or small-team use cases, because unified memory allows relatively large models to load. Speed and the number of simultaneous users are usually lower than on a server with a dedicated GPU. Test with your own model before deciding.

Does local AI solve LGPD compliance on its own?

No. Keeping data inside the company reduces some risks, such as sending it to third parties, but purpose, legal basis, access control, retention and security still apply. Treat the local server like any other system that processes personal data, and validate the setup with your legal team.

Sources

• ANPD: Autoridade Nacional de Proteção de Dados
• FinOps Foundation: unit economics
• NIST AI Risk Management Framework