Running Ollama Local Models in a Company: A Practical Guide

Running Ollama Local Models in a Company: A Practical Guide

Running Ollama in a company is viable when the server has an owner, hardware sized for the real workload, controlled access, a closed network and an update process. Without those, it remains an excellent testing tool but not infrastructure. If nobody wants to take on that work, an API with a contract and logs is usually the more prudent choice.

Key takeaways

• Ollama runs open models on a machine you control, but it does not provide governance, user authentication or monitoring on its own.
• Size the hardware for real use (how many people, which tasks, what context size), not for what runs well in a demo.
• Never expose the server directly to the internet or to the whole internal network: put an authenticated proxy in front and limit who can reach it.
• Models and versions change how answers behave, so updates need prior testing, version records and a way to roll back.
• If your team has nobody to operate the server, or volume is low and irregular, an API with a contract is usually safer and cheaper to maintain.

What changes when Ollama leaves the tester's laptop?

On a developer's laptop, Ollama has one user, test data and no consequences if it crashes. In a company, three things change at once: several people start to depend on it, the data becomes real, and downtime becomes an operations problem.

The useful question stops being "does the model answer well?" and becomes "who looks after this on Monday morning?" That includes who updates it, who grants access, who reads the log when something odd happens and who decides to switch it off. Without those answers, the project depends on one person and one machine, and that is the kind of improvisation that leads to incidents.

There is also the type of work to consider. A local model makes sense for well-defined tasks: summarizing internal documents, classifying messages, extracting fields from text, supporting search over your own knowledge base. For tasks that need long reasoning or complex tools, small open models may fall short, and that has to be measured with examples from your own operation, not assumed.

How do you size the hardware without guessing?

The central issue is memory. The model must fit in the graphics card's memory (or in system memory, with much lower performance) and still leave room for the conversation context. The more text you send with each request, the more memory is used on top of the model itself. To understand why text length matters, read about tokens in AI and about the context window.

A simple sizing routine:

• Define the tasks: what will be asked of the model, and the typical size of input and output.
• Define concurrency: how many people or systems may ask at the same time during peak hours, not on average.
• Test the candidate models in quantized (compressed) and non-quantized versions, comparing quality on your own samples.
• Measure response time under simultaneous load, because a model that is fast for one person can be slow for ten.
• Leave headroom: the production machine should not run at its memory limit.

Other items are often forgotten: stable power, cooling, disk space for several models, configuration backups and a plan for when the machine fails. If the workflow is critical, a single server is a single point of failure, and that must be accepted knowingly or solved with redundancy.

If the load is irregular (many requests at certain hours, almost none the rest of the time), dedicated hardware sits idle for much of the day. That belongs in the local versus API decision, which the article on self-hosted LLM versus API covers in more detail.

Who can access the model, and how do you control it?

By default, the Ollama server listens only on the machine's own local address, and the OLLAMA_HOST environment variable lets you change that. This is where many installations become a risk: someone changes the configuration so other computers can reach the server, and forgets the service was not designed to serve users directly. Check the documentation for the version you use to see what it offers for authentication, because the typical design is to place a layer in front of it.

That layer can be a reverse proxy or an internal gateway that handles:

• Authentication of whoever is calling, per user or per system, with individual, revocable credentials.
• Authorization by role: who can use which model and for what purpose. OWASP's authorization best practices help structure this.
• Usage limits per person or system, so a script stuck in a loop cannot bring the server down for everyone.
• Logging of who called, when and for what purpose, without necessarily storing the full content of conversations.

Logging deserves its own attention. Keeping everything helps investigate problems, but it creates a new store of personal and confidential data. Decide what to log, for how long and who can read it. For personal data, Brazil's LGPD still applies even when the model runs in-house: running locally reduces what is sent to third parties, but it does not remove the need for purpose, minimization and access control. When the use involves sensitive data or decisions about people, consider a data protection impact assessment for AI projects. Official guidance is published by ANPD, Brazil's data protection authority.

Finally, an internal policy rule: define what may and may not be put into the local model. Having the server in the company does not make every kind of data automatically suitable for it. The reasoning is similar to a company policy for employee ChatGPT use, just with a different perimeter.

Which network security measures are essential?

Think of the model server as any sensitive internal service. The basics are well known, but they tend to be skipped in AI projects because "it's just a test."

• Do not publish the Ollama port to the internet, not even as a temporary convenience. If remote access is needed, use a VPN or an authenticated proxy.
• Place the server in its own network or segment, with firewall rules that allow only the systems that genuinely need to call it.
• Use encryption in transit between the proxy and the clients.
• Keep the operating system, graphics drivers and Ollama itself updated, with an agreed maintenance window.
• Store keys and credentials outside the code and outside loose files on the server.
• Restrict who has administrative access to the machine, because whoever downloads models and changes configuration controls what the system answers.

There is also a risk that no network rule solves: content. If the model reads emails, PDFs or external pages, a malicious text can try to induce unwanted behavior, which is called prompt injection. OWASP's prompt injection prevention guide and its Top 10 for LLM applications are good references. The practical rule: the more actions the model can perform, the more limits and human approval it needs. A model that only suggests text for review is far easier to protect than one that changes records or sends messages.

If the plan includes agents with tools, the OpenClaw Gateway security checklist shows the kind of evidence worth requiring before any exposure.

How should you handle updates to Ollama and to models?

There are two layers of updates, and both change how the system behaves. The first is Ollama itself, which receives fixes and improvements. The second is the models: a new version, or a different model sold under the same name, may answer differently, run slower or use more memory.

For that reason, avoid "update and see what happens" in production. A minimal process:

• Record the Ollama version and the exact identifier of each model in use.
• Keep a set of real (anonymized) examples with expected answers, to compare before and after any change.
• Test the update in a separate environment with the same set of examples.
• Promote to production only if quality and response time stay within what was agreed.
• Keep the previous version and know how to return to it quickly.
• Assign someone to follow security notices for the software and for the models you use.

With that example set, you also get an objective way to compare models when a better option appears, instead of deciding by impression or fashion.

Step by step to a responsible local pilot

The method we apply in AI projects works here too: map, limit, pilot and hand over. With Ollama, it looks like this.

• Map: describe the current process, who does it, how long it takes and where the data lives. Set the baseline before switching on any model.
• Limit: choose a single workflow, the minimum data needed and the permitted actions. Start with the model only suggesting and a human deciding.
• Pilot: run the real workflow in a controlled environment, with logs, an example set and a success metric agreed in advance. The article on how to measure an AI automation pilot details this design.
• Hand over: at the end, document the operation (who updates, who has access, how to roll back) and decide based on results: expand, adjust or stop.

Illustrative scenario: a distribution company wants a model to summarize internal IT support tickets and suggest a category. The tickets contain employee names and sometimes personal data. The company chooses to run locally to keep that content inside its network, puts the server behind an authenticated proxy, grants access only to the ticketing system and leaves the final category to the support agent. After a few weeks, it compares the suggestions with the categories people chose and decides whether to expand. If nobody on the team could maintain the server, the same task could go to an API with an adequate data processing contract.

When is an API wiser than running Ollama?

Running locally is not automatically safer or cheaper. An API is generally the more prudent choice in these cases:

• No person or team is responsible for operating, updating and protecting the server.
• Usage volume is low or very irregular, and the hardware would sit idle.
• The task needs a quality that the open models fitting your hardware do not deliver in your tests.
• You need contractual availability and support that a single server cannot offer.
• The workflow is critical and there is no time budget for redundancy and monitoring.

On the other hand, keeping the model local tends to make more sense when there is a real restriction on where data can live, usage is predictable and steady, tasks are well defined and smaller models can handle them, and a technical team is willing to operate the service. Many companies end up in the middle: sensitive data handled locally and general tasks sent to an API, with clear rules about what goes where.

If the question is which commercial model to pick when the API wins, the article ChatGPT vs Claude vs Gemini for business helps compare criteria. To structure the risk of either route, the NIST AI Risk Management Framework offers a useful vocabulary.

Frequently asked questions

Is Ollama safe to use with company data? It can be, as long as the server sits on a closed network behind authentication, with restricted access and usage logging. The software alone does not settle your data policy: the configuration and the process around it determine the level of security. For personal data, the LGPD still applies.

Do I need a dedicated graphics card? Generally yes, for use by several people at good speed, because the card's memory defines which models fit and how quickly they respond. Running on the processor alone may work for tests or non-urgent batch tasks. The right approach is to test your models and tasks on the candidate machine before buying.

Can I let several employees use the same server? Yes, provided there is an authentication layer and usage limits in front, and the hardware has been sized for peak concurrency. Without that, one person can degrade the service for everyone, and there is no way to know who accessed what. It is also worth defining which data may be sent.

How do I decide between Ollama and an API? Compare operational responsibility, data sensitivity, how regular the volume is and the quality required, using real samples from your operation. If nobody can operate the server or usage is low and irregular, an API tends to be more prudent. If the data must stay internal and usage is steady, the local model becomes a serious option.

Sources

• OWASP: Top 10 para aplicações com LLM
• OWASP: prevenção de injeção de prompt
• OWASP: guia de autorização
• NIST: AI Risk Management Framework