Local models via Ollama

What does this solve?

It is not always possible to transfer correspondence with clients, source codes and internal regulations to someone else's API: in some cases it is prohibited by a contract, in others it is an industry requirement, in others there is simply no Internet at the facility. Ollama solves three issues at once - the data remains on the machine, the token counter does not tick, no one cuts quotas and limits. The entry threshold is lower than that of assembling llama.cpp by hand: downloading the model and launching it fit into two commands. The board has its own hardware and a quality ceiling for models that fit on this hardware.

How it works

Demon and client

Ollama consists of a background service and a CLI. The service listens to local port 11434 and sends HTTP API, the command line is a thin client to the same API. Any integration runs over HTTP, rather than launching a process for each request.

Models are stored in the GGUF format, and the engine of the llama.cpp family runs under the hood. They are downloaded in layers according to the manifest, like container images: identical layers between tags are reused, re-downloading does not start everything over again.

ollama pull <модель>     # скачать
ollama run <модель>      # интерактивный чат
ollama list              # что скачано и сколько занимает
ollama ps                # что сейчас в памяти и на чём считается
ollama show <модель>     # параметры, шаблон чата, лицензия

Quantization and memory

Weights are compressed to 8, 5 or 4 bits, and this is visible right in the tag name (q4_K_M and relatives). Rough estimate of consumption: multiply the number of parameters by the number of bytes per weight - with 4 bits this is half a byte - and add a reserve for the KV cache, which grows with the length of the context and the number of simultaneous requests. Hence the practical consequence: the model fits into the scales, and during a long dialogue it goes into swap and begins to issue a token per second.

Modelfile

Your preset is collected from a text file:

FROM <базовая модель>
SYSTEM "Отвечай кратко, только по документам компании."
PARAMETER temperature 0.2
PARAMETER num_ctx 8192

Team ollama create мой-ассистент -f Modelfile turns this into a regular local tag, which then runs like any other model. Directive TEMPLATE specifies a chat template - the arrangement of service role tokens. It is inherited from the base model, and there is no need to break it unnecessarily: a broken template leads to the fact that the model begins to finish speaking for the user.

API

Three main routes: /api/generate —single text continuation, /api/chat —message dialogue with roles, /api/embeddings —vectors to search. Plus an OpenAI-compatible layer on /v1: Client libraries are switched to the local model by changing the base URL and dummy key. The cheapest way to attach a local launch to code that has already been written for the cloud.

curl http://localhost:11434/api/chat -d '{
  "model": "<модель>",
  "messages": [{"role": "user", "content": "Сформулируй суть в двух предложениях"}],
  "stream": false,
  "options": {"temperature": 0.2, "num_ctx": 8192}
}'

Generation parameters are passed in the object options directly in the request and interrupts what is written into the Modelfile. This is convenient: one service can access the same model with different settings for different tasks, without creating separate tags. In this case, a separate model is taken for embeddings - the generative model will also return vectors, but the quality of the search on them will be noticeably worse than on a specialized one.

Life cycle in memory

The first request loads weights - from seconds to tens of seconds. Then the model hangs in memory and responds quickly. Parameter keep_alive determines how long to keep it after the last call; Environment variables control how many models are allowed to be loaded and how many requests to process in parallel. Two large models at the same time means two portions of memory, not one and a half.

Deserves special attention num_ctx. The context length is set at startup and by default is noticeably less than what the model card promises. The tail of a dialogue or the beginning of a long document is cut off silently, without warning, and from the outside it looks like a “dumb model.”

GPU and CPU

On a video card it is considered multiple times faster, but the model must fit entirely into VRAM. It doesn’t fit - some of the layers go to the processor, and the speed drops to unacceptable. Apple Silicon shares memory, so the laptop runs larger models than expected. Team ollama ps shows what share of the model was spent on the GPU - this is where any disassembly of “why is it so slow” begins.

Useful scripts

• Answers on internal documents: embeddings and generation are entirely on your own machine, not a single paragraph of the contract goes outside.
• Mass marking and classification of the archive, where what is important is not the genius of the answer, but the absence of a count for a million requests.
• Rough autocompletion and explanation of code in the editor via a local endpoint, without network delays.
• Demonstrations and field tasks where there is no Internet or it is paid and slow.
• Prototype pipeline before choosing a cloud model: the logic is debugged locally, switched externally by changing the base URL.

Limitations

• On complex reasoning, long contexts and rare languages, local models are inferior to top cloud models. “Almost the same” is obtained with simple and well-defined problems.
• Speed ​​is limited by memory bandwidth. One user works comfortably, a dozen simultaneous ones line up.
• The default context is truncated. Without explicit num_ctx model loses the beginning of the document and looks worse than it is.
• Quantization below four bits noticeably spoils accuracy in numbers, formats and following instructions.
• There is no authentication in the API at all. A port exposed to a public interface makes the model available to everyone at your electricity expense. Outside - only through a reverse proxy with authorization or through a private network.
• Model licenses vary, and commercial use is verified by model card, not by successful download.
• Not all models support calling tools and strict structured output, and the quality varies greatly there.
• Updating the tag changes the behavior. For production, the tag is fixed, and the update goes through a run of its own set of tasks.

How to check the result

• ollama ps during the query: you can see which part of the model is calculated on the GPU. If there is a processor, further speed measurements are pointless.
• ollama run <модель> --verbose: Timings and speed in tokens per second are displayed at the end of the response. This is the first number for comparing models and quantizations with each other.
• Checking the context by hand: put a tag at the beginning of a long text (“code word is granite”) and at the end ask what the code word was. I didn’t remember - we ran into num_ctx.
• Direct curl request to /api/chat past CLI: This checks what the application sees, including the chat template and system message.
• Your set of 20–30 real problems with expected answers, run through two or three candidates using the same prompt. Public ratings say nothing about a specific task.
• Observing memory and swap during the run: going into swap is visible as seconds per token and is treated with a smaller model or smaller context.
• ss -tlnp | grep 11434 - make sure the service is listening on 127.0.0.1 and not 0.0.0.0.

Rule: a local model is considered suitable only after running its own set of problems with an explicitly specified num_ctx and confirmed loading on the GPU.