Local AI agent: running the model on your own hardware

What does this solve?

Local AI-agent is needed when data cannot be sent outside, but the work must still be done by the model: medical statements, correspondence with clients, internal documents, trade secrets. The second motive is independence: no provider limits, no change in model behavior after an update on someone else’s side, no broken Internet. The third is expense: hardware is purchased once, the counter doesn’t turn any further, and a massive routine like marking up tens of thousands of records ceases to come up against the budget. Instead, you have to decide on your own the issues that are resolved for you in the cloud: how much memory, what quantization, why the model stopped returning tool calls.

How it works

Three-layer design: inference engine → weight file → agent binding.

Engine

llama.cpp — basic engine and server llama-server, works on CPU and GPU, eats GGUF format.
Ollama —a wrapper over it with a model catalog and download management, the shortest path to the first launch.
LM Studio —the same with a graphical interface, convenient for trying on models.
vLLM —server for GPU and parallel queries, takes weights in safetensors format, pays off when there is more than one user.
MLX —path for Apple Silicon using shared memory.

Weights and quantization

The model on disk is a matrix of numbers. In its original form, each parameter occupies two bytes; when quantized, it is compressed to eight, six, five or four bits with loss of accuracy. The estimation mechanics are simple: weight memory ≈ number of parameters × bits per parameter / 8. To this is added the KV cache - a storage of the attention state for all processed tokens. The cache grows with the length of the context, the number of layers and the dimension of attention, and in long dialogues it can take up more than the weights themselves. Plus engine overhead.

Key threshold: as long as it all fits into video memory, the speed is determined by the GPU. As soon as some of the layers go into system memory, generation slows down sharply, because each token drags data across the bus. It is this threshold, and not the name of the model, that determines whether the agent will be suitable for work.

Speed and why an agent is more expensive than a chat

Generation is divided into two phases: prefill (processing the entire input) and decode (issuing tokens one at a time). At each step of the cycle, the agent resends the entire accumulated history, so it is the prefill that becomes the bottleneck: the fifth step of the cycle processes the input that has grown on all previous results of the tools. Locally this is felt more strongly than in the cloud, where the prefix is cached on the provider's side.

Tie

Almost all engines provide an OpenAI-compatible endpoint /v1/chat/completions. Any agent code is connected by shift base_url and a dummy key - no need to rewrite the cycle.

Separate check - calling tools. Support tools depends on the model and its chat template: some models return the call as text inside the response instead of a structured one tool_calls, some of it gets confused when using several tools at once. This is checked on their circuits before any integration.

Rule: first measure whether the model, along with the working context, fits into video memory, and only then discuss its quality.

How to connect

Step 1. View the hardware. On NVIDIA - nvidia-smi, the Memory-Usage column will show the total volume and occupied space. On Mac - system_profiler SPDisplaysDataType and the total amount of RAM that is shared with graphics. At the same time, check the free disk space: the scale files are large.

Step 2. Install the engine.

brew install ollama        # или установщик с сайта
ollama serve               # адрес и порт печатаются в выводе

Step 3. Download the model. In the catalog, each tag has a specified size and quantization - it’s wise to start with an option that is obviously less than the available memory.

ollama pull <модель>:<тег>
ollama list

Step 4. Check endpoint with the same request that the agent will use. Take the address from the output ollama serve, not from memory.

curl http://127.0.0.1:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"<модель>","messages":[{"role":"user","content":"скажи ok"}]}'

Step 5. Make sure that the model is in the GPU. ollama ps shows how much of it went to the processor. Any share on the CPU is a signal to take a more compressed version or reduce the context.

Step 6. Connect an agent. Set in the client base_url to a local address and an arbitrary key. The rest of the loop code remains unchanged.

Step 7. Check tools. Send request with array tools and look at the answer: did you come tool_calls with a list of arguments, whether the JSON is valid, whether the call was retold in prose.

Step 8. Set up the context. In Ollama - parameter num_ctx, in llama-server - --ctx-size. The value is selected for the longest real request, and not with a reserve: each extra window token takes up memory for the cache, even when idle.

Step 9. Autorun. On Mac - launchd-agent in ~/Library/LaunchAgents, on Linux - unit in systemd. Check that after a reboot the server starts up on its own.

Step 10. Close access. Server listening on 127.0.0.1, outside - only through a tunnel with authorization. An endpoint open to the network without a key will give calculations to anyone.

Useful scripts

Analysis of documents with personal data without leaving the company perimeter.
Mass classification and marking, where volume is important, not subtlety of reasoning.
Offline assistant on a laptop: work on an airplane, in the field, behind closed circuits.
Local knowledge base search: index and answer generation on one machine.
Pre-filter before the cloud model: filter out the noise locally, send the remainder up.
Rough processing of code and texts, where the cost of an error is low and the frequency of calls is high.

Limitations

Quality on long chains of reasoning and on multi-step calls to tools is inferior to the strongest cloud models. An agent that takes ten steps at the cloud locally begins to lose track and repeat calls.

Long context is paid by memory. A model that lives comfortably on short queries no longer fits as soon as the window is expanded to accommodate a large document.

Parallel users require a different engine: Ollama and llama.cpp are designed for one or two, with several simultaneous requests the queue grows. vLLM is used for load.

Maintenance falls on you: engine update, chat template compatibility, new weight formats, behavioral regression after changing version. Automatic improvements do not happen “by themselves.”

Physics also sends the bill: noise, heating, electricity, downtime of expensive hardware between tasks.

Engineer time is the largest item at the start. If the task is one-time, renting a GPU by the hour will complete it faster than purchasing it.

How to check the result

ollama ps or log llama-server —is the entire model in the GPU. This is the first check, the rest are meaningless without it.
Speed at in its typical request, not “hello”: the server prints prefill time and tokens per second in the log. Measure separately the short input and the input equal to the worst operating case.
Golden set of several dozen of your own tasks: run locally and on a cloud model with the same prompts, compare the proportion of correct outcomes. The decision is made based on this table.
Share of valid tool_calls from the total number of agent steps. Low - change the model or switch to a strict answer format with a diagram.
Memory behavior: look at the nvidia-smi or memory monitor, catch the moment of displacement into the swap - there the speed drops without any error messages.
Isolation: disconnect the network and run the entire set. If everything worked, and lsof -i did not show outgoing connections, the circuit is indeed closed.