What does this solve?
There is a repeating operation: every day someone opens three systems, compares data, writes a letter and starts a task. There is no way to close it with a script - the entrance is different every time, and the solution depends on what was found along the way. The question “how to create an AI agent” appears precisely here: what is needed is not a demonstration in the chat, but a thing that performs the entire operation and does not spoil the data at the first surprise. Below is the route from the description of the operation to the version that can be left to work unattended.
How it works
The assembly is divided into six parts, and each lives in a separate file, and not in the author’s head.
Task contract. Input, output, readiness criterion, prohibited actions, behavior in case of data shortage. While the readiness criterion cannot be checked mechanically, it is too early to collect an agent.
Tools. Functions by which the agent reaches systems. Each with a description for the model and an argument diagram for the code.
Prompt. Role, procedure, rules for choosing tools, response format, prohibitions.
Hinge with budget. Iteration limit, timeout per instrument, token ceiling. The budget is exhausted - a stop with a clear message, not a silent loop.
Run log. Full trace of each run: step, tool, arguments, response, tokens.
Set of checks. A file with tasks and expected results, which is run after each prompt edit.
Project structure that lasts for a long time:
agent/
contract.md вход, выход, критерий готовности
prompt.md системный промпт
tools/ по файлу на инструмент
loop.py петля, лимиты, проверка прав
evals/cases.jsonl задачи с известным ответом
runs/ трассы прогоновThe assembly order goes from bottom to top: first tools and rights, then prompt, then autonomy. The prompt, written before the tools, describes what is desired, not what is available.
How to connect
Step 1. Record the operation by hand. Go through it yourself and record every action: what screen, what field, where the value comes from, what decision is made and on what grounds. Points like “here I’ll see if it’s normal” should be expanded to the condition being tested - otherwise this condition will have to be invented.
Step 2. Write a contract. One page: what is supplied as input, what is considered the result, how to check readiness mechanically, what is forbidden to touch, what to do if there is insufficient data. The page is not written - the task is not ready for the agent, and no model will fix this.
Step 3. Cut the operation into instruments. Each action from the first step is a candidate. One tool does one thing and returns data in a readable form. The “do everything at once” function is debugged worse than three separate ones, and it’s not clear from the route where it broke.
The description for the model is written like an instruction to a colleague:
name: find_orders_without_payment
описание: возвращает заказы за период, у которых нет платежа.
Период задаётся датами включительно. Удалённые заказы не возвращает.
При пустом результате возвращает пустой список, а не ошибку.
аргументы: date_from (YYYY-MM-DD), date_to (YYYY-MM-DD), limit (int)Step 4. Issue rights separately. Technical user for an agent, a role in the database only for reading the necessary tables, its own token for each external service, its own mailbox for sending. On the first assembly, write permission is not granted at all: first let him learn to read and respond.
Step 5. Write a prompt. Blocks: role and area of responsibility, procedure, rules for choosing tools, format of the final answer, prohibitions, behavior in case of lack of data. General calls like “be careful” do not change anything. They change specific phrases: “there are no dates - ask for the period”, “there are more lines than the limit - tell me and don’t think about the rest”, “the tool returned an error twice - stop and describe what you tried.”
Step 6. Assemble the fuse loop.
messages = [system, task]
for step in range(MAX_STEPS):
reply = model.create(messages=messages, tools=TOOLS)
log(step, reply)
if not reply.tool_calls:
return reply.text
for call in reply.tool_calls:
if not allowed(call.name, call.arguments):
messages.append(tool_result(call.id, "отказано: нет доступа"))
continue
try:
out = run_with_timeout(TOOLS[call.name], call.arguments)
except Exception as e:
out = f"ошибка инструмента: {e}"
messages.append(tool_result(call.id, out))The error is returned to the model as a text - the agent uses it to make the call himself. The process is stopped only by an exhausted budget.
Step 7. Collect a set of checks. File evals/cases.jsonl, one line per task:
{"task": "заказы без оплаты за прошлую неделю", "expect_tools": ["find_orders_without_payment"]}
{"task": "выручка за позапрошлый век", "expect": "отказ: данных за период нет"}Half of the tasks are ordinary, half are borderline: empty result, closed access, question beyond competence, contradictory input. Borderline ones are more important than normal ones: they show whether the agent is lying or refusing.
Step 8. Run in prompt mode. The agent prepares a letter, ticket or change - a person sends it. Each manual edit goes to one of three places: prompt, tool description, set of checks. After a few days, the edits stop bringing new things - this is a signal for the next step.
Step 9. Give autonomy one action at a time. Invertible first: draft, comment, mark, entry in the service table. Irreversible ones - a letter to the client, payment, deletion - stay behind confirmation longer than others, and some remain there forever.
Rule: the right to write is granted one action at a time and only after the person’s edits in the tooltip mode no longer bring anything new.
Useful scripts
- Daily summary of several systems, clearly indicating differences between them.
- Primary analysis of the request: classification, collection of client context, draft response.
- Preparing documents: collecting details and positions, the final button is up to the person.
- Error duty: sorting new tracker entries, searching for similar ones, short summary in chat.
- Filling out product cards according to the supplier’s description with checking required fields.
- Checking data before uploading: empty fields, duplicates, amounts that do not add up.
Limitations
An operation without a digital trace is not collected. A call, a paper act, an agreement in the smoking room - first a digital input, then an agent.
The agent does not speed up people's waiting. If an operation costs three days to coordinate, automating three minutes of work will not give anything.
Prompt does not replace data. When the required field is not in the system, no amount of wording will force the agent to get it. He will fill the field with a guess, and the guess will look confident.
Tools break following other people's APIs. Changing the supplier's schema drops the agent in the same way as any integration. Maintaining a tool set is an ongoing job, not a one-time job.
The first version does not break where expected. Usually the reason is not the model, but the description of the tool or the format it returns. Without a log it is not found.
Rare operations do not pay off. A one-off task once a quarter is done cheaper by hand.
How to check the result
Run the entire set of checks after each prompt edit. Prompt behaves like code: editing one line changes the behavior of neighboring tasks. The run is done before the rollout, and not after the complaint.
Count the proportion of interventions. How many results were left without corrections, how many required revisions, how many were completely redone. The share of edits is growing - look for the reason in the changed data, and not in the model.
Read routes in a row for a day, rather than selectively. Sampling shows successful cases; a continuous view over a short window shows those where the agent confidently went the wrong way.
Check failures separately. Take away access to the tool and look at the answer: a refusal message is the norm, numbers out of thin air are a reason to stop implementation.
Compare time and cost. Tokens and minutes from the journal against a person’s time for the same operation, on the same volume.
Hold the switch. One environment variable or flag that stops the agent without rolling out. Check that it really stops - on a live run.