What does this solve?
The first request to the model is assembled in two minutes by copy-pasting someone else’s curl, and this is enough before production. Then it begins: the answer breaks off in the middle of a sentence, 429 arrives once an hour, the monthly bill is twice as high as the estimate, and the model “does not hold the format.” Each of these four problems is caused by a specific request body field or a specific response field, which usually no one reads. Parsing the composition of the request is needed in order to pull the desired lever, and not to rewrite the prompt at random for the third time.
How it works
Vehicle and key
Outwardly, this is a regular HTTPS POST with JSON in the body. There are two required headings: Authorization: Bearer <ключ> i Content-Type: application/json. The key lives on the server. In a browser bundle or in a mobile application, it will be obtained in the evening, and the owner of the key will pay. The working scheme is the same: the client goes to his own backend, the backend goes to the provider, and he also calculates user limits and writes logs.
There are several endpoints - text generation (classic chat completions and a later format with a single input field), embeddings, audio, images, files, batch processing. The form is the same everywhere: JSON inside, JSON or event flow out.
Request body
| Field | What it controls | When to touch |
|---|---|---|
model | Which model considers the request Add | to the config: model names live their own lives, there is no need to re-roll the code because of them |
messages (or input in a new format) | the dialogue itself is an array of messages with roles | always |
tools, tool_choice | descriptions of the application functions that the model can call and the enforcement of the call | when the answer must be based on system data rather than model memory |
response_format / JSON schema | response form up to a strict scheme | always when the answer is parsing the code and not a person |
temperature, top_p | scatter when choosing the next token | down - data extraction, classification, calculations; above are draft texts |
max_tokens / max_output_tokens | response length ceiling | always set: this is also a cost-effective fuse |
stop | lines at which the generation is terminated | rigid formats, separators |
stream | give answer in parts | interfaces where a living person is waiting |
seed | attempt to repeat output | tests and debugging |
user, metadata | request tag for logs and analysis of abuses | multi-user products |
Roles and memory
Messages differ in roles. The system role (in new formats - the role of the developer) sets the framework: who is responsible, in what format, what not to do. The user has a task. Assistant - past responses of the model. The individual role returns the result of the tool call.
The server does not store the dialogue. Each request leaves entirely, with the entire history that the application has stitched together itself. Part of the API knows how to keep state on its side by reference to the previous response, but by default trimming old replicas, compressing them into summaries and ordering messages is the job of the application.
Tokens and money
Payment and limits are calculated in tokens, separately for entry and exit. Three things follow from this. The dialogue becomes more expensive with each remark, because the previous ones leave again. The ten-page regulations inserted into the system message are paid for each request, and not just once. And the constant part of the prompt is kept at the beginning and unchanged - this is how prefix caching works; everything that changes is sent to the tail.
What comes in response
In addition to the text in the response, there are two fields for which they read it. finish_reason tells why the generation ended: normal, hit the length limit, went into a tool call, got under the filter. usage shows the actual consumption of tokens for entry and exit. Trimmed JSON after finish_reason: length will not be parsed, and it is not the model that is to blame, but the lack of verification in the code.
Calling tools
Four-cycle cycle. First: the request leaves with a list tools —function name, description, JSON argument scheme. Second: the model does not return text, but a call - name, arguments and call identifier. Third: the application executes the function itself and adds a message to the history with the result and the same identifier. Fourth: the request is repeated with an extended history, and the model formulates an answer.
The model does not execute anything - it forms an intention, the decision to call the function is made by the code. Arguments come from the model and are checked like any external input: the model does not prevent you from sending a non-existent order ID or a negative quantity.
Streaming
At stream: true The response comes in a stream of server events: chunks with text deltas and a final marker. Treats “the interface is silent for ten seconds.” The fee is complicated error handling: the connection is already open, code 200 has already been sent, and a fall in the middle must be caught separately. The token counters in the stream come at the end and only if you ask for them with an explicit option.
Errors and repetitions
| Code | What happened | Reaction |
|---|---|---|
| 400 | request body or schema not valid | fix the code, repeating is pointless |
| 401, 403 | key, rights, region | repair access |
| 404 | model not available for project | check model name and access |
| 429 | request limit, token limit or balance has run out | pause and repeat with increasing delay |
| 5xx, timeout | provider side | repeat with increasing delay |
It only makes sense to repeat 429 and 5xx: exponential backoff with random addition, finite number of attempts, respect for the title Retry-After. Repeated POST has the risk of performing the operation twice - the API has an idempotency header for this. The request ID from the response headers is always written to the log: without it there is nothing to talk about in support.
Useful scripts
- Classification of incoming requests into a strict JSON schema with a confidence field and routing of controversial cases per person.
- Extracting fields from letters, invoices and invoices into a structure ready for recording in the database.
- Calling tools: catalog search, reading balances, draft order - the response is collected from system data, and not from the model’s memory.
- Streaming the response to the chat interface so that the first words appear immediately, and not after full generation.
- Batch processing of the archive: thousands of requests of the same type are sent as a file and are counted deferred, cheaper than individual calls.
Limitations
- No status. Remembering the dialogue is the application’s job, and it also pays for the history at every step.
- Context window is finite. Trimming, compression and priority of messages are written by hand; nothing is compressed automatically.
- There will be no complete repeatability even at zero temperature and a given seed. Tests are based on checking the properties of the response, and not on byte-by-byte comparisons.
- A strict scheme guarantees the form of the answer; The code checks the veracity of the values inside it.
- Limits on requests and tokens per minute make a naive loop
forfor thousands of documents that are inoperative - a queue with concurrency limitation is needed. - Data goes to someone else's circuit. For personal data and trade secrets, separate measures are needed: depersonalization, agreement, local model.
- The connection to the provider is deeper than it seems: the names of the models, the format of the tools, the behavior of the filters, and the counting of tokens are different. The layer is made thin, but it is done.
- The delay of someone else's service is not under control. A synchronous model call inside processing a web request will sooner or later run into a timeout.
How to check the result
- Send minimal request via curl and read in response
usageifinish_reason, not just text. - Specially expose tiny
max_tokensand make sure the code sees the break and doesn't try to parse incomplete JSON. - Count prompt tokens locally before sending and check with
usage—the discrepancy will show that excess is being included in the request. - Collect a golden set of 20–50 real examples with standard answers and run it after each prompt edit. Comparison “by eye” stops working on the third iteration.
- Check retrays on a stub that sends 429: the number of attempts is finite, the delay is increasing,
Retry-Afteris taken into account, the log does not turn into an endless loop. - Ensure that the request ID and token consumption are logged for each call, including unsuccessful ones.
Rule: everything that should be predictable is specified by the request fields and checked by the code; In words, only wishes are specified in the prompt.