How deployment works: build, release, rollback

What does this solve?

Two typical layout failures look like this: “they laid it out - it was easy” and “it was easy, but it was impossible to return it as it was.” The first is treated with checks and gradual rollout, the second - with the design of the process itself. When build, release, and launch are separated, the failed version is removed in a minute without rebuilding, and yesterday's state is reproduced by tag, not by memory.

The third problem is quieter and more expensive: no one can answer what exactly is working in production right now. The staged process answers this question as a team, rather than by asking colleagues.

How it works

Three stages

Assembly. Sources plus dependencies turn into an artifact: docker image, directory dist/, archive, package. The artifact does not depend on the environment and does not contain any database addresses or keys. It has an identifier - usually a short hash of the commit.

Release. The artifact is connected to the configuration of a specific environment: variables, secrets, addresses. The result is a launched unit with its own number. Changing one variable without rebuilding is also a new release, and it also needs to be numbered, otherwise history is lying.

Start. Release starts: application processes, queue workers, scheduled tasks.

The promotion rule is simple: the artifact is assembled once and goes through the stand, pre-sale and prod in unchanged form. Rebuilding before production means that something that no one has tested is going to production - the code is the same, but the build result may differ due to updated dependency versions.

Version identification

The artifact tag is a short hash of the commit. The application sends its version externally:

GET /version
{ "sha": "9f3c1a2", "built_at": "2026-09-08T11:20:14Z", "env": "prod" }

This endpoint closes half of the disputes when analyzing an incident and allows you to automatically verify that what has rolled out is exactly what was merged.

Release catalog and atomic switching

Classic scheme for applications laid out by files:

/srv/app/
  releases/
    2026-09-08-1120-9f3c1a2/
    2026-09-07-1804-7ab19cd/
  shared/
    storage/
    .env
  current -> releases/2026-09-08-1120-9f3c1a2

The new release is deployed next to the working one, warmed up (dependencies, caches, asset assembly), and only then a symbolic link current is rearranged in one motion, after which the process pool is restarted. The user does not see the moment when half of the files are new and half are old. Rollback - move the link back and restart.

For containers, the same logic by other means: raise the container from a new tag, switch traffic to it, leave the old image on the host and in the registry.

Rollout strategies

• Recreation. Stop, start new. Downtime for start time. Suitable for internal services and everything where a minute of unavailability costs nothing.
• Gradual replacement. Instances are updated in batches. There is no downtime, but for some time two versions work at once - the API and the database scheme must withstand this.
• Blue-green. A complete second set rises nearby, traffic is switched at once, the old set remains hot in case of return. Paid by doubling resources.
• Canary. The new version receives a small share of traffic, the metrics are compared with the main one, the share grows if there are no deviations. Requires both traffic and configured version metrics.

Migrations - what locks the rollback

The code is rolled back, the database is not. If a release contains a migration that removes a column, the previous version of the code after the rollback will encounter the missing field. A scheme that preserves the possibility of reverting spreads the change over several releases:

1. Migration only adds: the new column allows NULL, the new table is created empty. Rolled out separately and before the code.
2. The code learns to read both versions and write to the new one. Roll out.
3. Data is transferred by a background script in portions, with load control.
4. The code stops reading the old field. Roll out.
5. In a separate release, after a while, the old column is deleted.

Not a single step breaks the previous version of the application, so a rollback is possible on each one. Destructive operations (DROP, RENAME, addition NOT NULL without a default value) do not travel in the same release as the code that needs them.

The migration execution time based on sales volume is checked separately. A change that occurs instantly on a test database of a thousand rows can lock the table for minutes on millions of rows.

Configuration

The config lives outside the artifact. Secrets are in the secrets store or in environment variables on the machine, not in the repository or in image layers. Any change to the config goes through the same path as the code: editing, checking, release with a number. A value corrected manually on the server will disappear during the next rollout, and the search for the cause will take a long time.

Post-release checks

Smoke set is a short script that pulls several key addresses and checks the response code and the presence of the expected line:

set -e
curl -fsS https://app.example.com/version | grep -q "$TAG"
curl -fsS https://app.example.com/ | grep -q "Вход"
curl -fsS -o /dev/null -w '%{http_code}' https://app.example.com/api/health | grep -q 200

The health endpoint does not check whether the process is alive, but the dependencies: whether the database is accessible, whether the cache is responding, whether the queue is being cleared. A test that always returns 200 is useless.

A release mark is placed on the error and response time graphs. A spike immediately after the vertical line does not require investigation of its origin.

Cache, clients and queues

Assets are named with a hash of the contents, then the old and new file coexist and the cache does not need to be reset. The HTML entry point is cached briefly. For CDN, the reset is done with an explicit command at the end of the rollout.

Open tabs continue to run on old JavaScript and knock on old endpoints. The API is required to maintain compatibility at least one release back, otherwise the layout breaks those who are already on the site.

Queue workers must be restarted after rollout: otherwise the old code will parse messages of the new format. The opposite is checked separately - the new code must survive the messages queued before release.

Rollback

Rollback works under four conditions: the previous artifact is intact and accessible, the configuration is compatible with it, the database schema is compatible, the rollback command is known and has already been executed. A return that requires rebuilding from source has nothing to do with a rollback - it is a new release with an untested artifact in the middle of a disaster.

Feature flags provide a faster tool: the code is rolled out disabled, turned on by a flag, if there is a problem, the flag is extinguished in seconds and the artifact is not touched at all. The release and activation of a function takes time, and most unsuccessful changes are closed without a rollback.

Useful scripts

• Night rollout of a major flag change: the code was on production in the evening, turned on in the morning when the developer was on duty.
• Splitting the migration and code into two releases before changing the key field format.
• Blue-green rollout before the seasonal peak, when downtime costs revenue.
• Canary rollout of an edit that affects conversion: share of traffic and comparison of metrics instead of a dispute about tastes.
• Automatic reconciliation /version after deployment to catch a rollout to the wrong environment.
• Storing three to five past artifacts and releases on the machine as insurance in case of an inaccessible registry.

Limitations

No-downtime deployment costs either double the resources or compatibility requirements for every schema and API change. On a small service, an honest downtime per minute is cheaper than supporting a blue-green scheme.

Rollback has no effect. Letters have been sent, payments have been processed, webhooks have been delivered to partners, files have been stored in storage. Returning code returns behavior, not data, and everything that managed to leak outside will have to be parsed manually.

The blue-green schema on a common base does not isolate the data schema: both versions work with the same table, and incompatible migration puts both sets at once.

Canary rollout requires traffic. On a service with a dozen requests per hour, a share of five percent will not gain statistics until a week has passed.

The process requires discipline and it should be reinforced by rules: migrations are a separate release, flags should be cleared after stabilization, the artifact should not be rebuilt. Flags that no one removes turn into a forest of branches within a year, where not a single combination is tested.

The pipeline itself is a code that breaks: the runners have been updated, the registry token has gone bad, the cache space has run out. Support takes time, and it needs to be invested.

For a small site, a full-fledged conveyor does not pay for itself. Uploading files with a backup of the previous version closes the task while releases are rare, and a rollback is unpacking the archive.

How to check the result

Check version: curl -s https://app.example.com/version should return the hash of the commit that was merged. A discrepancy means that the rollout did not arrive or went to another environment.

Run a rollback on the stand and note the time. This is done before it is needed in production: a command executed for the first time under pressure is executed incorrectly.

Make sure the previous artifact is in place: docker image ls on the host, list of tags in the registry, contents releases/. A registry with a cleanup policy may remove the exact tag it was intended to return to.

Check backward compatibility: run the previous version of the code against an already migrated database. If it falls, the rollback is blocked, and it is better to know about this in advance.

Run the migration on copies of the product database and measure the execution time and duration of blocking. The copy should be fresh in volume, not in structure.

Check queues: the start time of worker processes must be later than the release time. Live processes with old code are a common cause of errors that “appear on their own after half an hour.”

View a graph of errors and response times in the window after the release mark, and not after complaints. The observation lasts as long as the main scenarios need to scroll at least once.

One line rule: the release is considered complete only after the rollback has been checked - the previous artifact is available, the database is compatible with the old code, and the rollback command has already been executed on the bench.