What does this solve?
The client’s message “nothing opens for you” as the first source of information about the accident is expensive: by the time of the letter, the breakdown has been alive for several hours, reputational damage has been caused, and you have to figure it out from memory. Monitoring rearranges the queue: the system recognizes first, the person on duty second, and ideally the client does not recognize at all. Three different cases are closed: a silent crash at night, degradation without a crash (the service responds, but slowly or partially) and breakdowns at the edges - an expired certificate, a clogged disk, a cron that silently did not work, a failed integration.
How it works
1. External sample
HTTP request to a service from a point outside its infrastructure, with an interval of about a minute. Not only the response code is checked, but also the presence of the expected line on the page: the hoster’s stub returns an honest 200, but there is no website. An external point is required - a probe living on the same server will die along with it and will not send anything.
2. Health-check
A separate route in the application that checks dependencies and returns a short JSON: the database is responding, the queue is alive, storage is available, migrations have been applied, there is enough free space. It is pulled by both the external test and the balancer. Checks inside are kept light: health-check, which itself creates a load on the base, finishes off the service at the worst moment.
3. Metrics
Time-based numbers: requests per second, percentage of erroneous responses, response time by percentile, processor, memory and disk load, task queue length. The average response time hides the problem - p95 and p99 are considered, because it is the tails that suffer. Metrics are added to a time series database, drawn as graphs, and serve as the basis for thresholds.
4. Bug tracker
An exception from the code goes into a tracker (Sentry, GlitchTip and the like) with a call stack, release version, user ID and “breadcrumbs” - a chain of last actions before the crash. Identical errors are grouped, the group has an event counter and the number of people affected. A regression is caught separately: an error marked closed appeared again after the rollout.
5. Logs
Last authority for analysis. They are useful when they are written structurally (JSON, not free text) and contain a request identifier, which is used to collect the entire chain from input to response. Logs answer the question “what exactly happened with this particular order”, not “is everything okay?”
6. Heartbeat
Reverse test for what should happen according to schedule: night unloading, backup, mailing, recalculation. The task itself pulls the selected URL after successful completion. If there is no signal within the specified window, there is an alarm. A regular test of server availability does not tell anything about a failed cron: the server is alive, but there is no unloading for the second week.
7. Business metric
Orders per hour, payments per hour, registrations per day. It catches breakdowns that are technically invisible: all answers are 200, there are no errors in the tracker, and the payment button does not work due to layout changes or the payment gateway silently rejects everything. The threshold is set taking into account the daily and weekly profile, otherwise Sunday night will ring every week.
Alarm routing
The threshold is formulated along with the duration: not “an error occurred”, but “the proportion of 5xx responses above the threshold for five minutes in a row.” Identical triggers are glued together so that two hundred letters turn into one in a minute. During the rollout, a window of silence is installed. Channels are divided by importance: what they wake up for at night goes to the messenger or call, the rest goes to the morning digest. For each rule, a short instruction is written: what it means, where to look, what to do first, who to call if it doesn’t help.
Useful scripts
- External probe of the main and login pages once a minute, checking a specific line in the response.
- Heartbeat for nightly uploading and backup: silence is as alarming as an error.
- Threshold for the share of responses 5xx and separately for p95 response time: degradation comes before the fall.
- The validity period of the TLS certificate and paid domain with a warning in advance.
- Free disk space according to trend - warning a week before it is full, and not at the moment of recording failure.
- Zero payment for two hours of working time as a business canary on top of the entire technical chain.
- The length of the task queue and the age of the oldest task in it.
- New error type after release and return of previously closed error.
Limitations
Monitoring within the same infrastructure falls along with it. At least one probe must live outside and be able to reach a person regardless of the main server.
Noise degrades the entire system. When there are more alarms than the person on duty can handle, they stop reading them in their entirety, and the real accident drowns among the usual ones. The sign of health is simple: all operations for a month can be listed and for each one you can say what was done.
It costs money and time. External services count by the number of checks and the volume of events, their stack - server, disk for rows and logs, updates and duty. Log storage is growing faster than it seems at the start.
Monitoring catches a breakdown in the product, but does not prevent it from getting there - tests and reviews do this job. One cannot be replaced by the other.
It is impossible to cover everything. There will remain scenarios that only a living user can see on his device with his own set of browser extensions.
Rollouts give false bursts. Without a window of silence, the team quickly gets used to brushing off worries during the release, and along with them, missing the real ones.
An alarm that does not have an addressee obligated to respond works like a journal entry - someone will get around to it someday.
Personal data leaks to error trackers through request parameters, form bodies and log breadcrumbs. The filter on the sending side is configured before the first combat event, and not after.
How to check the result
Training breakdown. Stop the service at the stand or return 500 on a specially established route and use a stopwatch to measure the time from the breakdown to the message on the phone. This is the only honest metric of the entire system; everything else is intentions.
Heartbeat check: turn off the crown on the stand and wait for the alarm in the designated window. Silence means that the schedule and the threshold have diverged.
Delivery path check: make sure that the night message arrives where the person actually looks at night. A mailbox that opens at ten in the morning is not suitable for a critical channel.
Revision for the month. Write down all the alarms that were triggered and note which actions required. If less than half of the actions required, clean the thresholds rather than adding new rules.
Checking instructions. Give a random alarm runbook to someone who didn't write it, and see if they follow the steps to the end without question.
Checking graphs with reality. Take a famous incident from last week and find it on the metrics charts. If the numbers are indistinguishable from a normal day, then the wrong thing is being measured.
Monitoring of the monitoring itself. The external service has a notification about its own downtime; its own installation has a second independent point that checks that the first one is alive and continues to send data.