Whisper and speech recognition in the work loop

What does this solve?

An hour of recording a call is worth an hour of someone else's attention, and therefore no one listens to the call archive. Whisper translates recordings of meetings, calls and voice messages into text with time codes, including Russian, without manual transcription and without sending every minute to the freelance exchange. The text is then searched, cut into quotes, and turned into a summary and tasks. A special benefit accrues to the archive: records that have been lying around for years and have not been read become a search index overnight.

How it works

Model

Audio is reduced to 16 kHz and one channel, cut into windows of a fixed length of thirty seconds, each window turns into a log-mel spectrogram - a picture of frequencies and time. The encoder compresses it into a sequence of internal representations, the decoder autoregressively produces text tokens, word by word.

In addition to words, the decoder predicts service tokens: what language is spoken, what problem is being solved (decoding in the original language or translation), whether time stamps are needed. Two properties grow from this construction. The language is detected automatically based on the first seconds, and is detected incorrectly on the music screensaver at the beginning of the recording. The model can only translate into English; it does not do the opposite direction.

The auto-regressive decoder also explains the main defect: when there is nothing meaningful in the input window, the model still continues to generate text. This is how polite phrases and fragments of credits appear that were not in the recording.

Dimensions and implementations

The line goes from tiny to large: the larger, the more accurate and slower, and the more memory is needed. There are several implementations, and the choice between them affects the cost of operation more than the choice of size:

• original implementation on PyTorch - the standard of behavior, the most voracious;
• faster-whisper on the CTranslate2 engine - noticeably more memory efficient and faster with the same weights;
• whisper.cpp - runs on a processor and Apple Silicon, convenient for a laptop and for a server without a video card;
• cloud endpoint - there is nothing to expand, but the file goes outside, the request has a size limit, and long records still have to be cut.

Working circuit

The steps that go into processing are the same in almost any product:

1. File reception and normalization: ffmpeg -i call.m4a -ac 1 -ar 16000 -c:a pcm_s16le call.wav. One channel, 16 kHz, no exotic codecs.
2. Voice Detector (VAD) cuts out silence and divides the recording into replicas. This step removes most of the fictional text before the model.
3. Job queue. Recognition lasts minutes; a synchronous HTTP request does not last that long.
4. Recognition by chunks with overlap and assembly of transcript with timecodes.
5. Post-processing: replacing terms according to the dictionary, bringing names and articles to the accepted spelling.
6. Saving to the database and search index, link to the position in the audio.
7. Language model over text: summary, solutions, tasks, topics of appeal.

What is being tightened

• Hint at the beginning (initial_prompt): a list of terms, product names, surnames that will appear. The cheapest way to improve the quality of domain-specific vocabulary.
• Explicit language specification when the language is known in advance. Eliminates the risk of erroneous auto-detection.
• Temperature rollback: if there are bad signs of a segment (low confidence, suspicious text compressibility), decoding is repeated with other parameters.
• Word-by-word timestamps and forced alignment with a separate tool - needed for subtitles and jumping to the exact second in the recording.
• Diarization, that is, marking “who is speaking”, is done as a separate model. Whisper can't do this at all.

Output formats

Plain text is suitable for searching and for passing into the language model. Subtitle formats (srt, vtt) are needed by the video editor and player. Detailed json is what the entire automatic quality check is based on: it contains segments with a beginning, an end, text and service characteristics - the average confidence of the model for the segment, an estimate of the probability that there is no speech in it at all, and the text compressibility coefficient. The last three fields are used as a filter: a segment with a high probability of silence and low confidence is discarded before the made-up phrase gets into the report to the client.

It makes sense to store everything at once: full json as a source of truth, text in the search index, generate subtitles on demand. Re-recognizing an archive costs computer time, and doing it because of a format change is annoying.

Useful scripts

• Transcription of calls with clients and automatic summary with solutions and tasks, so that the manager does not write a report by hand.
• Search through the archive of negotiations: text in the index, the result is a link to the timecode in the recording.
• Subtitles for videos in srt and vtt formats, where only the terminology is edited by hand.
• Voice notes on the go that turn into structured tasks in your tracker.
• Selective support control: topics, promises and complaints are searched by text instead of continuous listening.

Limitations

• In silence, noise and music, the model adds non-existent phrases. The typical look is repeated polite formulas and fragments of credits. Treated with a speech detector and rejecting segments based on confidence.
• Overlapping speech, several participants and interruptions without diarization produce a continuous text without indicating the speakers, from which it is not clear who promised what.
• Rare surnames, titles, articles and abbreviations are recognized with systematic errors. Dictionary hints and post-processing of substitutions are required.
• Time stamps on segments are approximate; word-by-word precision requires a separate alignment step.
• There is no real time out of the box: the model is batch, stream decoding is assembled using a speech detector and short windows, with a delay and with errors at the junctions.
• Large model requires video memory. On a processor, it runs slower than real time, and the queue must be scheduled based on actual measurement, and not on hope.
• Conversation recording - personal data. Consent to recording, retention period, and access to transcripts are regulated as strictly as access to the audio itself.
• Mixed speech in two languages ​​in one phrase is recognized worse than each language separately.

How to check the result

• Take 10-20 minutes of your typical recordings, transcribe the standard manually and calculate WER: the sum of substitutions, insertions and omissions divided by the number of words of the standard. This is the only number by which models and settings are compared with each other.
• Separately run a fragment of silence and a fragment with music. No sentences should appear in the output.
• Search the transcript for the same line repeated several times in a row: a sign of decoder cycling.
• Check the latest timecode with the file duration. A discrepancy means a missing tail of a record.
• Check the junctions of chunks: the word at the boundary should not be missing or duplicated.
• Measure RTF - processing time divided by audio duration - on the target hardware. From this number the queue capacity and the required number of workers are calculated.
• Run a list of domain terms before and after the hint and count how many were recognized correctly in each case.

Rule: recognition quality is measured by WER on your own recordings and behavior in silence; The size of the model and the promises of the card are not relevant.