A transcript is not evidence that anything was recorded

I was building a meeting recorder: capture the microphone and the system audio, mix them, hand the result to whisper.cpp. The first test produced a file, a transcript, and a plausible-looking result:

 *sings along to the song*

I read that as good news. Audio had clearly reached the model, it had heard music from the browser, the route worked. Then I measured the actual waveform:

 sec  peak   rms
   0-15   50–1500   16–153     ← noise floor
  16     22482      3828       ← one transient
  17-23  150–1050   33–276     ← noise floor

Twenty-four seconds of near-silence with a single click in the middle. No music, no speech, nothing. The transcript was invented — whisper produces fluent, confident text from noise, and *sings along to the song* is exactly the shape of its hallucination on non-speech input.

The failure mode is nastier than a wrong answer, because the output was consistent with the story I already believed. I had predicted “system audio works, mic maybe not”, and the model handed me a sentence that fit.

What replaced it

The recorder now measures every leg before it trusts anything:

• each captured track is judged separately — peak, RMS, and the share of samples pinned at full scale
• the verdicts (SILENT / FAINT / SIGNAL / CLIPPING) go into the recording’s metadata, not just the terminal
• a mix that measures silent refuses to transcribe at all rather than emitting fiction

Two of those thresholds exist because the obvious version was not enough. A mix of one live track and one dead track measures as healthy — silence plus signal is signal — so the mix-level check cannot see a dead microphone, and each leg needs its own verdict. And “loud” is not “good”: a microphone driven into the rail reads at maximum peak with 72% of samples clipped, and transcribes as (R2D2's "I'm a Hero" plays). A peak-and-average test called that a strong signal.

The general shape

A generative model’s output is not an observation. It is a plausible reconstruction, and its plausibility is uncorrelated with whether the input existed.

Anywhere a model sits at the end of a pipeline, the thing to instrument is the input — levels, byte counts, checksums, anything measured rather than described. Otherwise the first component that fails silently is laundered into a confident sentence by the last component, and the pipeline reports success.

Related: Whisper picks the wrong GPU · Reading wrap