Speech-to-Text

Speech-to-text transcribes spoken audio into written words.

Audio arrives as a waveform, thousands of amplitude samples per second.

That is first converted into a spectrogram, which shows frequency over time.

A spectrogram is effectively an image of sound, and models handle images well.

Classical systems needed separate acoustic, pronunciation and language models.

Modern systems are end to end, mapping audio straight to text.

Whisper is the well-known example, an encoder-decoder transformer trained on a vast weakly labelled set.

The encoder reads the spectrogram and the decoder writes text one token at a time.

Being a language model, the decoder uses context to resolve ambiguous sounds.

That handles accents, noise and many languages far better than older systems.

It also means it can hallucinate text during silence, which is a real failure mode.

Word error rate is the standard metric, counting substitutions, insertions and deletions.

Under five per cent is good for clean speech, and noisy multi-speaker audio is much worse.

Speaker separation is a distinct problem, since transcription alone does not say who spoke.

Streaming transcription trades accuracy for latency, since it cannot wait for the end of a sentence.