Speech-to-text
Speech-to-text, also called automatic speech recognition or ASR, is software that converts spoken words in audio or video into written text. A neural model turns short slices of sound into likely words and picks the most probable sentence, often helped by a language model, and the system adds punctuation and a timestamp for each word. In a video pipeline, speech-to-text produces the raw transcript that captions, search, chapters, summaries and translations are all built from, so any error it makes travels into every one of them.
How does speech-to-text work?
First the audio track is pulled out of the video container and decoded, usually from AAC, then resampled to a single channel at a fixed rate the model expects. The model reads the signal as a spectrogram, a picture of which frequencies are loud at each moment, in overlapping frames about 25 milliseconds long, one every 10 milliseconds.
Modern systems are end to end: one neural network maps those frames straight to characters or word pieces, with a language model, either built in or added at decoding, choosing between words that sound alike. A second pass restores punctuation and capitals. Alignment then ties every word to the moment it was spoken, which is what makes the text usable as captions or as search results that jump to a second. Many systems also run speaker diarization, which labels who spoke when, so a two-person interview reads as a conversation rather than one block of text.
How accurate is speech-to-text?
Accuracy is measured with word error rate, or WER: the number of words substituted, deleted or inserted, divided by the number of words a careful human transcript contains. Lower is better. WER is only comparable when both transcripts are normalized the same way, for example in how numbers and punctuation are written, and a single figure hides a lot. One wrong drug name or one wrong digit in an account number matters more than ten missed filler words.
Audio decides most of the result. A close microphone, one speaker at a time and no music bed give clean text. Crosstalk, room echo, a phone on a desk, heavy compression and strong background noise all push errors up. Accents and specialist vocabulary, such as product names, legal terms or medical jargon, are the next biggest factor.
A reliable test is your own footage. Take a sample of real recordings, have a person correct them, and compare. Pay special attention to names, numbers and the terms your users will search for.
Speech-to-text vs. transcription vs. captions
These three words get used as if they meant the same thing, but they describe different layers. Speech-to-text is the engine: the recognition step that turns sound into words with timings. A transcription, or transcript, is the document that comes out, the full text of what was said, often with speaker labels, read on its own or searched.
Captions are a presentation format built from that transcript. The words are split into short lines that fit on screen, timed to appear and disappear with the speech, and saved as a file such as WebVTT or SRT that a player loads beside the video. Good captions also mark sounds that matter, such as music or laughter, which raw speech-to-text does not produce.
So one speech-to-text pass can feed all three. Fixing a misheard name in the transcript should fix it in the captions too, which only happens when they share one source and one timeline.
Batch vs. streaming speech-to-text
Batch speech-to-text runs on a finished file. The model can look at the whole recording, use the words after a sentence to correct the words before it, and take as long as it needs. This is the mode for uploaded lectures, interviews, proctoring sessions and recorded calls, and it gives the cleanest transcript.
Streaming speech-to-text runs on audio as it arrives, for live captions. It emits partial guesses within a fraction of a second and revises them as more audio comes in, then commits a final line. That revision is why live captions sometimes change a word after it appears. Streaming trades some accuracy for speed because the model never sees what comes next.
Many products use both: streaming during a live event so viewers can follow along, then a batch pass over the recording so the replay and its search index get the more accurate text.
How does FastPix handle speech-to-text?
Speech-to-text in FastPix is part of in-video AI and runs in parallel with encoding, not as a second job. The upload that builds the bitrate ladder also triggers transcription, so there is no audio to export and no result to match back to the right media ID afterwards.
Each video returns one timestamped JSON. It holds the transcript with word-level timestamps and speaker diarization, and in the same file the chapters, people and brands, scenes and on-screen text. Transcription covers 50+ languages. From the transcript FastPix generates captions and subtitles in WebVTT and SRT, with editable cue points, so a misheard name can be fixed in the caption track, plus a searchable transcript export. The open-source FastPix Player plays the tracks and styles them through its captions API. A webhook tells your backend when the video is ready, and it is signed with HMAC SHA-256 so you can verify it. Live streams can carry real-time captions while the event airs. For KYC or telehealth recordings with data-location rules, FastPix can set the storage region on request; ask FastPix which regions are available.
Speech-to-text is billed per minute of video processed, on the same bill as encoding, with no token billing and no separate AI invoice. Standard encoding of that video is free up to 4K. The free plan's subtitle and AI sample minutes, no card, are enough to run a batch of your own recordings and check names and jargon.
By the numbers
- OpenAI's Whisper speech recognition model, released in September 2022, was trained on 680,000 hours of multilingual audio collected from the web. (Introducing Whisper (OpenAI))
