Video search
Video search is the ability to find videos, or exact moments inside them, by typing a query. Basic video search matches titles, tags and descriptions. Deeper search also indexes what is said and shown: transcripts, on-screen text, people, objects and scenes, each with a timestamp, so a result can open the player at the right second. Search can match exact words, match meaning through embeddings, or combine both. Inside a product, it lets someone ask a question and land on the minute of a lecture, call or recording that answers it.
How does video search work?
Video search has two halves: indexing, which happens once per video, and querying, which happens every time someone searches.
Indexing turns a video into searchable data. Metadata comes first: title, description, tags. Speech-to-text adds a transcript with a time on every word. OCR reads text that appears on screen, such as slides or lower thirds. Vision models detect scenes, faces, objects and brands, again with timestamps. For search by meaning, each segment is also converted into an embedding, a list of numbers that places similar content close together.
At query time, the search system matches the query against that index, ranks the results and returns the video plus the start time of each matching segment. The player then seeks straight to that point. In HLS, seeking means loading the segment that contains the timestamp, so a search result can start playing within a moment, without downloading the video from the beginning.
Keyword vs. semantic video search
Keyword search uses an inverted index, a lookup from each word to every place it appears. It is precise and predictable. A search for an order number, a person's name or a product code finds exactly those words in titles and transcripts, and it is easy to explain why a result matched. It misses synonyms: a query for cancel my plan will not find a clip that says end your subscription.
Semantic search uses embeddings. The query and every indexed segment are turned into vectors, and results are ranked by how close they are in meaning. It handles natural questions and different wording well, and multimodal embeddings can match a query to what is shown, not only what is said. Its weakness is exactness: it can rank a near miss above the precise name you typed.
Most production systems run both and merge the results, which is called hybrid search.
What does a video search index include?
The more layers a video search index holds, the more kinds of questions it can answer. Each layer needs a timestamp, or results can only point to the whole video. Recorded review workflows add their own needs: in India, RBI rules for video KYC ask that V-CIP recordings be kept in systems located in India with a date and time stamp for easy historical search, so the index has to carry recording time as well as media time.
- Metadata: title, description, tags and custom fields you assign.
- Transcript: every spoken word with its time and speaker label.
- On-screen text: slides, captions, signs and lower thirds read by OCR.
- Visual entities: people, objects, brands and scene changes.
- Chapters: topic boundaries that group long videos into sections.
- Embeddings: vectors for search by meaning across speech and visuals.
How do you add video search to an app?
Start with the questions your users ask. A course platform needs to find where a lecture explains a concept; a review team needs the second a phrase was spoken; a newsroom needs every clip with a given person on screen. Those questions decide which layers to index.
Next, make indexing part of ingest. If transcripts and visual data are produced when a video is uploaded, every new video is searchable as soon as it plays. A separate indexing job that runs later leaves gaps.
Then respect access. Search results expose content, so the search layer has to apply the same permissions as playback. Private recordings, such as interviews or telehealth visits, should open through signed, expiring playback URLs, not public links.
Finally, link results to time. Each result should carry a start time, and the player should seek to it. Showing the matching transcript line beside the result helps people choose before they press play.
How does FastPix handle video search?
FastPix builds the data for video search while the video is processed. In-video AI runs in parallel with encoding on each upload and returns one timestamped JSON per video with the transcript, including word-level timestamps and speaker diarization, plus chapters, people and brands, scenes and on-screen text. Transcription covers 50+ languages. Auto chapters are written to the player or the manifest, so viewers can jump between sections without any search at all.
For search by meaning, FastPix produces multimodal embeddings across visual content, audio, speech and on-screen text, and supports conversational video search over them. You can also take the timestamped JSON and feed it into your own keyword index if you need exact matching on names and codes.
Your app maps each result to a playback ID and a start time, and the FastPix Player, or any HLS player, opens the video at that second. For private recordings, the playback ID can be signed with an expiring JWT, so a search result only plays for someone your app has authorized. For KYC recordings with data-location rules, FastPix can set the storage region on request; ask which regions are available. A ready webhook tells your backend when to pull the JSON into your index.
The analysis runs once per video and is priced per minute, alongside encoding on one invoice. Standard encoding is free up to 4K. Index a small set of your own recordings with the free plan's AI sample minutes, no card, and try the questions your users ask.
