Vision language model (VLM)
Vision language models (VLMs) are AI models trained on both images and text, so they can look at a picture or video frame and describe, answer questions about or reason over what they see in plain language. A VLM typically pairs an image encoder with a large language model. Given sampled frames from a video, a VLM can write captions, list objects or explain a scene.
Why it matters
VLMs let software describe video in sentences instead of fixed labels, so one model can caption, search and review footage. That reduces the need for many narrow detectors. The tradeoff is compute cost and occasional confident mistakes, so outputs still need checks in sensitive workflows.
How it works
- An image encoder turns each sampled frame into a set of numeric features.
- A connector layer translates those features into a form the language model can read.
- The language model combines the visual input with a text prompt, such as a question.
- It writes a plain language answer, caption or list grounded in what it saw.
What is the difference between a VLM and an LLM?
An LLM (large language model) works only with text. A VLM adds an image encoder so it can also take pictures or video frames as input. Most VLMs are built by attaching a vision encoder to an existing LLM, so they keep the language skills and gain the ability to see.
Can vision language models understand video?
Partly. Most VLMs process video as a set of sampled frames, sometimes with the transcript added. That works well for describing scenes and answering questions about short clips. Long videos are harder because only a limited number of frames fit into the model at once, so brief events between samples can be missed.
Where it's used
Newsrooms use VLMs to describe incoming footage in plain English so editors find the right shot fast, and creator platforms use them to review uploads with context rather than simple labels.
