The term "Graphics Processing Unit" (GPU) was introduced by NVIDIA in 1999 with the launch of the GeForce 256. This card was marketed as the first true GPU, capable of handling both graphics rendering and complex calculations related to graphics processing, including hardware transformation and lighting.
Video processing has advanced quickly, driven by the increasing demand for high-quality content and the need for efficient systems. As the focus shifts toward higher resolutions and more complex visual effects, GPUs and VPUs have become essential tools for developers and content creators.
What are GPUs?
GPUs are built to handle tasks like image and video rendering, improving performance for graphics-heavy activities like gaming, design, or machine learning. They excel at managing many tasks at once by breaking them into smaller parts and running them in parallel. This makes them perfect for graphics rendering, video encoding, and processing large amounts of data quickly.
On the other hand, CPUs are great for tasks that require a step-by-step approach, like running operating systems and general apps. They focus on sequential tasks, making them versatile for day-to-day computing. While GPUs handle heavy lifting in graphics and data, CPUs keep things running smoothly behind the scenes.
Why use GPUs instead of CPUs for video processing?
GPUs are much better at parallel processing than CPUs, especially for repetitive tasks like audio and video encoding and decoding. While CPUs focus on sequential processing and can handle only a few tasks simultaneously, GPUs can manage thousands of threads simultaneously.
GPUs are great for video and audio tasks because they spread repetitive calculations across multiple cores. For example, when encoding a video, GPUs can process many frames at the same time, resulting in faster render times and smoother playback.
GPU architecture
At the heart of a GPU are individual processing units called cores. These cores are much smaller and more specialized than CPU cores, which focus on handling more complex tasks one at a time.
GPU cores, on the other hand, are designed to execute many simpler tasks all at once. This allows GPUs to perform thousands of operations, making them ideal for parallel processing tasks like rendering graphics or video processing.
By using a model called Single Instruction Multiple Threads (SIMT), each core can execute the same instruction while working on different pieces of data. This design helps reduce delays and increase processing speed, allowing for quick data handling.
How do GPUs achieve parallelism?
To understand how GPUs handle large tasks, it’s important to look at how they manage parallel work. A key part of this is the Streaming Multiprocessors (SMs), which split up tasks into smaller pieces and assign them to thousands of threads that can run at the same time.
SMs make sure everything is handled efficiently by managing both the processing and memory, allowing the GPU to easily take on complex tasks like video processing or machine learning.
- Threads are the smallest units of work in a GPU, each responsible for executing a specific task, such as performing calculations on a single pixel in video processing. A GPU can manage thousands of threads simultaneously, which allows it to perform many small operations in parallel.
- Blocks are groups of threads. When a task is executed, it’s broken down into multiple threads organized into blocks. Each block operates independently and can share data through shared memory, which is accessible by all threads within the block. This setup is particularly efficient for tasks that require threads to collaborate, like applying filters or processing parts of a video frame.
Memory hierarchy in a GPU
- Global memory: Global memory is shared across all cores and acts as the primary storage area for all the data the GPU handles. It’s like a vast library to hold video files, textures, and other essential assets. Global memory is accessible by all threads in a GPU, but it is slower compared to other types, which can create some latency.
In video processing, global memory is where raw video streams and intermediate frames are stored. To keep things running smoothly, GPUs try to limit how much data is sent to and from global memory by utilizing faster memory types during demanding tasks.
- Shared memory: Shared memory is a smaller, faster memory space shared by threads within the same block. It helps threads communicate and synchronize more effectively. This memory is used for sharing data that multiple threads need, reducing delays and speeding up processing.
- Registers: Finally, there are registers, which are the fastest type of memory. Each thread in a GPU has access to a limited number of registers that store local variables and intermediate results.
In video processing, registers are important for speeding up shader and pixel processing. During shader operations, registers temporarily hold pixel data, such as color values and texture coordinates, needed for rendering frames. This setup enables GPUs to execute quick calculations and transformations, resulting in a smooth and efficient rendering process.
What are VPUs?
VPUs (Video Processing Units) are designed to handle tasks like video decoding, encoding, and image processing. By offloading these jobs from the CPU, VPUs help videos run smoothly while using less power. They’re built to make video-related tasks more efficient, which is handy for streaming or media-heavy apps.
VPUs are a type of ASIC (Application-Specific Integrated Circuit), meaning they’re made for specific tasks. Unlike CPUs or GPUs that handle many tasks, ASICs focus on one area. You’ll find ASICs in everything from phones to cars and even cryptocurrency mining, where their specialized design speeds up certain processes.
VPU architecture
A VPU is built with dedicated processing cores specifically for video tasks. These cores can handle multiple operations at once, making processes like rendering, decoding, and encoding faster. This parallel processing is what gives VPUs their edge in video-related work.
The VPU’s design tackles challenges like handling motion in video, applying real-time effects, and keeping things running smoothly even with high-resolution video or heavy workloads. It’s what makes VPUs ideal for demanding tasks like streaming or live video editing without losing performance.
Memory hierarchy in a VPU
VPUs, like GPUs, use a layered memory system with fast local memory (on-chip or cache) and external memory like DRAM. Cache memory offers quick access to frequently used data, reducing delays during processing tasks. Meanwhile, DRAM handles larger video files and assets but works slower than cache memory.
VPUs have specialized buffers called frame buffers. These buffers temporarily hold the pixel data that will be displayed on the screen, allowing the GPU to render images while keeping the frame rate consistent. VPUs store this data in a buffer until it's ready for the next screen refresh, which prevents tearing or visual glitches during transitions.
These buffers work on the FIFO (First In, First Out) principle. FIFO ensures that the data stream is processed in the correct order, handling video streams and sequential tasks to avoid bottlenecks or skipped frames.
What are image signal processors?
Image Signal Processors (ISPs) are hardware chips within VPUs designed to improve image quality. They implement various algorithms for tasks like noise reduction and color correction, which improve the overall visual output of video data. (this whole sentence)
ISPs also have built-in support for specific codecs and video formats, which helps them handle real-time encoding and decoding smoothly. This support reduces the workload on the main CPU, allowing for better management of high-resolution video streams in applications like video conferencing, live streaming, and video editing.
