How does a computer play a video while doing something else?

Question

How is video playback done on a computer? It's obviously not relying purely on the CPU, since video playback continues when a user performs another activity, such as typing into a YouTube comment field. This task seems further complicated in the case of streaming video, where data presumably needs to be transferred from the network interface to the video/graphics controller.

EDIT: I should have made clear that I understand that the CPU switches among multiple processes and threads all of the time. My question was meant to be more specifically about video playback. Is the video processing done on the graphics chipset/card? Do those typically offer buffering? Or can it all be handled by a single-core CPU with time to spare for other tasks, or am I wrong to think that low-end CPUs can play back video without (not-network) delays?

Answer 1

This is implemented using several different techniques. Within the application (for example, the web browser), the program can execute different threads of execution. One thread can perform the video streaming/playback, while a different thread can handle the user typing into the comment field.

These threads, along with many other threads and processes, are in turn scheduled by the operating system (even if you only have the web browser open, the operating system has many background tasks running, such as the windowing manager, the print spooler, the USB device manager, etc). It is the job of the OS scheduler to determine which thread can actively run, on which processor core, and for how long. In a typical system, a thread can run for up to some threshold value, suppose 100 milliseconds, before the scheduler allows another task to run.

This task switching is fundamental to all modern operating systems (Windows, Linux, UNIX, etc), and is usually a significant portion of an undergraduate operating systems course.

Answer 2

A modern graphics card works more or less (somewhat) the same as a regular cpu.

On a graphics card, you will have (sometimes) multiple processors, each processor will have (sometimes) multiple multiprocessors, which each have multiple cores.

When a video is loaded into a graphics card it gets transcoded into the output buffer by a certain allocation of multiprocessor cores. This output buffer is generally globally accessible memory, which means that any core from any multiprocessor from at least one processor (but often all) has direct r/w access to it.

Usually one multiprocessor is limited to one instruction set, which means that a graphics card can only concurrently operate as many different "flavours" of threads as there are multiprocessors on the card. A "flavour" might for instance be a physics simulation, a video render, an OS render, or a cryptographic function. Naturally one multiprocessor can schedule multiple thread types to run intermittently, but this is usually not necessary.

Decoding a video is often a lot of work, considering how it's usually compressed and sometimes encrypted, so the displaybuffer isn't always busy. Thus it's relatively easy to render a mouse cursor traveling over a video frame. However, sometimes this doesn't quite work, and you will see how in some applications your cursor disappears. This is not necessarily because the application is "on top" of the os, but simply because it's hogging part of the output buffer.

Answer 3

Video playback isn't that special. Sure, GPU-accelerated decoding is common these days, which frees up the CPU for other work, but it wasn't always so. Smooth video playback with only a single CPU is definitely possible, as long as there's enough CPU available for the resolution and complexity of the compression algorithm, and doing it along with other tasks is just as possible, as long as there's enough CPU for both, and the OS schedules well. 30fps video is common, which allows 33ms to decode each frame. Even fairly old chips can manage that for SD-quality MPEG-2, and modern chips can manage it for HD MPEG-4; the laptop I'm typing this on requires about 20% of the CPU to playback 720p High Profile AVC.

As for buffering, there's a limited amount available. The minimum for anything that doesn't want to have screen tearing is "double buffering", where there are two frames in memory: the video card displays one while the software updates the other, and the roles of the two are swapped during the vertical refresh. With "triple buffering", there are two offscreen frames, the next and next + 1, which reduces the amount of jitter. On modern cards with large amounts of RAM it's possible to go beyond that to larger numbers of frames, with the only downsides being user-visible latency in seeking, starting playback, etc. and the fact that other apps might want that video RAM for themselves.

Answer 4

Let's say a computer has just one core. You ask "how can it play video while at the same time allowing me to type." Answer: It doesn't. Your video may play at a rate of 30 frames per second, that is 33.3 milliseconds per frame. Maybe your computer takes only 20 milliseconds to draw that frame, then it has plenty of time to handle you typing in text. It's just so fast that you don't notice that it doesn't happen at the same time.