Audio and Video
Deck Multimedia Basics Part 2 (Video, Audio): frames, resolution, aspect ratio, flicker, interlacing, NTSC and PAL, composite and component coding, MPEG with I, P and B frames, motion estimation and GOP design; then sound, the sound spectrum, sampling and the Nyquist theorem, quantisation and bit depth, PCM and pulse modulation.
Video basicsSearch on YouTube
Video is a sequence of still images, called frames, shown in rapid succession to give the illusion of motion. The rate is the frame rate, measured in frames per second (fps).
- 30 fps: typical of many TV broadcasts.
- 60 fps: video games and high-quality video, for smoother motion.
Every frame is like a photograph. Played rapidly, the brain reads the frames as continuous motion.
Frame rate effectsSearch on YouTube
- Capture frame rate and display frame rate are usually the same.
- Different capture and display rates create special effects.
Slow motion from the slide. Captured at 240 fps, displayed at 24 fps.
Step 1. One second of real action gives 240 frames.
Step 2. At 24 fps these frames play for seconds.
Step 3. The action looks 10 times slower and stays smooth, because every displayed frame is a real captured frame.
ResolutionSearch on YouTube
Resolution is the number of pixels in each frame. More pixels give better image quality.
| Name | Pixels |
|---|---|
| 720p (HD) | 1280 x 720 |
| 1080p (Full HD) | 1920 x 1080 |
| 2K | 2048 x 1080 |
| 4K (Ultra HD) | 3840 x 2160 |
Raw (uncompressed) video bit rate, 1080p at 30 fps, 24-bit colour.
Step 1. Pixels per frame: .
Step 2. Bits per frame: bits.
Step 3. Bits per second: Gbit/s.
Step 4. One minute: GB. This is why video must be compressed.
Representation of video signals (human perspective)Search on YouTube
A key goal is to give the viewer as realistic a representation of the scene as possible.
Aspect ratioSearch on YouTube
Aspect ratio is the ratio of frame width to height, such as 16:9 or 4:3. It decides how content looks on different screens.
| Ratio | Use |
|---|---|
| 1.33:1 (4:3) | Standard definition (SD) |
| 1.78:1 (16:9) | Widescreen / HD |
| 2.39:1 (21:9) | CinemaScope (widescreen cinema) |
| 1.85:1 | Standard theatrical movies |
| 1:1 | Square format |
| 9:16 | Vertical video |
Depth perceptionSearch on YouTube
- Humans perceive depth by comparing the images from the two eyes, which view from different angles.
- The focal length of the camera lens and changes in depth of focus also influence depth perception.
LuminanceSearch on YouTube
- Colour perception uses three signals, proportional to the red, green and blue light (RGB) in each part of the scene.
- These go to the monitor separately, and the tube reproduces them at each point in time.
- For transmission and storage a different split is often used: one brightness signal (luminance) and two colour-difference signals (chrominance).
Temporal aspects of illuminationSearch on YouTube
Sound is continuous, but images are shown at discrete times. A continuous impression of motion needs two conditions:
- The repetition rate must be high enough for smooth movement from frame to frame.
- The rate must be high enough that the dark intervals between pictures do not disrupt perception.
Continuity of motionSearch on YouTube
- Persistence of vision: the eye retains an image for a fraction of a second after it is gone.
- When frames come fast enough, the brain merges them into continuous motion.
- At about 12 to 15 fps the brain starts to perceive fluid motion.
- Below this rate, motion looks jerky and disjointed.
FlickerSearch on YouTube
If the refresh rate is too low, the perceived brightness fluctuates periodically. This is the flicker effect. The minimum refresh rate to avoid flicker is 50 Hz.
InterlacingSearch on YouTube
Interlacing divides an image into two sets of lines, odd and even, called fields.
Why it is needed:
- Sending a complete frame fast enough to avoid flicker was difficult.
- Showing twice as many fields per second reduces perceived flicker.
Video broadcast standardsSearch on YouTube
| Property | NTSC (National Television System Committee) | PAL (Phase Alternating Line) |
|---|---|---|
| Resolution | 720 x 480 | 720 x 576 |
| Frame rate | 29.97 fps (often called 30) | 25 fps |
| Scan lines per frame | 525 | 625 |
| Interlacing | 2 fields per frame, 262.5 lines per field | 2 fields per frame, 312.5 lines per field |
| Fields per second | 59.94 | 50 |
| Power grid | 60 Hz: USA, Canada, Japan, parts of the Americas | 50 Hz: Europe, Australia, parts of Asia and Africa |
PAL is the "Fifty Five" system: 50 Hz grid, 50 fields, 25 fps, 625 lines, 576 rows. NTSC is the "sixty" system: 60 Hz grid, about 60 fields, 525 lines, 480 rows.
Lines per field and fields per second.
Step 1. NTSC: lines per field; fields per second.
Step 2. PAL: lines per field; fields per second, which meets the 50 Hz flicker limit.
Digitisation of video signalsSearch on YouTube
Digitisation has three steps: sampling, quantisation, coding. Coding comes in two forms.
Composite codingSearch on YouTube
Composite coding digitises the video by combining all components (luminance and chrominance) into a single signal. It is the simplest way to digitise video. Limitations:
- Crosstalk: luminance and chrominance interfere and cause image artifacts.
- Standard dependency: tied to a TV standard (NTSC, PAL, SECAM), so it causes compatibility issues and quality differences between regions.
- Inefficient bandwidth usage: luminance matters more than chrominance, but composite coding treats them equally.
Signal formatsSearch on YouTube
YUV (used with PAL):
YIQ (used with NTSC):
Component codingSearch on YouTube
Component coding digitises the components separately: the luminance and the colour-difference (chrominance) signals.
- Luminance is sampled at a higher frequency, 13.5 MHz.
- Each chrominance signal is sampled at 6.75 MHz.
- Both are quantised with 8 bits, so each sample has possible values.
Bit rate of component-coded video.
Step 1. Samples per second: million.
Step 2. Bits per second: Mbit/s.
| Composite coding | Component coding | |
|---|---|---|
| Signals | One combined signal | Y and two chrominance signals digitised separately |
| Sampling | One rate | Y at 13.5 MHz, chroma at 6.75 MHz each |
| Problems | Crosstalk, tied to NTSC/PAL/SECAM, wastes bandwidth | Needs more channels, but cleaner |
MPEG (Motion Picture Experts Group)Search on YouTube
MPEG is an encoding standard for video and motion pictures. It is lossy.
Why not use JPEG on every frame?
- JPEG compresses each image independently. It removes only spatial redundancy.
- In a video, consecutive frames are highly similar.
- MPEG also exploits this temporal redundancy.
Flow of MPEGSearch on YouTube
Data preparation:
- Each frame holds a full image at some resolution: 1920 x 1080 (Full HD), 720 x 576 (PAL), 720 x 480 (NTSC).
- MPEG uses 4:1:1 sampling (Y : Cb : Cr).
- PAL signal: Y at 352 x 288; Cb and Cr at 176 x 144.
- NTSC signal: Y at 352 x 240; Cb and Cr at 176 x 120.
- Sampling blocks: 16 x 16 for Y, 8 x 8 for Cb and Cr.
Temporal compression: predictionSearch on YouTube
MPEG predicts frames from past and future frames.
| Prediction | Uses |
|---|---|
| Forward | Previous frames |
| Backward | Future frames |
| Bidirectional | Both previous and future frames |
I, P and B framesSearch on YouTube
| Frame | Coded from | Notes |
|---|---|---|
| I-frame (intra-coded) | Itself only | Full image compressed independently. Anchor point; occurs periodically. Largest. |
| P-frame (predicted) | Previous I- or P-frame | Motion compensation stores only the difference from the reference. |
| B-frame (bi-directional) | Previous and future I- or P-frames | Most compressed, because it uses data from both directions. |
I = Independent, P = Past, B = Both.
Motion estimationSearch on YouTube
Motion estimation is used in P- and B-frames to detect how objects or regions move between frames. Instead of storing the whole frame, the encoder stores motion vectors for blocks of pixels called macroblocks.
Slide example: a block showing part of a moving car shifts 5 pixels right and 3 pixels down. The motion vector is (+5, +3).
How it works:
- The encoder divides the current frame into macroblocks, such as 16 x 16 pixels.
- For each block it searches for the best match in a previous frame (P-frame) or in both previous and next frames (B-frame).
- It compares candidate blocks using the Sum of Absolute Differences (SAD).
- The motion vector gives the shift in x (horizontal) and y (vertical).
is the current block and the reference frame. The best match is the shift with the smallest SAD.
SAD on a 2 x 2 block. Current block 10 12 / 11 13.
Step 1. Candidate A in the reference: 10 12 / 12 13. SAD .
Step 2. Candidate B: 20 22 / 21 23. SAD .
Step 3. Candidate A wins. The encoder sends its motion vector and the small residual (0, 0, -1, 0).
Spatial compression and entropy codingSearch on YouTube
Besides temporal redundancy, MPEG removes spatial redundancy inside each frame, as JPEG does:
- Block processing: blocks are split into 8 x 8 pixels.
- DCT: spatial to frequency domain, so high frequencies can be removed.
- Quantisation: reduces DCT precision; loses data but gives large compression.
- Entropy coding: RLE and Huffman coding give short codes to frequent values.
Group of Pictures (GOP)Search on YouTube
MPEG organises frames into a Group of Pictures (GOP). The GOP sets the pattern of I-, P- and B-frames: how often I-frames come and how many P- and B-frames lie between them. The GOP length (frames between I-frames) is a trade-off between compression efficiency and random access.
Common patternsSearch on YouTube
- MPEG-2 and MPEG-4 (broadcasting, DVDs): GOP length 12 to 15. Example
IBBPBBPBBP(1 I-frame, 2 B-frames between P-frames). MPEG-2 may also useIBPBPBPor evenIPPPto reduce complexity. - H.264/AVC: flexible GOP; typically 1 I-frame every 30 to 60 frames. GOP size 30 is common for streaming and broadcast. Example
IBBPBBPBBPBBPfor a longer GOP.
Compression efficiencySearch on YouTube
| Long GOP (30 or more) | Short GOP (12 or fewer) |
|---|---|
| Fewer I-frames, more P and B | More I-frames |
| Higher compression efficiency | Lower compression efficiency |
| Slower seeking; an error spreads further | Better random access (seeking) and error resilience |
GOP sizes by applicationSearch on YouTube
| Application | GOP size | Structure example | Comment |
|---|---|---|---|
| DVD video (NTSC) | 18 | IBBPBBPBBPBBPBBPBB | Common for DVD MPEG-2; good compression |
| Broadcast (PAL) | 12 to 15 | IBBPBBPBBP | Common for MPEG-2 broadcast |
| Streaming (H.264) | 30 to 60 | IBBPBBPBBPBBPBBPBB | Efficient streaming; longer GOP for compression |
| Video editing (H.264) | 12 to 15 | IBBPBBPBBP | Shorter GOP for fast editing and seeking |
| Real-time streaming (H.265) | 10 to 15 | IPPPPPPPPPP | Reduced latency for live streaming |
| High-quality archival (H.265) | 60 to 120 | IBBPBBPBBPBBPBBPBB | Maximum compression for archival |
Seek delay for two GOP sizes at 25 fps.
Step 1. Decoding must start at an I-frame. With GOP 12, the worst case decodes 11 extra frames: s.
Step 2. With GOP 48: s.
Step 3. In 48 frames, GOP 12 stores 4 large I-frames; GOP 48 stores 1. GOP 48 compresses better but seeks slower.
GOP design questions appear in both papers (End Sem 2025 Q4, Mid Sem 2026 Q4). Always state the trade-off: a long GOP saves space, a short GOP gives easy seeking and limits error spread. Then pick the side the scenario needs.
SoundSearch on YouTube
Audio versus soundSearch on YouTube
| Sound | Audio |
|---|---|
| Physical: a mechanical wave caused by vibrations | Mainly an electronic representation |
| Covers audible and inaudible frequencies | Covers only the range audible to humans |
| Originally produced: clap, bird song, thunder | Recorded, manipulated, reproduced: music files, voice recordings, sound effects |
Sound is a form of energy produced by vibrating particles. The vibrations create waves that travel through a medium and reach our ears. A drum skin pushes air molecules together (compression) and lets them spread out, so the air pressure varies with distance from the source.
Can sound travel in space? No. Sound needs a medium such as air or water. Space is a near-perfect vacuum with almost no particles to vibrate.
Sound spectrumSearch on YouTube
| Band | Range | Sources and uses |
|---|---|---|
| Infrasound | 0 to 20 Hz | Earthquakes and volcanoes (early warning of disasters); elephants, whales and giraffes for long-distance communication and navigation |
| Audible range | 20 Hz to 20 kHz | Most multimedia systems operate here |
| Ultrasound | 20 kHz to 1 GHz | Electronic devices for imaging the fetus in pregnancy; physical therapy for pain and inflammation; bats, dolphins, moths and mosquitoes for echolocation, navigation and communication |
| Hypersound | 1 GHz to 10 THz | Generated by laser-based methods and piezoelectric transducers; non-destructive testing, acoustic microscopy (nanoscale imaging), communication, data storage, medical imaging, sensors |
Parameters of audioSearch on YouTube
Sound is a physical phenomenon caused by vibrating materials such as a violin string or a wooden log. The vibrations create pressure waves in air. The pattern of these oscillations is a waveform.
| Parameter | Definition | What you hear |
|---|---|---|
| Frequency | Number of complete cycles per second | Pitch: higher frequency, more cycles, higher pitch |
| Amplitude | Maximum displacement from the equilibrium position | Loudness |
| Phase | Position of a point in time on the waveform cycle; the starting point relative to a reference | Matters when waves combine |
Exp 8 to 11 (Audacity).
- Exp 8: two identical tones played together. The amplitude doubles (constructive interference); the frequency does not change.
- Exp 9: the second tone is inverted (Effect > Invert). The two cancel to silence: destructive interference.
- Exp 10: 440 Hz and 550 Hz combined. The waveform becomes a complex, repeating shape; Plot Spectrum shows two separate peaks.
- Exp 11: Effect > Change Speed. Faster playback raises frequency and pitch; slower playback lowers both.
These match the deck's task sheet: tones of different frequency, amplitude changes, phase inversion, speed changes and mixing two frequencies.
Audio representation on computersSearch on YouTube
Sound is a continuous wave. A computer needs it in discrete form. Taking values at discrete instants is sampling.
- An analog-to-digital converter (ADC) stores sound in the computer.
- A digital-to-analog converter (DAC) reproduces it at the output.
How many samples to take?Search on YouTube
The number of samples is not arbitrary. It is set by the Nyquist-Shannon sampling theorem:
Any band-limited continuous-time signal can be accurately converted to and from digital form when sampled at a rate at least twice the highest frequency component of the waveform.
A component at is aliased. It reappears at for the integer that brings it into .
The deck's own example: for a 1 Hz signal, the sampling rate should be more than 2 samples per second, say 3 Hz. Anything above 3 Hz is fine; 3 Hz is the minimum it suggests.
Hertz in a sampling rate is not the same as hertz in a frequency. Frequency is cycles per second. Sample rate is samples per second.
- With few samples the shape of the wave can be misread. With a higher sampling rate there is less chance of misinterpretation.
- Misconception: a higher sampling rate does not mean better audio quality. Above the Nyquist rate, extra samples add no audible information.
Nyquist and aliasing.
Step 1. A signal has components at 500 Hz and 750 Hz. Hz, so Hz.
Step 2. Sample it at 1000 Hz instead. The Nyquist frequency is 500 Hz, so 750 Hz is above it.
Step 3. Alias: Hz. The 750 Hz tone is heard as a false 250 Hz tone.
Step 4. Audio up to 18 kHz needs kHz. At 30 kHz an 18 kHz tone aliases to kHz.
Practice from the deckSearch on YouTube
- Set the project sampling rate to the lowest value, 8000 Hz.
- Generate a 1 kHz signal and listen: it plays correctly.
- Generate a 4 kHz signal: all sampled points fall at 0, so no wave is recovered. 4 kHz is exactly .
Interpretation of samples (band limiting)Search on YouTube
The samples are not joined dot to dot. In practice the output passes through an electronic low-pass filter (LPF) that rebuilds a smooth curve. This is band limiting. In theory, sampling a band-limited signal is mathematically lossless.
Limitation of the low-pass filter: an ideal filter cuts off vertically at the cut-off frequency. A real filter rolls off gradually. So the sampling rate must leave a guard band between the highest wanted frequency and the Nyquist frequency.
Common audio sampling ratesSearch on YouTube
All three use a band limiter with a 20 kHz cut-off.
| Sample rate | Nyquist frequency | Guard band above 20 kHz |
|---|---|---|
| 44.1 kHz | 22.05 kHz | 2.05 kHz |
| 48 kHz | 24 kHz | 4 kHz |
| 96 kHz | 48 kHz | 28 kHz |
Exp 12: sampling frequency versus audio (Audacity). A 3 kHz tone at project rates 8000, 11025 and 44100 Hz. The Nyquist frequencies are 4000, 5512.5 and 22050 Hz, so 3 kHz plays cleanly in all three. At 8000 Hz a 4 kHz tone (exactly ) can land on the zero crossings and vanish. 3999 Hz is slightly below the limit, so it is in theory recoverable, though it looks like a slowly beating wave. Above 4 kHz the tone aliases to a lower frequency. Nyquist criterion: .
Quantisation and bit depthSearch on YouTube
After sampling we have values at discrete times. Each value still has infinite precision, for example 0.2123523421343123464236. The amplitude axis also needs a limited resolution.
- This resolution strongly affects signal accuracy.
- Sampling is lossless, but choosing the resolution is lossy.
Quantisation maps analog signal values to a limited range of discrete values. The sampling resolution (the size of the step on the amplitude axis) sets the maximum dynamic range.
With bit depth there are levels:
| Bit depth | Levels |
|---|---|
| 3 bits (slide figure) | 8 |
| 16 bits (common) | 65,536 |
| 24 bits (common) | 16,777,216 |
Quantisation noise is the difference between the original wave and the quantised staircase. The slide shows it as original minus quantised.
For full range and bits:
The SQNR rule (about 6 dB per bit) is standard signal theory, not printed on the slides.
8-bit versus 12-bit (Mid Sem 2026, Q1).
Step 1. Levels: against , that is 16 times more.
Step 2. Step size and maximum error shrink 16 times.
Step 3. SQNR: about 49.9 dB against 74 dB, roughly 24 dB better.
Step 4. Quiet passages use only a few of the 256 levels, so their error is large relative to the signal. With 12 bits the distortion in quiet parts drops. Cost: 50% more bits per sample.
Audio bit rate and file size: CD quality. 44.1 kHz, 16 bits, stereo.
Step 1. Bit rate: bit/s, about 1411 kbit/s.
Step 2. One minute: bits.
Step 3. In bytes: bytes, about 10.6 MB per minute.
Step 4. General rule: size = sample rate x bit depth x channels x duration / 8.
Exp 13: bit depth versus audio (Audacity). A 16-bit track is reduced with the Nyquist Prompt quantize function to 8-bit and 4-bit. To isolate the noise, invert the quantised track, then Mix and Render it with the original. What remains is the quantisation noise. The 4-bit version has the loudest noise and the most visible staircase, because it has only 16 levels and the widest step.
Encoding and transmission (digital audio)Search on YouTube
Encoding converts raw data into another form suited to transmission or storage. The most common lossless encoding scheme for audio is PCM.
Pulse Code Modulation (PCM) digitally represents analog signals. It converts an analog audio signal into a digital bitstream that can be stored, processed or transmitted.
Why modulation matters in transmissionSearch on YouTube
- Low-frequency signals have low energy, so they cannot travel long distances.
- Noise easily erodes the message.
- Frequent amplification is needed, which is expensive.
- Amplification also adds noise.
- Lower frequencies demand larger antennas.
Modulation varies one or more properties of a periodic waveform, the carrier signal, with a separate modulation signal that holds the information. The slide shows amplitude modulation and frequency modulation of a carrier.
Pulses for digital signalsSearch on YouTube
These ideas are for the analog domain. A digital signal has already been sampled and quantised, for example 44.1 kHz at 16-bit depth. It is represented with pulses: time-controlled electric signals.
| Pulse modulation | What the sample value controls |
|---|---|
| Pulse Width Modulation (PWM) | The width of the pulse |
| Pulse Position Modulation (PPM) | The position of the pulse in time |
| Pulse Amplitude Modulation (PAM) | The height of the pulse |
| Pulse Code Modulation (PCM) | A binary code word sent as a train of on/off pulses (for example 10000000, 11110010) |
Wide, Placed, Amplified, Coded: width, position, amplitude, code.
Quick self-testSearch on YouTube
Why is MPEG needed when JPEG already compresses images?
JPEG removes only spatial redundancy inside one image. Consecutive video frames are highly similar, so MPEG also removes temporal redundancy with motion-compensated prediction. Spatial plus temporal redundancy equals MPEG.
Compare I, P and B frames.
I-frames are coded independently and are the largest. P-frames are predicted from a previous I- or P-frame. B-frames use both previous and future I- or P-frames and are the most compressed.
What is a motion vector and how is the best match found?
A pair of numbers giving the horizontal and vertical shift of a macroblock between frames, such as (+5, +3). The encoder picks the candidate block with the smallest Sum of Absolute Differences.
Long GOP versus short GOP?
A long GOP (30 or more) has fewer I-frames, so it compresses better. A short GOP (12 or fewer) gives better random access and error resilience.
Give NTSC and PAL frame rate, lines and fields per second.
NTSC: 29.97 fps, 525 lines, 59.94 fields per second, 720 x 480. PAL: 25 fps, 625 lines, 50 fields per second, 720 x 576.
What is interlacing and why was it introduced?
Each frame is split into two fields of odd and even lines. Sending a full frame fast enough to avoid flicker was hard. Showing twice as many fields per second reduces perceived flicker.
State the Nyquist-Shannon theorem and give the minimum rate for audio up to 18 kHz.
A band-limited signal can be reconstructed when sampled at least twice its highest frequency. kHz.
What happens if you sample a 4 kHz tone at 8 kHz?
4 kHz is exactly . The samples can all land on zero crossings, so no wave is recovered. The deck's practice shows all sampled points at 0.
Why is 44.1 kHz used instead of the theoretical 40 kHz for 20 kHz audio?
Real low-pass filters cannot cut off instantly. 44.1 kHz puts the Nyquist frequency at 22.05 kHz, which leaves a 2.05 kHz guard band for the filter roll-off.
How many levels does 16-bit audio have, and what is quantisation noise?
levels. Quantisation noise is the difference between the original signal and its quantised value; fewer bits give larger steps and more noise.
- Video is frames at a frame rate; 12 to 15 fps looks continuous, 50 Hz refresh avoids flicker.
- Interlacing sends odd and even fields; NTSC is 525 lines at 29.97 fps, PAL is 625 lines at 25 fps.
- Component coding samples Y at 13.5 MHz and chroma at 6.75 MHz with 8 bits each.
- MPEG removes spatial and temporal redundancy: I, P and B frames, motion vectors found by SAD.
- GOP length trades compression (long) against seeking and error resilience (short).
- Sound needs a medium; audible range is 20 Hz to 20 kHz.
- Sample at least twice the highest frequency, or aliasing creates false tones; real filters need a guard band.
- Bit depth gives levels; PCM (sampling, quantisation, encoding) is the standard lossless audio coding.