Digital Images, JPEG and PNG
Second part of the deck Multimedia Basics (up to PNG): how a camera turns a scene into a file (lens to storage), lossy JPEG compression step by step (YCbCr, chroma downsampling, DCT, quantisation, zigzag, RLE, Huffman) and lossless PNG (Huffman, RLE, LZSS, the five filters).
An image: from scene to screenSearch on YouTube
Cameras are everywhere and drive digital image acquisition. Image forensics needs this background: to tell if an image was tampered with, you must know how it is captured, processed and saved to memory.
A photo is a snapshot of time. Of the infinite light bouncing off a scene, the camera captures only the light that enters it. The basic acquisition chain has seven components:
Lazy Apes Sleep Calmly So Pandas Sing: Lens, Aperture, Shutter, CFA, Sensor, Processing, Storage.
LensSearch on YouTube
Camera lenses work like our eyes. The eye focuses light onto the retina. A lens gathers the scattered light from objects and directs it along the camera's internal path. This focused light forms the image.
ApertureSearch on YouTube
The aperture sits inside the lens. It has two functions:
- Controlling light: it sets how much light enters. A wider aperture gives a brighter image; a narrower one gives a darker image.
- Adjusting focus: changing the aperture size changes the depth of field. The subject stays sharp while the surroundings blur, which directs attention to the main subject. (The slide shows f/2.8 with a shallow depth of field and f/16 with a deep one.)
Mirror box and shutterSearch on YouTube
- Mirror box: sits before the shutter in a DSLR. It flips the image from the lens so the photographer sees it the right way up in the viewfinder.
- Shutter: when the shutter button is pressed, the mirror moves up and light passes to the image sensor.
- Shutter speed is how long the sensor is exposed to light. Higher shutter speeds suit fast-moving objects: they reduce the light entering and prevent motion blur.
Mid Sem 2026, Q2: motion blur on a fast athlete. Answer: raise the shutter speed. The exposure time gets shorter, so the subject moves less during the exposure. Less light reaches the sensor, so open the aperture or raise ISO to compensate.
Colour filter array (CFA)Search on YouTube
The CFA is part of the sensor and captures colour. It is a mosaic of microscopic coloured filters placed over the photodiodes. It mimics the rods and cones of the eye:
| Eye cell | Count | Role |
|---|---|---|
| Rods | about 100 million | Capture illumination (brightness) detail; crucial for night vision |
| Cones | about 6 million | Colour vision; distinguish hues and shades in daylight |
The Bayer filter is the most common CFA. It repeats a pattern of red, green and blue filters. Each filter lets only its colour through to the photodiode below, which records that intensity.
- Ratio R : G : B = 1 : 2 : 1, because the eye is more sensitive to green.
- So the CFA captures red at 25%, green at 50% and blue at 25% of the pixels.
Demosaicing algorithms rebuild the missing two colours at each pixel. They interpolate from neighbouring pixels that hold the complementary colours. The result is a full-colour image.
SensorSearch on YouTube
- Each pixel on the sensor contains a photodiode.
- Light hitting a photodiode excites electrons; the count follows the light intensity.
- An analog-to-digital converter (ADC) turns the electron count into a digital value.
- Each value is the brightness of red, green or blue at that pixel.
- The collected values form the RAW image: a map of light intensity for the whole sensor.
Processing unitSearch on YouTube
The processor estimates the missing colour values and produces the full-colour image. It also performs:
- white balance adjustment,
- camera noise reduction,
- demosaicing,
- brightness and gamma correction,
- sharpening.
StorageSearch on YouTube
Images go to removable memory cards or internal storage.
| Format | What it keeps | Trade-off |
|---|---|---|
| RAW | All unprocessed sensor data, RGB intensity per pixel | Maximum detail, more storage |
| Compressed (commonly JPEG) | Discards some data | Small files for everyday photos, some detail lost |
Squeezing images: from screen to storageSearch on YouTube
Why compress?
- It reduces file size, so images are easier to store and share.
- It speeds up transfer over networks.
- It optimises memory use, so more images fit in limited storage.
Two classes:
- Lossless compression preserves all image data with no loss in quality. Use it where integrity matters, such as archival storage.
- Lossy compression sacrifices some quality for a much higher compression ratio. Use it for everyday images where a little detail loss is acceptable.
Why JPEG? JPEG (Joint Photographic Experts Group) balances compression efficiency and image quality. Its compression level is adjustable, so the user trades file size against quality. Almost every device and program supports it. The slide example: a 40 MB image becomes 4 MB.
JPEG: how it worksSearch on YouTube
JPEG has five stages:
Cats Do Dance Quietly Right Here: Colour conversion, Downsampling, DCT, Quantisation, RLE, Huffman.
1. Colour space conversionSearch on YouTube
The eye has rods (intensity) and cones (colour). JPEG copies this split: it separates brightness from colour.
Each pixel has R, G and B values. JPEG converts them into:
- Luminance (Y): the brightness or intensity of the pixel.
- Chrominance blue (Cb) and chrominance red (Cr): the colour components.
The conversion keeps all data and is reversible, so no information is lost yet.
The matrix on the slide:
The JFIF file format then adds 128 to Cb and Cr so they fit the range 0 to 255.
Convert a pure red pixel (R, G, B) = (255, 0, 0).
Step 1. .
Step 2. .
Step 3. .
Step 4. Red is fairly dark in luminance (76 of 255) but has a large positive Cr, as expected.
2. Chrominance downsamplingSearch on YouTube
This stage discards a large amount of data.
- Y is crucial for visibility, so it stays unchanged.
- Cb and Cr are averaged over every 2 by 2 block of pixels, which cuts each to one fourth of its size.
- The resulting ratio is .
- On reconstruction, Cb and Cr are scaled back up to the size of Y. RGB values are then rebuilt pixel by pixel from the rescaled chrominance.
Data saved by 2 by 2 chroma averaging.
Step 1. Before: 3 values per pixel (Y, Cb, Cr). For 4 pixels: 12 values.
Step 2. After: 4 Y values + 1 Cb + 1 Cr = 6 values for 4 pixels, that is 1.5 values per pixel.
Step 3. Saving: of the data, with almost no visible change.
Exp 3 and 4 (Python, OpenCV, NumPy, Matplotlib).
- Exp 3: read a colour image, split and display the R, G and B channels, convert to YCbCr and display Y, Cb and Cr separately. Y looks like the grayscale version of the image. Cb and Cr carry only colour difference.
- Exp 4: implement chroma sampling 4:4:4 (no subsampling), 4:2:2 (horizontal only) and 4:2:0 (horizontal and vertical), rebuild and compare. The three look almost identical because the eye is less sensitive to colour detail. 4:2:0 gives the highest compression and is the usual choice in JPEG. It matches the 2 by 2 averaging on the slides.
3. Discrete cosine transform (DCT)Search on YouTube
No compression has happened so far. Now JPEG must decide which data to throw away without hurting the image. It does this in the frequency domain.
- Spatial-to-frequency conversion measures the rate of change of pixel intensity in a region.
- High-frequency components (rapid changes, fine detail) can be removed.
- Our eyes are less sensitive to high frequencies.
The DCT converts image data from the spatial domain to the frequency domain. It breaks the image into frequency components that describe variations in brightness and colour. It packs the signal energy into a few low-frequency coefficients, which makes compression efficient.
Block-based processing:
- The image is divided into 8 by 8 pixel blocks.
- The DCT is applied to each block independently.
- High-frequency coefficients, which hold fine detail, are later quantised away.
Level shift: subtract 128 from every pixel so values lie in -128 to 127. The slide gives three reasons:
- DCT efficiency: the DCT works best with input centred on zero.
- Mathematical simplification: centring around zero simplifies the cosine calculations.
- Reduced redundancy: removing the constant bias lets the DCT focus on actual variation.
The DCT assumes any numeric signal can be rebuilt from a combination of cosine functions, the basis functions. An 8 by 8 block has 64 basis images, from flat (top left) to a fine checkerboard (bottom right). Each DCT output coefficient says how much of its basis image the block contains. A high coefficient means a strong presence of that frequency. The top-left coefficient is the DC term (average brightness). The other 63 are AC terms, ordered from low through mid to high frequency.
2-D DCT of an 8 by 8 block (standard JPEG form, not printed on the slide):
with and for . For the DC term this reduces to
4. QuantisationSearch on YouTube
The DCT does not discard any data, so no compression has happened yet. Quantisation is where the actual compression (and the loss) occurs. Each DCT coefficient is divided by the matching entry of a quantisation table and rounded to the nearest integer:
- The table has smaller numbers on low frequencies and higher numbers on high frequencies.
- Most high-frequency coefficients become zero. That is data our eyes do not perceive.
Quantising the first three rows of the slide's block.
| DCT row 1 | 560 | -41 | -57 | -12 | -4 | 0 | 1 | 1 |
|---|---|---|---|---|---|---|---|---|
| Q row 1 | 4 | 3 | 4 | 4 | 4 | 6 | 11 | 15 |
| Result | 140 | -14 | -14 | -3 | -1 | 0 | 0 | 0 |
Step 1. . . .
Step 2. Row 2: , , , , .
Step 3. Row 3: , , , .
Step 4. Small values divided by large table entries (high frequencies) round to 0. From row 5 down, the whole quantised block is zero.
Lab Exp 5 block: the DC coefficient by hand. The 8 by 8 block starts 52 55 61 66 70 61 64 73 (the classic JPEG test block).
Step 1. Level shift: subtract 128 from all 64 pixels. The first row becomes -76, -73, -67, -62, -58, -67, -64, -55.
Step 2. Sum of all 64 shifted values: .
Step 3. .
Step 4. Quantise with the standard luminance table, whose top-left entry is 16: .
Step 5. Counting the whole block: 20 coefficients stay non-zero with the table scaled by 1, 24 with 0.5, and 8 with 4. Larger table values mean more zeros, more compression and a worse rebuild (largest pixel error 9, 15 and 29 for 0.5, 1 and 4).
Mid Sem 2026, Q5(b): the stage that gives JPEG its flexibility is quantisation. The quality slider scales the quantisation table. Bigger table values give more zeros, a higher compression ratio, and more blocking and blurring. Smaller values give better quality and bigger files.
5. Zigzag scan, RLE and Huffman codingSearch on YouTube
The quantised block is read in zigzag order, from the top-left (low frequency) to the bottom-right (high frequency). This groups the zeros at the end.
Run-length encoding (RLE) replaces a sequence of the same value with a count and the value. Slide example: WWWWWWWWBBBBBBBBBBBBZWK becomes 8W-12B-1Z-1W-1K.
From the quantised block to the RLE stream (slide example).
Step 1. Zigzag list of the 64 luminance values: 140, -14, -60, 14, -8, -14, -3, 14, 4, 0, 0, -2, 4, 4, -1, 0, 2, -1, -2, then six 0s, -1, then 38 0s.
Step 2. RLE: 140, -14, -60, 14, -8, -14, -3, 14, 4, 0[x2], -2, 4[x2], -1, 0, 2, -1, -2, 0[x6], -1, 0[x38].
Step 3. Check: 9 + 2 + 1 + 2 + 1 + 1 + 1 + 1 + 1 + 6 + 1 + 38 = 64 values.
Step 4. The RLE symbols then go to Huffman coding, which gives the most frequent symbols the shortest codes.
Exp 5 and 6: DCT, quantisation, zigzag and RLE (Python).
- Exp 5: level-shift the given 8 by 8 block, apply the 2-D DCT, quantise with the standard luminance table at 0.5, 1 and 4 times strength, count non-zero coefficients, then dequantise and apply the inverse DCT. Only quantisation loses information. The DCT keeps all 64 values; it only moves the energy to the top left.
- Exp 6: zigzag scan the quantised matrix into 64 values and run-length encode the zeros. Q x 4 gives longer zero runs and fewer RLE symbols than Q x 1. The zigzag scan itself is lossless; it only reorders.
JPEG summarySearch on YouTube
| Stage | What happens | Lossy? |
|---|---|---|
| 1. Colour space conversion | RGB to Y, Cb, Cr | No (reversible) |
| 2. Chroma downsampling | Cb, Cr averaged per 2 by 2 block | Yes |
| 3. DCT | 8 by 8 blocks, level shift, spatial to frequency | No |
| 4. Quantisation | Divide by table and round; high frequencies become 0 | Yes (the main loss) |
| 5. Zigzag, RLE, Huffman | Group zeros, count runs, entropy code | No |
A common mistake: "the DCT compresses the image". It does not. The DCT only re-expresses the 64 pixels as 64 coefficients. Compression starts at quantisation.
Mid Sem 2025, Q3: two JPEGs look the same but differ in size. Reasons: different quality settings (quantisation tables), different chroma subsampling, and different image content. Busy, textured images keep more non-zero high-frequency coefficients, so their RLE and Huffman output is longer.
Lossless compression: PNGSearch on YouTube
PNG (Portable Network Graphics) is a raster graphics format with lossless compression. It suits graphics images.
A lossless codec must:
- preserve 100% of the original information;
- recover the exact original file after decompression;
- compress without losing any data.
Key principle: compression is achieved by exploiting redundancy, not by removing information.
The deck builds PNG step by step through three example images. Each shows why one technique alone is not enough.
Image 1: Huffman aloneSearch on YouTube
A 64-pixel image has only four values. About 40% of the pixels are 251.
| Pixel value | 251 | 92 | 147 | 252 |
|---|---|---|---|---|
| Count | 25 | 21 | 14 | 4 |
| Huffman code length (bits) | 1 | 2 | 3 | 3 |
Bits per pixel with Huffman.
Step 1. Fixed length: 4 unique values need 2 bits (00, 01, 10, 11), so 2 bits per pixel.
Step 2. Huffman: bits.
Step 3. bits per pixel.
Image 2: Huffman is not enough, use RLESearch on YouTube
Huffman looks only at how often values occur. It ignores where they occur. Images have high spatial redundancy: neighbouring pixels are often equal. RLE exploits this:
| Original row | RLE |
|---|---|
| 95 95 95 95 45 48 46 78 | 4 x 95, 45, 48, 46, 78 |
| 126 128 255 255 255 134 123 12 | 126, 128, 3 x 255, 134, 123, 12 |
| 120 120 120 120 120 120 120 134 | 7 x 120, 134 |
| 123 93 93 93 93 93 93 12 | 123, 6 x 93, 12 |
Image 3: RLE fails on repeated sequencesSearch on YouTube
RLE is inefficient for natural images when the redundancy is a repeated sequence, not a repeated value. Image 3 repeats 200 50 500 20 along rows. No two neighbours are equal, so RLE gains nothing.
LZSS solves this with back-referencing: "copy 4 values from 4 positions back". Its effectiveness depends on the buffer size and the amount of redundancy. For PNG the sliding window is 32 KB and the look-ahead is 258 bytes.
Problem again: gradientsSearch on YouTube
LZSS does not work well when pixels have very low redundancy. The slide's 8 by 8 gradient has values 0, 4, 8, ..., 252, all different.
Solution: transformation (filtering). Replace each pixel with its difference from the left neighbour. Each row then becomes its first value followed by seven 4s, for example 32 4 4 4 4 4 4 4. Filtering introduces redundancy, so LZSS can now compress well.
Filtering in PNGSearch on YouTube
PNG has five basic filter types: None, Sub, Up, Average, Paeth. Let X be the current pixel, L (A) the left pixel, U (B) the pixel above, and UL (C) the upper-left pixel. A missing neighbour (first row or first column) counts as 0.
Paeth predictor : compute . Then compute , , . Take the neighbour with the smallest distance as . On a tie, prefer L, then U, then UL.
Never Say Under Any Pressure: None, Sub, Up, Average, Paeth (filter types 0 to 4).
Slide examples (one row of 8 pixels)Search on YouTube
| Filter | Row above | Current row | Filtered output |
|---|---|---|---|
| None | none | 132 138 139 135 136 138 132 127 | 132 138 139 135 136 138 132 127 |
| Sub | none | 127 128 139 134 139 133 136 129 | 127 1 11 251 5 250 3 249 |
| Up | 127 128 139 134 139 133 136 129 | 131 132 129 131 138 139 137 139 | 4 4 246 253 255 6 1 10 |
| Average | 131 132 129 131 138 139 137 139 | 129 131 134 134 136 128 134 127 | 63 0 4 1 0 246 1 246 |
| Paeth | 133 136 136 134 133 136 128 127 | 139 128 135 135 130 137 136 135 | 6 245 7 1 252 4 8 255 |
Checking the Sub and Paeth rows.
Step 1. Sub, fourth pixel: , and . Negative residuals wrap around.
Step 2. Paeth, first pixel: L = 0, U = 133, UL = 0. . Distances: , , . Predict U: .
Step 3. Paeth, second pixel: L = 139, U = 136, UL = 133. . Distances 3, 6, 9. Predict L: .
Step 4. Paeth, fifth pixel: L = 135, U = 133, UL = 134. . Distances 1, 1, 0. Predict UL: .
The Average row on the slide rounds the half up (first pixel: , so ). The PNG standard uses the floor, which gives 64. Use the floor unless the question says otherwise, and state it.
Another example (two rows)Search on YouTube
Input rows 52 55 61 66 70 and 63 59 55 90 109. The slide filters the last four pixels of row 2:
| Filter | Row 2 residuals |
|---|---|
| None | 59 55 90 109 |
| Sub (X - A) | -4 -4 35 19 |
| Up (X - B) | 4 -6 24 39 |
| Average | 0 -5 30 29 |
| Paeth | -4 -6 29 19 (computed) |
The slide prints the Paeth row as -4 -4 35 19, the same as Sub. Working it out gives -4 -6 29 19. Example: for 55, L = 59, U = 61, UL = 55, so ; distances 6, 4, 10; predict U = 61; residual .
Exp 14: PNG filtering in Python. Input matrix rows 50 52 54 56 58 60, 51 53 55 57 59 61, 52 54 56 58 60 62. Missing neighbours are 0.
- Sub: every row becomes its first value, then 2 2 2 2 2.
- Up: row 1 unchanged; rows 2 and 3 become all 1s.
- Average: rows 2 and 3 become 26 or 27, then 2 2 2 2 2.
- Paeth: row 1 like Sub (50, then 2s), rows 2 and 3 all 1s.
Paeth gives the most repetitive data here. Rule of thumb from the lab: rows smooth horizontally favour Sub; rows similar vertically favour Up. Running zlib (DEFLATE) on each output confirms that more repetition gives a smaller file.
PNG encoding and decodingSearch on YouTube
LZSS followed by Huffman coding is the DEFLATE algorithm (as used by zlib). Filtering makes the data repetitive, LZSS removes repeated sequences, and Huffman removes the remaining frequency redundancy. Every step is reversible, so PNG is lossless.
| Property | JPEG | PNG |
|---|---|---|
| Compression | Lossy | Lossless |
| Core steps | YCbCr, downsample, DCT, quantise, RLE, Huffman | Filter, LZSS, Huffman |
| Best for | Photographs | Graphics, text, line art, archival |
| Where the loss is | Downsampling and quantisation | None |
Quick self-testSearch on YouTube
List the seven stages of image acquisition in a camera.
Lens, aperture, shutter, colour filter array, sensor, processing unit, storage.
Why does the Bayer filter have twice as many green filters as red or blue?
The human eye is most sensitive to green light. The ratio R : G : B = 1 : 2 : 1 gives 25% red, 50% green and 25% blue.
What is demosaicing?
Each sensor pixel records only one colour. Demosaicing interpolates the two missing colours at each pixel from neighbouring pixels to produce a full-colour image.
Name the five stages of JPEG and say which are lossy.
Colour space conversion, chroma downsampling, DCT, quantisation, RLE and Huffman coding. Downsampling and quantisation are lossy; quantisation is the main loss.
Why does JPEG subtract 128 before the DCT?
To centre pixel values on zero (range -128 to 127). This suits the DCT, simplifies the cosine arithmetic and removes the constant bias.
Quantise a DCT coefficient of -57 with table value 4.
, which rounds to .
Why does JPEG read the block in zigzag order?
Zigzag goes from low to high frequency. After quantisation most high-frequency values are zero, so the zigzag order puts them in one long run at the end, which RLE codes in a single symbol.
Apply the Sub filter to the row 240 244 248 252 0 4.
240, 4, 4, 4, 4, 4. The step wraps to .
Why does PNG filter before LZSS?
Smooth images and gradients have few exact repeats. Filtering turns them into small, repeated residuals, which LZSS and Huffman compress well. Filtering is reversible, so nothing is lost.
What are PNG's LZSS window and look-ahead sizes?
Sliding window 32 KB, look-ahead 258 bytes.
- Camera chain: lens, aperture, shutter, CFA, sensor, processing, storage; shutter speed controls motion blur.
- Bayer CFA is 25% R, 50% G, 25% B; demosaicing fills the missing colours.
- JPEG: RGB to YCbCr, 2 by 2 chroma averaging (1 : 1/4 : 1/4), level shift and 8 by 8 DCT, quantisation, zigzag, RLE, Huffman.
- Only downsampling and quantisation lose data; quantisation is the quality knob.
- PNG is lossless: Huffman uses frequency, RLE uses runs, LZSS uses repeated sequences.
- Filters None, Sub, Up, Average and Paeth turn pixels into small residuals mod 256.
- Paeth predicts with and picks the nearest neighbour.
- PNG pipeline: filter, LZSS, Huffman (DEFLATE); the decoder reverses each step.