Sound and video as samples

Why this matters

A phone records a voice memo in a few seconds, and a camera records a minute of video in a few more. Both end up as files on disk, and both files are made of numbers. Nothing in the file is a sound or a picture in the way you experience one. A player reads the numbers and rebuilds the sound or the picture from them.

Three settings decide how big those numbers get: how often the sound is measured, how precisely each measurement is written down, and how many pictures are taken per second. Set one of them too low and the recording sounds thin or the video looks jerky. Set them too high and the file fills a phone or takes too long to upload. An app that records or streams media makes these choices in code, so an engineer needs to know what each one costs.

The binary and data lesson covers how numbers become bits. A bit is a single 0 or 1. A byte is eight bits. This lesson uses both words without a full introduction.

The idea: a recording is a list of numbers

A sound is a moving pressure

Sound is air pressure that rises and falls very fast. When you speak, the air near your mouth is pushed and pulled thousands of times per second. A microphone turns those changes into an electric signal, a voltage that rises and falls in the same shape. A voltage is a measure of electric push.

A computer cannot store a smooth curve. It stores numbers. So it measures the height of the signal at fixed moments, writes each height down as a number, and drops whatever happens between the moments. Each measurement is called a sample. A sample is one number that records the height at one moment.

A video is a stack of still pictures

A video is a fast series of still pictures. Each still picture is a frame. Show frames one after another quickly enough and the eye reads them as motion.

A camera measures each frame the same way a microphone measures sound. It records the colour of many small dots at fixed positions. Each dot is a pixel, and a frame is a grid of pixels. A frame that is 1920 pixels wide and 1080 pixels tall has 1920 times 1080 pixels, which is 2,073,600 pixels.

A worked example

Take one second of a sound that goes up and down exactly once per second. Measure its height eight times, at equal steps through that second. The heights follow the sine wave, a standard wave shape. The program below computes them. Each number it prints is one sample.

Output:

[0.0, 0.71, 1.0, 0.71, 0.0, -0.71, -1.0, -0.71]

The file would store these eight numbers in order. A player that knows the rate is eight per second draws a smooth curve through the points and sends it to a speaker. Eight samples per second is far too few for real sound, which is the next topic.

Sample rate: how often the sound is measured

The unit

The sample rate is the number of samples taken per second. It is measured in hertz, written Hz, which means "times per second". A rate of 44100 Hz means 44,100 samples per second.

Recordings often have more than one stream of sound. Headphones play a left stream and a right stream. Each stream is a channel. A mono recording has one channel and a stereo recording has two. The sample rate applies to each channel separately. A stereo file at 44100 Hz therefore holds 88,200 numbers per second.

The highest pitch a rate can keep

A frequency is the number of up-and-down cycles a sound makes per second. A sound at 440 Hz cycles 440 times per second, and that is the pitch of the A that orchestras tune to.

A sample rate sets a ceiling on the frequencies that can be kept. The rule is short: the highest frequency you can keep is half the sample rate. To see why, picture drawing a wave with dots. You need at least two dots per cycle, one on the way up and one on the way down. With fewer dots, two different waves can leave the same pattern of dots.

Most young people hear up to about 20,000 Hz, and the top of that range fades with age. A rate of 44100 Hz keeps frequencies up to 22,050 Hz, which is above what most people hear. That is one reason CD audio uses 44100 Hz.

What happens when the rate is too low

The next program makes a 6000 Hz tone and samples it at 8000 Hz. Old telephone networks sampled speech at about that rate. Then it prints the samples of a 2000 Hz tone, with the sign flipped, for comparison.

Output:

[0.0, -1.0, 0.0, 1.0, -0.0, -1.0, 0.0, 1.0]
[-0.0, -1.0, -0.0, 1.0, 0.0, -1.0, -0.0, 1.0]

The two lists match. The 6000 Hz tone and the 2000 Hz tone leave identical numbers behind, so the file cannot tell them apart. Zeros show as -0.0 or 0.0, and both are zero.

When a sound above half the sample rate is recorded, it comes back as a false, lower sound. This effect is called aliasing. A 6000 Hz whistle recorded at 8000 Hz plays back as a 2000 Hz tone, which is a different pitch. Good recording devices remove frequencies above half the rate before they sample, which keeps aliasing out of normal recordings.

How the sample rate changes file size

Each extra sample per second adds one more number to the file, so the size grows in a straight line with the rate. The table shows real file sizes for one second of 16-bit mono sound, written with Python's wave module. Each size includes a small header, the block of information at the start of the file that says the channels, the rate and the length.

Take the 44100 Hz row. One second has 44100 samples, and each 16-bit sample takes 2 bytes, so the sound takes 88200 bytes. The file is 88244 bytes, so the header takes the other 44 bytes.

Bit depth: how precisely each sample is written down

Levels

The bit depth is the number of bits used to store each sample. More bits give more possible values for the height. One bit gives 2 values. Two bits give 4. The number of values is 2 multiplied by itself as many times as the bit depth. "2 to the power of 8" means 2 times 2 times 2 times 2 times 2 times 2 times 2 times 2, which is 256.

So 8 bits give 256 levels, 16 bits give 65,536 levels, and 24 bits give 16,777,216 levels. Each sample is stored as the nearest level to the true height.

In a WAV file, 8-bit samples take 1 byte, 16-bit samples take 2 bytes, and 24-bit samples take 3 bytes.

Rounding error

When the true height falls between two levels, the sample is stored as the closer one. The gap between the true height and the stored level is the rounding error. Fewer levels give a bigger error.

The next program stores the value 0.3 at four bit depths. The value is a height on a scale from minus 1 to 1, so 0.3 sits a little above the middle. Each output line has four numbers: the bit depth, the stored whole number, the value read back from it, and the error.

Output:

2 0 0.0 0.3
4 2 0.28571 0.01429
8 38 0.29921 0.00079
16 9830 0.3 0.0

This program uses levels from minus the top value to plus the top value, so it has one level fewer than the counts in the Levels section (2 bits give -1, 0 and 1). At 2 bits, the value is stored as 0 and the error is 0.3, which is most of the height. At 4 bits the error falls to about 0.014. At 16 bits the error is too small to see at five decimal places. A large error in every sample shows up as hiss or distortion in the sound.

How the bit depth changes file size

Each extra bit adds to the size of every sample, so the file grows with the bit depth. The table shows real sizes for one second of mono sound at 44100 Hz.

Each step adds 44100 bytes, which is one byte for each of the 44100 samples.

Channels and the size formula

Channels multiply the size too. One second of 16-bit, 44100 Hz sound takes 88244 bytes in mono and 176444 bytes in stereo. Stereo stores two numbers for every moment, so it takes twice the space.

Put the three sound settings together:

bytes per second = sample rate x (bit depth / 8) x channels
total bytes      = bytes per second x seconds + header

The header for these WAV files is 44 bytes. For a long recording it is a tiny part of the total, so you can often leave it out when you estimate.

Worked example one: a voice memo

A voice memo is recorded at 16000 Hz, with 16-bit samples, in mono, for 30 seconds.

1. The sample rate is 16000 samples per second.
2. Each sample takes 16 divided by 8, which is 2 bytes.
3. There is one channel.
4. Bytes per second: 16000 x 2 x 1 = 32000.
5. Total for 30 seconds: 32000 x 30 = 960000 bytes, which is 0.96 MB.

Here MB means one million bytes. A rate of 16000 Hz keeps frequencies up to 8000 Hz, which is enough for clear speech.

Worked example two: a CD track

A CD track is 44100 Hz, 16-bit, stereo, and one minute long. Run this to get the numbers:

Output:

176400
10584000

One second takes 176400 bytes, and one minute takes 10584000 bytes, which is about 10.6 MB. The // in the code means whole-number division, so bits // 8 gives 2 for 16 bits.

Worked example three: a studio file

A studio file is 3 minutes long, at 48000 Hz, with 24-bit samples, in stereo. The total is larger, and the calculation needs one more step: the time is in minutes and the formula needs seconds.

1. Convert the time: 3 minutes is 3 x 60 = 180 seconds.
2. Each sample takes 24 divided by 8, which is 3 bytes.
3. Bytes per second: 48000 x 3 x 2 = 288000.
4. Total: 288000 x 180 = 51840000 bytes, which is 51.84 MB.

Per second, the studio file takes 288000 bytes and the CD track takes 176400 bytes, so the studio file is about 1.6 times as large for each second of sound.

Frame rate: how often the picture is taken

Frames per second

The frame rate is the number of frames per second, written as fps. Film for cinemas has often used 24 fps. Higher rates show smoother motion, and each extra frame per second adds another full picture to every second of the file.

Colour numbers in a pixel

Each pixel stores three numbers: one for red, one for green and one for blue. Each of these colour numbers is written with 8 bits, so it runs from 0 to 255. Three numbers of 8 bits make 24 bits, which is 3 bytes per pixel. Those three numbers together give 16,777,216 colours.

A frame that is width pixels wide and height pixels tall takes width x height x 3 bytes. This is the raw size, before any compression.

Worked example: a small raw clip

A raw clip is a video with no compression. Here each frame is 320 pixels wide and 240 pixels tall. That is 320 times 240 pixels, which is 76800 pixels, and 76800 times 3 gives 230400 bytes per frame.

The raw bytes for one second of frames at three frame rates are:

The 24 fps file is 2.4 times the 10 fps file, because it has 2.4 times as many frames. The size follows the frame rate in a straight line, just like the sample rate does for sound.

Full HD numbers

Full HD is 1920 pixels by 1080 pixels. The program below computes the raw size of a full HD frame and of one second of video at three frame rates.

Output:

6220800
24 149299200 8.957952
30 186624000 11.19744
60 373248000 22.39488

One frame takes 6220800 bytes, which is about 6.2 MB. At 24 fps, one second takes 149299200 bytes, and one minute takes about 8.96 GB. Here GB means one billion bytes. Doubling the frame rate doubles the size.

Compression

Real video files are much smaller than these raw numbers. A codec is a program that compresses video when it is saved and decompresses it when it is played. A common trick is to store only the parts of each frame that changed since the last one. A still background then costs little after the first frame. The sizes in this lesson are the raw sizes, and this lesson does not measure how much a codec saves. The sample rate, bit depth and frame rate still set the raw size that the codec starts from.

Common mistakes

Bits per second and bytes are different units

Internet speeds are usually written in bits per second, such as 10 Mbps, which means 10 megabits per second. File sizes are written in bytes. Eight bits make one byte, so you divide by 8 to convert. A video that streams at 10 Mbps needs 10 divided by 8, which is 1.25 MB of download for each second. A student who divides by 1 instead of 8 gets an answer eight times too large.

A sample rate written in kilohertz

Some people write 44.1 when they mean 44100 Hz. The wave module does not complain about that value. It stores the whole-number part, so the file is written with the wrong rate:

Output:

read back rate: 44

No error appears, and the file is wrong. A player reads it as 44 samples per second, so the sound plays about a thousand times too slowly. The fix is to pass the rate in hertz, as 44100.

Negative numbers in an 8-bit file

In an 8-bit WAV file, each sample is an unsigned number from 0 to 255. Silence is 128, not 0. Writing a negative number into a byte fails:

data = bytes([-5])

The last line of the error reads:

ValueError: bytes must be in range(0, 256)

Add 128 to each signed sample before writing it, so the value fits in the byte.

A sample value too large for its width

A 16-bit sample is a signed number from -32768 to 32767. A value outside that range fails when packed:

import struct
 
packed = struct.pack("<h", 40000)

The error reads:

struct.error: 'h' format requires -32768 <= number <= 32767

The usual cause is a sample that was never scaled. Samples between minus 1 and 1 are multiplied by the largest value the bit depth allows, which is 32767 for 16 bits, before they are written.

Opening a file that is not a WAV file

Opening any existing file with the WAV reader fails if the file does not start with the WAV marker:

import wave
 
wave.open("mistakes.py", "rb")

The error reads:

wave.Error: file does not start with RIFF id

A file's name does not decide its format. Check the first bytes or the source that produced the file before you read it.

Practice

Exercise 1

A voice memo is recorded at 16000 Hz, with 16-bit samples, in mono, for 30 seconds. How many bytes of sound does it hold, not counting the header?

Show the answer

960000 bytes. The calculation is 16000 x 2 x 1 x 30.

Exercise 2

The same voice memo is recorded at 8 bits instead of 16. How many bytes of sound does it hold now?

Show the answer

480000 bytes. The bit depth halves from 2 bytes per sample to 1 byte, so the total halves.

Exercise 3

A raw video is 1280 pixels wide and 720 pixels tall, runs at 30 fps, and lasts 10 seconds. How many bytes does one frame take, and how many bytes does the whole video take?

Show the answer

One frame takes 1280 x 720 x 3 = 2764800 bytes. One second takes 2764800 x 30 = 82944000 bytes. Ten seconds take 829440000 bytes, which is 829.44 MB.

Exercise 4

A recorder samples at 16000 Hz. What is the highest frequency it can keep without aliasing? What sample rate would you need to keep a tone at 12000 Hz?

Show the answer

The highest frequency is half the sample rate, so 8000 Hz. To keep 12000 Hz you need a rate of at least twice that, which is 24000 Hz.

Exercise 5

A clip is 5 seconds long, at 22050 Hz, with 16-bit samples, in stereo. What is its total file size, including a 44-byte header?

Show the answer

The sound takes 22050 x 2 x 2 x 5 = 441000 bytes. Adding the 44-byte header gives 441044 bytes.

Summary

• A sound or a video is a list of numbers measured at fixed moments.
• The sample rate sets the highest frequency that can be kept, which is half the rate, and the file size grows in a straight line with it.
• The bit depth sets how many levels each sample can take, and how many bytes each sample uses.
• Channels multiply the size of sound, and the frame rate, width, height and 3 colour bytes per pixel set the raw size of video.
• Compression makes real files much smaller, but the formulas here give the raw size that a codec starts from.