Columns with awk and cut

Why this matters

Many useful files are tables with one record per line. A sales export has one sale per line. A web server log has one request per line. Each line has columns, and you often need only one or two of them: the region, the number of units, the status code.

You could open the file in a spreadsheet, but that gets slow with thousands of lines, and it is hard to repeat the same steps next week. Two small tools cover most column work in the terminal. cut takes columns out. awk takes columns out and can also compute with them. This lesson uses both on two real-looking files and shows where each one gives a wrong answer without any error message.

Regular expressions, the pattern language used to search text, are covered in the neighbouring lesson on grep and sed. Here the patterns are simple: a separator character and a column number.

Lines, fields and separators

A line is the text up to a newline character. A field is one column inside a line. A separator is the character that marks where one field ends and the next begins.

Take this line from a CSV file (comma-separated values, a plain text table where commas separate the columns):

2026-01-03,north,widget,4,2.50

Here the separator is the comma. The five fields are:

Field numbers start at 1. Keep that in mind, because most mistakes with these tools come from an off-by-one error in the field number.

A web server log uses spaces instead of commas:

10.0.0.5 GET /home 200 1034 0.120

Here the separator is a space. Field 1 is the client address, field 2 the HTTP method, field 3 the path, field 4 the status code, field 5 the size in bytes and field 6 the time in seconds.

The example files

Make a scratch folder so the commands in this lesson do not touch your real projects. Then create the two files.

$ mkdir -p ~/scratch/columns && cd ~/scratch/columns
$ cat > sales.csv << 'EOF'
date,region,product,units,price
2026-01-03,north,widget,4,2.50
2026-01-03,south,gadget,1,12.00
2026-01-04,north,gadget,2,12.00
2026-01-05,east,widget,10,2.50
2026-01-06,south,widget,3,2.50
2026-01-07,"north, coast",gadget,3,12.00
EOF
$ cat > access.log << 'EOF'
10.0.0.5 GET /home 200 1034 0.120
10.0.0.7 POST /login 401 88 0.045
10.0.0.5 GET /products 200 5120 0.210
10.0.0.9 GET /home 500 310 1.502
10.0.0.7 GET /home 200 1034 0.098
EOF

The << 'EOF' part is a here-document: everything up to the line EOF is written into the file. The quotes around EOF stop the shell from changing $ or other characters inside the text.

Look at the sales file. The last row has a quoted field, "north, coast". The quotes mean that the comma inside them is part of the region name and not a separator. Keep this row in mind. It causes a real problem later in the lesson.

$ cat sales.csv
date,region,product,units,price
2026-01-03,north,widget,4,2.50
2026-01-03,south,gadget,1,12.00
2026-01-04,north,gadget,2,12.00
2026-01-05,east,widget,10,2.50
2026-01-06,south,widget,3,2.50
2026-01-07,"north, coast",gadget,3,12.00

Pulling columns with cut

cut does one job: it prints the fields or characters you name. It has two options that matter for this lesson.

• -d sets the delimiter, the separator character. Without it, cut uses a tab.
• -f names the fields to keep. You can list them with commas (-f2,4) or give a range (-f1-3).

Print the region (field 2) and the units (field 4) from the sales file:

$ cut -d, -f2,4 sales.csv
region,units
north,4
south,1
north,2
east,10
south,3
"north,gadget

The header and the first five rows are correct. The last row is wrong. cut split at the comma inside the quotes, so field 2 came out as "north and field 4 as gadget. cut does not understand quoting. It counts every comma as a separator. We come back to this after the other tools are introduced.

A range keeps the first three columns:

$ cut -d, -f1-3 sales.csv | head -n 3
date,region,product
2026-01-03,north,widget
2026-01-03,south,gadget

head -n 3 keeps only the first three lines of the output. The pipe | sends the output of cut into head. That is the same pipe you met in the lesson on flags and pipes.

Now the log file. The fields are separated by single spaces, so the delimiter is a space. Note the quotes around the space: the shell must see one space, not a bare space that it would split on.

$ cut -d' ' -f3 access.log
/home
/login
/products
/home
/home

cut also works on character positions with -c. It is useful for fixed-width files, where every column has the same number of characters. Use it only when you know the positions are fixed. For example, cut -c1-10 sales.csv keeps the first ten characters of each line.

Pulling columns with awk

awk is a small programming language built for line-by-line text. You give it a program: a set of rules. Each rule has a pattern that decides which lines it applies to, and an action in braces that runs on each matching line. A rule with no pattern applies to every line.

pattern { action }

The braces are part of the awk language. Put the whole program in single quotes on the command line so the shell does not try to read the $ signs.

The simplest awk program prints every line:

$ awk '{print $0}' access.log
10.0.0.5 GET /home 200 1034 0.120
10.0.0.7 POST /login 401 88 0.045
10.0.0.5 GET /products 200 5120 0.210
10.0.0.9 GET /home 500 310 1.502
10.0.0.7 GET /home 200 1034 0.098

Inside awk, $0 is the whole line, $1 is field 1, $2 is field 2, and so on. NF is the number of fields in the current line. NR is the number of the current line, counting from 1.

By default awk splits fields on runs of spaces and tabs. You do not need to name a delimiter for the log file:

$ awk '{print $3, $4}' access.log
/home 200
/login 401
/products 200
/home 500
/home 200

For the CSV file, set the separator with -F:

$ awk -F, '{print $2, $4}' sales.csv
region units
north 4
south 1
north 2
east 10
south 3
"north gadget

Again the last row is split inside its quotes. The output looks almost the same as cut, and the wrong row is wrong in the same way.

A pattern picks the lines

A pattern can be any test. Only lines where the test is true get the action.

Show only the log lines where the status code is 500 or higher:

$ awk '$4 >= 500' access.log
10.0.0.9 GET /home 500 310 1.502

Show the sales rows for the north region. Here the pattern compares field 2 with a string in double quotes:

$ awk -F, '$2=="north"' sales.csv
2026-01-03,north,widget,4,2.50
2026-01-04,north,gadget,2,12.00

A pattern with no action prints the whole matching line. That is why there is no {print} in these examples.

BEGIN and END

awk has two special rules that run once: BEGIN before the first line and END after the last line. They are the place for setup and for totals.

$ awk 'BEGIN {print "start"} {n++} END {print n, "lines"}' sales.csv
start
7 lines

The variable n starts at zero, and awk sets new variables to zero when they are used as numbers, so n++ works without a setup step. The 7 lines count includes the header line.

Adding up a column

This is the most common reason to use awk. Here is the total of the units column for the sales file:

$ awk -F, 'NR>1 {sum += $4} END {print sum}' sales.csv
20

Read the program piece by piece:

• NR>1 is a pattern. It is true for every line except the first, so the header is skipped. Without it, awk would try to add the text units as a number. Awk turns text that is not a number into 0, so the header would not break the sum, but it is still good practice to skip it on purpose.
• {sum += $4} adds field 4 to the variable sum. The += operator means "add this to the current value".
• END {print sum} runs once at the end and prints the total.

The answer, 20, is wrong. The units in the first five rows add up to 20 (4 + 1 + 2 + 10 + 3). The sixth row has 3 units, so the correct total is 23. The sixth row was split at its quoted comma, so awk read gadget as field 4 and tried to add it as a number. The result is 0 for that row, and nothing reported an error. This is the main danger of these tools, and the section on the CSV trap explains how to avoid it.

Multiplying two columns

Revenue is units times price. Multiply the two fields inside the action:

$ awk -F, 'NR>1 {total += $4*$5} END {print total}' sales.csv
78.5

Again the sixth row adds 0. Rows one to five give 4 × 2.50 + 1 × 12.00 + 2 × 12.00 + 10 × 2.50 + 3 × 2.50 = 78.5. The correct answer with the sixth row is 114.5. Checking a result by hand for a few rows is a good habit, and it is how the wrong number was spotted here.

Totals for each group

To total units for each product, use an array: a named set of boxes, each with a key. Here the key is the product name in field 3.

$ awk -F, 'NR>1 {units[$3] += $4} END {for (p in units) print p, units[p]}' sales.csv | sort
 coast" 0
gadget 3
widget 17

The for (p in units) loop visits every key. Awk does not promise an order for those keys, so the output is piped into sort to get a stable order. The line coast" 0 is the wrong row again: the product column of that row became coast", with a space before it. The widget total of 17 (4 + 10 + 3) is correct, and so is the gadget total of 3 (1 + 2), which is why the other lines look reasonable.

A log example

Log files are the other common job for these tools. Here are three questions about the access log, each answered with one short command.

How many requests returned a server error? Use the pattern from before and count the lines:

$ awk '$4 >= 500' access.log | wc -l
1

wc -l counts lines. The command wc stands for word count, and -l asks for lines.

What is the average response time? Field 6 holds the time in seconds. Add it up over every line and divide by the count:

$ awk '{sum += $6; n++} END {printf "%.3f\n", sum/n}' access.log
0.395

printf prints a formatted number. The text %.3f means "a decimal number with three digits after the point". Plain print would show the full number with all its digits.

How many requests came from each client address? Cut out field 1, sort it so equal values sit next to each other, then count the groups:

$ cut -d' ' -f1 access.log | sort | uniq -c | sort -rn
      2 10.0.0.7
      2 10.0.0.5
      1 10.0.0.9

uniq -c counts runs of equal lines, and that is why the sort comes first. sort -rn sorts by number, largest first. This pipeline uses cut for the column and the common tools for counting, and needs no awk program at all.

The CSV trap

A CSV file can put a comma inside a field if the field is wrapped in double quotes. cut and plain awk -F, cannot see the quotes. They split at every comma. The sales file has one such row, and the earlier commands showed the damage: a wrong sum, a wrong total and a broken product name. Nothing printed a warning. The output looked like valid data.

There are three ways out.

Use a CSV-aware mode. GNU awk (gawk) version 5.3 on this machine has a --csv option that understands quotes:

$ gawk --csv 'NR>1 {print $2, $3, $4}' sales.csv
north widget 4
south gadget 1
north gadget 2
east widget 10
south widget 3
north, coast gadget 3

The last line is now correct. The region is north, coast and it stays one field. Totals with the same tool:

$ gawk --csv 'NR>1 {units += $4; rev += $4*$5} END {print units, rev}' sales.csv
23 114.5

Both numbers are now correct: 23 units and 114.5 in revenue. The --csv option is not in every awk. The smaller mawk, which is also common on Linux systems, rejects it:

$ mawk --csv 'NR>1 {print $2}' sales.csv
mawk: not an option: --csv

Before you rely on a script on another machine, check which awk it will run with, by typing awk --version or gawk --version.

Use Python's csv module. Python ships a CSV reader that handles quotes. Here is a short program that prints the region and units of each row. Save it as read_sales.py in the same folder:

import csv
import sys
 
with open(sys.argv[1], newline="") as f:
    reader = csv.DictReader(f)
    for row in reader:
        print(row["region"], row["units"])

Run it with the file name as an argument:

$ python3 -I read_sales.py sales.csv
north 4
south 1
north 2
east 10
south 3
north, coast 3

The option -I tells Python not to look in the script's own folder first for modules, so a file such as json.py next to the script is ignored. That is a safety habit for scripts you did not write. The csv module returns each row as a dictionary keyed by the column names from the header, so the code uses row["region"] and not a field number. That is safer than field numbers when a column moves.

Use a spreadsheet or a data tool when the file is big, or when several people need to check the numbers. These are outside this lesson.

Common mistakes

Each example below shows a real error message or a real wrong result.

Mistake 1: awk program with a missing brace. The action in braces was not closed:

$ awk -F, '{print $1' sales.csv
awk: cmd. line:1: {print $1
awk: cmd. line:1:          ^ unexpected newline or end of string

Awk points at the place where it stopped reading. Here the closing } is missing. Check the brace count first when awk reports an unexpected end.

Mistake 2: forgetting the delimiter in cut. Without -d, cut splits on tabs. A comma file has no tabs, so every line comes back whole:

$ cut -f2 sales.csv | head -n 2
date,region,product,units,price
2026-01-03,north,widget,4,2.50

Nothing is an error here, which is why this mistake is easy to miss. If the output looks like the whole file, check for -d.

Mistake 3: field numbers start at 1. Field 0 is the whole line, and cut does not accept it:

$ cut -d, -f0 sales.csv
cut: range '0' was invalid: fields and positions are numbered from 1

Other versions of cut can word this message differently. In awk, $0 is the whole line, so awk uses 0 for it. Do not mix the two up.

Mistake 4: two spaces where one was expected. The log file uses single spaces, so cut works. If a tool pads the columns with several spaces, cut -d' ' gives an empty field between the pads. Here is a line with two spaces after the address:

$ printf '%s\n' '10.0.0.5  GET /home 200' | cut -d' ' -f3
GET

Field 3 is GET, not /home, because cut counts the empty field between the two spaces. Awk treats a run of spaces as one separator, so the same line works there:

$ printf '%s\n' '10.0.0.5  GET /home 200' | awk '{print $3}'
/home

When a file may have uneven spacing, use awk or squeeze the spaces first.

Mistake 5: a missing file. Both tools report the file name and stop:

$ cut -d, -f1 nofile.csv
cut: nofile.csv: No such file or directory
$ awk -F, '{print $1}' nofile.csv
awk: fatal: cannot open file `nofile.csv' for reading: No such file or directory

The names are in the message, so check the spelling and the folder you are in with pwd.

Mistake 6: trusting the total. The worst mistake in this lesson is not an error message at all. A sum that looks plausible can be wrong because one row was split in the wrong place. When a sum matters, check the row count first, using awk 'END {print NR}' sales.csv, and compare the total with a second tool such as gawk --csv or Python.

When awk is worth learning

Here is a plain guide for the choice between the tools.

• Use cut when you need a few fixed columns from a file with a single, simple separator, and no arithmetic. It is fast to type and easy to read. It breaks as soon as a field can contain the separator.
• Use awk when you need a condition on a column, arithmetic, totals for each group, or a few steps in one pass over the file. Its programs are short. It is also installed on almost every Unix-like system, including servers where you cannot install anything.
• Use a real CSV parser (Python's csv module, or a data tool) when the file has quoted fields, when the numbers feed a report someone else reads, or when the work grows past a few lines of awk.

What you gain by learning awk is the ability to answer a question about a text file in one line, without opening an editor. What it costs is a new small language, with its own rules about fields, braces and quoting. The main risk is the one shown in this lesson: a quiet wrong answer from data you did not expect. The habit that protects you is simple. Check the row count, check one result by hand, and run the same question through a second tool when the answer matters.

Practice

Use the sales and log files from this lesson. Each exercise has an answer under it. Try the exercise first.

Exercise 1. Print the date and the units for every sale in the south region. Use awk.

Show the answer

$ awk -F, '$2=="south" {print $1, $4}' sales.csv
2026-01-03 1
2026-01-06 3

The pattern $2=="south" keeps only the south rows. The action prints fields 1 and 4.

Exercise 2. Count the requests in the access log that returned status 500.

Show the answer

$ awk '$4==500' access.log | wc -l
1

The count is 1, from the line 10.0.0.9 GET /home 500 310 1.502. The same answer comes from awk '$4==500 {n++} END {print n+0}' access.log. The +0 makes the output print 0 instead of an empty line when nothing matches.

Exercise 3. Use cut to print the client address and the path of each request in the access log, separated by one space.

Show the answer

$ cut -d' ' -f1,3 access.log
10.0.0.5 /home
10.0.0.7 /login
10.0.0.5 /products
10.0.0.9 /home
10.0.0.7 /home

cut keeps the original separator between the fields it prints, so the output still uses a single space.

Exercise 4. Find the highest response time in the access log.

Show the answer

$ awk 'NR==1 || $6 > max {max = $6} END {print max}' access.log
1.502

The first line sets max to its own value. After that, any larger time replaces it. Awk compares these fields as numbers because they look like numbers. A shorter program, sort -k6 -n access.log | tail -n 1, gives the same line.

Exercise 5. Find the total units for the north region. The quoted row "north, coast" should count as its own region, not as part of north. Use gawk with the CSV option, and state what the answer is.

Show the answer

$ gawk --csv 'NR>1 && $2=="north" {u += $4} END {print u}' sales.csv
6

The north rows are the first and third sales, with 4 and 2 units. The quoted row has the region north, coast, which does not equal north, so it is not counted. The total is 6.

Summary

• A line is a row, a field is a column, and field numbers start at 1.
• cut -d<separator> -f<fields> picks columns. Its default separator is a tab, so -d is needed for commas and spaces.
• awk runs pattern { action } on each line. Use -F for the separator, $1 for field 1, $0 for the whole line, and NR for the line number.
• END { ... } runs once after the last line, which is where totals are printed.
• Plain cut and awk -F, split inside quoted CSV fields and give wrong numbers without any error. Use gawk --csv or Python's csv module for quoted data, and check the row count when a total matters.