Text Processing with sed and awk

Why text processing matters

Most things on Linux are text: logs, configuration files, CSV files and the output of commands. If you can reshape text quickly, you can answer questions that would take a very long time by hand. This lesson covers the main tools for that: sed, awk, sort, uniq and cut.

Create a sample file to practise on:

printf "alice 30 london\nbob 25 paris\ncarol 35 london\ndave 28 berlin\n" > people.txt

Each line has three fields separated by spaces: a name, an age and a city.

sed: find and replace

sed is the stream editor. It reads text line by line, changes it and prints the result. Its most common job is substitution, written s/old/new/.

sed 's/london/LONDON/' people.txt
sed 's/a/A/g' people.txt

• The first command replaces the first london on each line.
• The g flag at the end means "global": replace every match on the line, not only the first.

Put the sed command in single quotes so the shell does not change it.

sed options

sed -i 's/paris/rome/' people.txt
sed -i.bak 's/paris/rome/' people.txt
sed -n '2,3p' people.txt
sed '/bob/d' people.txt

• Without -i, sed only prints the result and the file stays unchanged. This lets you test safely.
• -i edits the file in place, which changes the file permanently.
• -i.bak edits in place and keeps a backup copy named people.txt.bak.
• -n '2,3p' prints only lines 2 to 3. -n turns off the normal printing, and p prints chosen lines.
• '/bob/d' deletes lines that contain bob.

awk: work with columns

awk splits each line into fields at whitespace. The fields are called $1, $2, $3 and so on, and $0 is the whole line.

awk '{print $1}' people.txt
awk '{print $1, $3}' people.txt

The first command prints only the first column, the names. The second prints the names and cities. The part inside the braces is the action, what to do with each line.

awk conditions and calculations

The general form is awk 'condition {action}'. If you leave out the condition, the action runs on every line. If you leave out the action, matching lines are printed.

awk '$2 > 28' people.txt
awk '$3 == "london" {print $1}' people.txt
awk '{sum += $2} END {print sum}' people.txt
awk '{sum += $2} END {print sum/NR}' people.txt
awk -F, '{print $2}' data.csv

• $2 > 28 prints lines where the age is more than 28.
• $3 == "london" {print $1} prints the names of people in London.
• {sum += $2} adds the age to a running total on every line. END {print sum} runs once after the last line and prints the total.
• NR is the number of rows read so far, so sum/NR is the average.
• -F, uses a comma as the field separator, which is needed for CSV files.

sort and uniq

sort people.txt
sort -n -k2 people.txt
sort -r people.txt
sort -u names.txt
uniq -c sorted.txt

• sort orders lines alphabetically. -n compares numbers, -k2 sorts by column 2, -r reverses and -u removes duplicates.
• uniq removes repeated lines, but only when they are next to each other. -c counts how many times each line repeats.

Because uniq only sees neighbouring duplicates, you almost always run sort first so identical lines end up together.

cut: take columns quickly

cut -d' ' -f1 people.txt
cut -d, -f2,4 data.csv
cut -c1-3 people.txt

• -d sets the delimiter, the character that separates fields.
• -f chooses which fields to keep. -f2,4 keeps fields 2 and 4.
• -c1-3 keeps the first three characters of each line.

cut is shorter than awk for simple column extraction. cut -d, -f2 data.csv prints the second comma-separated field.

Putting it together

How many people live in each city?

cut -d' ' -f3 people.txt | sort | uniq -c

Step by step: pick the city column, sort so equal cities are together, then count each group.

A longer example: for people older than 26, list their cities and count them from most to least common.

awk '$2 > 26 {print $3}' people.txt | sort | uniq -c | sort -rn