Embercode
✕
The TerminalLesson 6 of 14

60 min read

Module assignment: The Log Detective

The situation

You have joined a small online shop as its first developer. Last night the checkout page failed for some customers. Your manager sends you the web server's access log and asks three things:

  1. How many requests came in, and from how many different visitors?
  2. Which pages are visited most?
  3. Which requests failed with a server error, and can you keep those lines in a file for later?

Next week the same question will arrive with a different log file. So you will not answer by hand. You will write a script, a saved list of commands, that produces the same report for any log in the same format. Every skill from this module appears: navigating, creating files, pipes, grep, variables and permissions.

This is a real task. On-call engineers write exactly this kind of script on their first week.

The log format

Each line is one request. Fields are separated by single spaces:

IP TIMESTAMP METHOD PATH STATUS BYTES
FieldExampleMeaning
IP203.0.113.5The visitor's address.
TIMESTAMP2026-11-04T09:00:01When the request arrived.
METHODGETGET reads a page, POST sends a form.
PATH/checkoutThe page asked for.
STATUS500Result code: 2xx fine, 3xx redirect, 4xx the visitor's mistake, 5xx the server's fault.
BYTES120Size of the reply.

Remember the cutting tool from lesson four: cut -d' ' -f4 takes field 4 (the path), and -f5 takes the status.

Step 0: set up the workspace

Create a clean folder and the log file. Type this exactly. The here-document keeps every line in place.

$ cd ~/forge-lab
$ mkdir -p log-detective
$ cd log-detective
$ cat > access.log <<'EOF'
203.0.113.5 2026-11-04T09:00:01 GET /home 200 512
198.51.100.7 2026-11-04T09:00:03 GET /products 200 2048
203.0.113.5 2026-11-04T09:00:09 GET /products/42 200 1024
192.0.2.44 2026-11-04T09:01:15 POST /login 302 0
192.0.2.44 2026-11-04T09:01:16 GET /account 200 900
198.51.100.7 2026-11-04T09:02:02 GET /cart 200 640
203.0.113.5 2026-11-04T09:02:30 POST /checkout 500 120
198.51.100.9 2026-11-04T09:03:01 GET /home 200 512
198.51.100.9 2026-11-04T09:03:05 GET /missing 404 80
203.0.113.5 2026-11-04T09:03:40 POST /checkout 500 120
192.0.2.44 2026-11-04T09:04:11 GET /products 200 2048
198.51.100.7 2026-11-04T09:04:50 GET /home 200 512
198.51.100.9 2026-11-04T09:05:20 GET /old-page 404 80
203.0.113.77 2026-11-04T09:05:59 GET /products 200 2048
203.0.113.77 2026-11-04T09:06:10 POST /checkout 503 90
192.0.2.44 2026-11-04T09:06:45 GET /home 200 512
198.51.100.7 2026-11-04T09:07:30 GET /products 200 2048
203.0.113.5 2026-11-04T09:08:02 GET /home 200 512
198.51.100.9 2026-11-04T09:08:44 GET /cart 200 640
203.0.113.77 2026-11-04T09:09:00 GET /products 200 2048
EOF
$ wc -l access.log
20 access.log

If wc -l does not print 20, a line was lost when you pasted. Fix it before going on.

Before you write the script, check the expected answers independently. This Python program computes them from the same text. Having a second source of truth is a habit worth keeping: if your shell pipeline disagrees with it, one of the two has a bug.

Python · editable
total: 20
unique ips: 5
by status: [('200', 14), ('302', 1), ('404', 2), ('500', 2), ('503', 1)]
top path counts: [5, 5, 3]
5xx lines: 3

Keep those numbers: 20 requests, 5 visitors, 14 successes, 3 server errors.

The specification

Write a script named report.sh in the log-detective folder.

Usage: ./report.sh LOGFILE

Behaviour:

  1. If no argument is given, print usage: ./report.sh LOGFILE to standard error (not standard output) and exit with code 2.
  2. If the file does not exist, print error: no such file: NAME to standard error and exit with code 1.
  3. Otherwise print a report to standard output, in this order:
    • a heading line == Report for NAME ==
    • Total requests: N
    • Unique visitors: N (number of different IP addresses)
    • a blank line, the line Requests per status code (count, code):, then one line per status code, most frequent first
    • a blank line, the line Top K paths (count, path):, then the K most-requested paths, most frequent first
    • a blank line, then Server errors: N (saved in errors.txt)
  4. K defaults to 3, and can be changed with an environment variable: TOP_N=5 ./report.sh access.log lists five paths.
  5. Write every line with a status in the 500 to 599 range to a file errors.txt in the current folder, replacing any older copy. Set that file's permissions so that only you can read or write it.
  6. The script must be executable, and must start with a shebang line.
  7. The script must not modify the log file.

Expected output

For the log above, with no TOP_N set, the report is:

== Report for access.log ==
Total requests: 20
Unique visitors: 5
 
Requests per status code (count, code):
     14 200
      2 500
      2 404
      1 503
      1 302
 
Top 3 paths (count, path):
      5 /products
      5 /home
      3 /checkout
 
Server errors: 3 (saved in errors.txt)

Two details about tied counts. Where counts tie (500 and 404 both have 2, /products and /home both have 5), the order of the tied lines may differ on your machine, because it depends on how sort compares text in your locale setting. Either order is correct. Also, the width of the count column depends on uniq -c; do not try to match the spaces exactly.

errors.txt must hold exactly these three lines, and ls -l errors.txt must start with -rw-------:

203.0.113.5 2026-11-04T09:02:30 POST /checkout 500 120
203.0.113.5 2026-11-04T09:03:40 POST /checkout 500 120
203.0.113.77 2026-11-04T09:06:10 POST /checkout 503 90

Error behaviour

$ ./report.sh
usage: ./report.sh LOGFILE
$ echo $?
2
$ ./report.sh nope.log
error: no such file: nope.log
$ echo $?
1
$ ./report.sh nope.log 2> /dev/null
$ echo $?
1

The last command proves the message goes to standard error: with 2> /dev/null nothing is shown, but the exit code is still 1.

Requirements checklist

Copy this list into a note and tick the boxes as you verify each one.

  • report.sh exists in ~/forge-lab/log-detective and starts with #!/usr/bin/env bash
  • ls -l report.sh shows an x for the owner (it runs with ./report.sh)
  • Running with no argument prints the usage line to stderr and exits with 2
  • Running with a missing file prints the error to stderr and exits with 1
  • Total requests is 20 for the sample log
  • Unique visitors is 5 for the sample log
  • Status codes are listed with counts, highest first
  • The top paths section lists 3 paths by default
  • TOP_N=5 ./report.sh access.log lists 5 paths and the heading says Top 5 paths
  • errors.txt contains exactly the three 5xx lines
  • ls -l errors.txt starts with -rw-------
  • The log file is unchanged afterwards (wc -l access.log still says 20)
  • You tested at least one extra log of your own (see below)

How you will be graded

LevelWhat it looks like
60%The script runs and prints total requests and the status-code table correctly for the sample log. It may be missing the error handling, TOP_N, or the private errors.txt. Some commands may be hard-coded to access.log instead of using the argument.
80%Everything in the specification works for the sample log: correct report, errors.txt with the right lines and mode 600, usage and missing-file errors with the right exit codes and messages on stderr, and TOP_N honoured. Variables are quoted.
100%All of the above, plus: it works on a log with zero lines and on a path containing a space, the log is never modified, the script uses set -u or equivalent care with unset variables, you tested a second log of your own with expected numbers worked out beforehand, and you can explain each pipeline stage aloud. At least one extension below is done.

Hints

Try each step yourself for ten minutes before opening a hint. Each hint is hidden so you can choose how much help you want.

Hint 1: start with the pipelines, not the script

Do not open an editor yet. At the prompt, with access.log in the folder, find each answer with a one-line pipeline.

  • Total requests: wc -l < access.log
  • Unique visitors: take field 1 with cut -d' ' -f1, then sort -u, then count lines.
  • Status counts: take field 5, then sort | uniq -c | sort -rn.
  • Top paths: take field 4, then the same counting pipeline, then head -n 3.

When each pipeline prints what you expect (compare with the Python cross-check), only then copy it into a file.

Hint 2: the 5xx filter

The status is field 5, and a 5xx code is three digits starting with 5. grep -E ' 5[0-9][0-9] ' access.log matches the code surrounded by spaces. The spaces matter: without them, a byte count such as 5120 or a path containing 500 could match by accident. grep -c counts the matches, but you also need the lines themselves, so redirect with > into errors.txt.

Hint 3: variables, defaults and the argument

The first argument to a script is $1. Give variables readable names at the top: log="$1". A default for an environment variable uses ${TOP_N:-3}, which means "the value of TOP_N, or 3 if it is unset or empty". Always quote variables that might contain spaces: "$log".

If you use set -u (an error on unset variables), $1 itself is unset when no argument is given. Write ${1:-} to say "the first argument, or empty if missing".

Hint 4: errors go to stderr, and exit codes

To send a message to stderr, redirect the echo: echo "usage: $0 LOGFILE" >&2. The >&2 means "send stdout to wherever stderr goes". exit 2 stops the script with that code. $0 is the name the script was run as, which is ./report.sh. Test for a missing argument with [ -z "$log" ] (true if empty) and a missing file with [ ! -f "$log" ] (true if not a regular file).

Hint 5: making it executable and private

chmod +x report.sh lets you run it. For errors.txt, chmod 600 errors.txt gives read and write to you alone. Do the chmod after the file is created. Verify with ls -l: -rw------- for the data file and -rwxr-xr-x (or similar) for the script.

Hint 6: pipelines inside echo

To print a number after a label, use command substitution: echo "Total requests: $(wc -l < "$log")". Notice the quotes nested inside $( ): they are allowed, and they protect a log path with spaces. On macOS, wc -l pads the number with spaces, so Total requests: 20 is normal there.

Reference solution

Try before you look. If you peek, close it and rewrite from memory after reading.

Solution: report.sh
#!/usr/bin/env bash
# Usage: ./report.sh LOGFILE
# Set TOP_N to change how many paths are listed (default 3).
set -u
 
log="${1:-}"
top="${TOP_N:-3}"
 
if [ -z "$log" ]; then
  echo "usage: $0 LOGFILE" >&2
  exit 2
fi
if [ ! -f "$log" ]; then
  echo "error: no such file: $log" >&2
  exit 1
fi
 
echo "== Report for $log =="
echo "Total requests: $(wc -l < "$log")"
echo "Unique visitors: $(cut -d' ' -f1 "$log" | sort -u | wc -l)"
 
echo
echo "Requests per status code (count, code):"
cut -d' ' -f5 "$log" | sort | uniq -c | sort -rn
 
echo
echo "Top $top paths (count, path):"
cut -d' ' -f4 "$log" | sort | uniq -c | sort -rn | head -n "$top"
 
# Server errors (status 5xx) go to a private file.
grep -E ' 5[0-9][0-9] ' "$log" > errors.txt
chmod 600 errors.txt
echo
echo "Server errors: $(wc -l < errors.txt) (saved in errors.txt)"

Create it and run it:

$ nano report.sh          # paste the script, Ctrl+O, Enter, Ctrl+X
$ chmod +x report.sh
$ ./report.sh access.log
== Report for access.log ==
Total requests: 20
Unique visitors: 5
 
Requests per status code (count, code):
     14 200
      2 500
      2 404
      1 503
      1 302
 
Top 3 paths (count, path):
      5 /products
      5 /home
      3 /checkout
 
Server errors: 3 (saved in errors.txt)
$ ls -l errors.txt
-rw------- 1 anisha anisha 165 Nov  4 15:30 errors.txt
$ TOP_N=1 ./report.sh access.log | grep -A1 Top
Top 1 paths (count, path):
      5 /products
Solution walk-through: why each line is there
  • set -u makes the script stop with an error if it reads an unset variable, which catches typos such as $lgo.
  • ${1:-} and ${TOP_N:-3} give safe defaults. The first lets the script run its own usage check even with set -u. The second makes TOP_N optional.
  • The usage message goes to stderr with >&2, and the exit codes (2 for bad usage, 1 for a missing file) let other scripts react to failures, which is the same idea as the exit codes of grep.
  • wc -l < "$log" prints only the number, because with < the program receives input on stdin and has no file name to print.
  • cut -d' ' -f1 | sort -u | wc -l counts distinct visitors: take the field, deduplicate, count lines. sort -u is the short form of sort | uniq.
  • The status table and the top-paths table use the counting pipeline from lesson four. They differ only in the field number.
  • head -n "$top" keeps K lines. Quoting keeps the value as one word.
  • The grep -E ' 5[0-9][0-9] ' pattern includes the surrounding spaces so it only matches the status field.
  • > replaces any old errors.txt, which is what "replacing any older copy" asked for. Running twice never doubles the lines.
  • chmod 600 runs after the file exists, so it is never readable by others even for a moment longer than needed. (Strictly, a umask of 077 set before creating the file would remove even that moment.)

Test it like a professional

A script that works once on the sample is a demo. A script you trust has been tried on awkward input. Do these checks and record the result of each.

  1. Empty file: : > empty.log creates a zero-byte file. ./report.sh empty.log should print Total requests: 0 and Server errors: 0 with empty tables. It must not crash.
  2. A name with a space: cp access.log "my log.txt", then ./report.sh "my log.txt". If the output breaks, you forgot quotes somewhere.
  3. A second log: write a five-line log of your own. Work out the expected answers by hand before you run the script. Then compare. If they differ, decide who is wrong, you or the script.
  4. Run it twice: errors.txt should have the same three lines the second time, not six.
  5. From another folder: cd .. then ./log-detective/report.sh log-detective/access.log. It still works, but errors.txt appears in the folder you ran it from. Think about whether that is what you want, and say how you would change it (a hint: write the file next to the log, using dirname).

Here is a tiny second log to try, with answers you can verify:

$ printf '10.0.0.1 2026-11-04T10:00:00 GET /a 200 1\n10.0.0.1 2026-11-04T10:00:01 GET /a 200 1\n10.0.0.2 2026-11-04T10:00:02 GET /b 500 1\n' > tiny.log
$ ./report.sh tiny.log
== Report for tiny.log ==
Total requests: 3
Unique visitors: 2
 
Requests per status code (count, code):
      2 200
      1 500
 
Top 3 paths (count, path):
      2 /a
      1 /b
 
Server errors: 1 (saved in errors.txt)

Three requests, two visitors, one server error: that matches what you can read off the three lines by eye.

Extensions (for the top marks)

Choose at least one.

  1. Busiest minute. The timestamp's hour and minute are characters 12 to 16 of field 2. cut -d' ' -f2 | cut -c12-16 extracts them. Add a line Busiest minute: HH:MM (N requests) using the counting pipeline.
  2. Error rate. Print Error rate: 15% (3 of 20) using shell arithmetic: echo $(( 100 * errors / total )). Handle an empty log without dividing by zero.
  3. Bandwidth. Sum the bytes (field 6). Hint: cut -d' ' -f6 | paste -sd+ | bc joins the numbers with plus signs and calculates. Print the total in bytes.
  4. Many logs. Allow several arguments (./report.sh a.log b.log) and print one report for each. Look up the shell's for log in "$@"; do ... done loop.
  5. Install it. Move report.sh into ~/bin, add ~/bin to your PATH in ~/.bashrc, and run it from any folder with the bare command report.sh access.log. Explain in two sentences why the folder must be on PATH and what the x bit is for.

Reflect

When you finish, write three or four sentences in a NOTES.txt file in the folder (using echo and >>, as practice). Answer: which pipeline stage was hardest to get right, what mistake did you make first, and which flag or command from this module surprised you most? Re-reading it in a month is a quick measure of what you learnt.

What you practised

  • Navigation and files: a project folder, a here-document, a script file.
  • Pipes and filters: cut, sort, uniq, wc, head, grep -E.
  • Streams and exit codes: errors on stderr, meaningful exit statuses.
  • Variables: arguments, defaults, and an environment variable that changes behaviour.
  • Permissions: an executable script and a private output file.

Stuck? Ask the tutor

Ask about anything in this lesson. Answers are short and use simple words.

Sign in to ask the tutor.

← Environment variables and permissions