Routers, hops and traceroute

Why this matters

A page loads slowly. A colleague says "it is probably your internet provider". You do not know whether that is true. A ping to the site tells you one number for the whole trip, and that number does not say where the time went.

A traceroute splits the trip into steps and gives a time for each step. You can see which router the path reaches, how long each part takes, and where replies stop coming back. This lesson shows how packets move through routers, what TTL does, and how to read a traceroute output line by line.

The earlier lesson "Clients, servers and IP" introduced routers and TTL in one paragraph. The lesson on latency covered ping and round-trip time. This lesson uses both to look inside the path.

The idea: a chain of handoffs

A router is a box whose job is to receive a packet, look at its destination address, and send it to the next router that is closer to that address. Each router knows only its own neighbours. It does not know the full route.

Take Ana's laptop and a web server far away. A packet from her laptop may pass through:

1. her home router,
2. a router at her internet provider,
3. several routers in the provider's wider network and in other networks,
4. a router at the server's network,
5. finally, the server.

Each handoff is called a hop. The example above has about six hops. A path across a country often has 10 to 20. No router has the whole map. Each one only knows which neighbour to send a packet to for a given range of addresses, much like a sorting office that knows only which truck goes to which region.

TTL: a counter that stops packets forever

Every packet carries a number called TTL (time to live). The sender sets it to a starting value, such as 64. Each router that handles the packet subtracts 1. If a router would subtract 1 and get 0, it throws the packet away instead of forwarding it. Then it sends a message back to the sender. That message is called "time exceeded".

The message is part of ICMP (Internet Control Message Protocol). ICMP is a side protocol that devices use to send status and error messages to each other. Ping uses it too. A packet itself does not carry a message about why it was dropped, so the router sends an ICMP message to the sender.

TTL exists so that a packet cannot circle forever if two routers send it back and forth. Here is the counting in a small model. The path has four routers, and the destination is example.com. The model starts a packet with three different TTL values.

Output:

start TTL 2:
  passed home router, TTL now 1
  dropped at provider router, which sends back a time exceeded message
start TTL 3:
  passed home router, TTL now 2
  passed provider router, TTL now 1
  dropped at backbone router, which sends back a time exceeded message
start TTL 5:
  passed home router, TTL now 4
  passed provider router, TTL now 3
  passed backbone router, TTL now 2
  passed far-side router, TTL now 1
  arrived at example.com with TTL 1

Read the first block. A packet with TTL 2 gets through one router and dies at the second. The router that dies is the one that sends the "time exceeded" message back. The sender learns the address of that router from the message.

This is the whole trick behind traceroute. If the sender can control the TTL, and it hears back from each router that kills a packet, it can map the path one router at a time.

Using TTL to list the hops: traceroute

Traceroute works in rounds:

1. Send a probe to the destination with TTL 1. The first router kills it and replies "time exceeded" with its address. The sender records the address and the time the reply took.
2. Send a probe with TTL 2. The second router replies the same way.
3. Keep increasing TTL by 1 until a probe reaches the destination. The destination answers in its own way, and traceroute stops there.

The commands are traceroute example.com on macOS and most Linux systems, and tracert example.com on Windows. Each output line is one hop. Some systems send UDP probes by default (UDP is the lossy protocol from lesson 01), and Windows tracert sends ICMP echo requests (the same kind of message ping sends). The result is the same list of routers.

Traceroute may not be installed on every system. The script below uses the same method in about 40 lines, so you can see each step. It is Linux code and it needs administrator rights, because it opens a raw socket. A raw socket reads every ICMP message on the machine, and only an administrator may do that. The script prints one line per hop.

import select
import socket
import sys
import time
 
target = sys.argv[1]
base_port = 33434
dst = socket.gethostbyname(target)
print(f"traceroute to {target} ({dst}), 15 hops max, UDP probes")
 
recv = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
for ttl in range(1, 16):
    send = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
    send.setsockopt(socket.IPPROTO_IP, socket.IP_TTL, ttl)
    port = base_port + ttl
    t0 = time.time()
    send.sendto(b"probe", (dst, port))
    found = None
    deadline = t0 + 2.0
    while time.time() < deadline:
        ready, _, _ = select.select([recv], [], [], deadline - time.time())
        if not ready:
            break
        data, addr = recv.recvfrom(1500)
        ihl = (data[0] & 0x0F) * 4
        icmp_type = data[ihl]
        inner = data[ihl + 8:]
        inner_ihl = (inner[0] & 0x0F) * 4
        inner_dport = int.from_bytes(inner[inner_ihl + 2:inner_ihl + 4], "big")
        if (icmp_type == 11 or (icmp_type == 3 and data[ihl + 1] == 3)) and inner_dport == port:
            found = (addr[0], (time.time() - t0) * 1000, icmp_type)
            break
    send.close()
    if found is None:
        print(f"{ttl:2}  *")
        continue
    ip, ms, kind = found
    label = "destination reached" if kind == 3 else "router (TTL ran out here)"
    print(f"{ttl:2}  {ip:15} {ms:7.2f} ms  {label}")
    if kind == 3:
        break

The script sends one UDP probe per TTL value. It then waits up to 2 seconds for an ICMP reply that quotes that probe's port number. Type 11 means "time exceeded", which comes from a router. Type 3 means "destination unreachable", and the destination sends it with the code for "port unreachable" when no program listens on the probe's port. The script accepts only that code, so other type 3 replies show as stars. That is what a normal end to a trace looks like. The probe reached the server, and the server had no program on that port.

Save the script as mini_traceroute.py and run sudo python3 mini_traceroute.py example.com. sudo runs a command with administrator rights. This output came from a run on the machine used to write this lesson. Your addresses will differ.

traceroute to example.com (104.20.23.154), 15 hops max, UDP probes
 1  172.27.192.1       1.14 ms  router (TTL ran out here)
 2  192.168.29.1       1.50 ms  router (TTL ran out here)
 3  10.13.64.1         4.10 ms  router (TTL ran out here)
 4  172.26.86.195     10.57 ms  router (TTL ran out here)
 5  192.168.162.160   11.33 ms  router (TTL ran out here)
 6  172.17.186.137    10.53 ms  router (TTL ran out here)
 7  172.17.186.98      9.64 ms  router (TTL ran out here)
 8  192.168.31.112     9.23 ms  router (TTL ran out here)
 9  *
10  *
11  *
12  *
13  104.20.23.154     31.49 ms  destination reached

Reading a traceroute, line by line

Here is the output above read as a checklist. Each step uses only the numbers in the output.

Line 1: the first router is the gateway. Hop 1 is the first router your packets reach. It is the gateway, the router that leads out of your local network. Its address is private (lesson 01 covered those ranges), because it sits inside your network. In this run it is 172.27.192.1. Hops 2 to 8 also use private addresses, which are common inside internet providers' networks. You do not need to know who runs each one to read the trace.

Lines 1 to 4: the time jumps. The times are round trips. Each time is the time for the probe to reach that hop and for the reply to come back. The times are about 1 ms, then 1.5 ms, then 4 ms, and then they jump to about 10.5 ms at hop 4. The jump is the largest step among the first eight lines, and it is the place to look first. A jump that stays after hop 4 tells you that the extra time is added on the way to hop 4, on the link between hop 3 and hop 4. (The router at hop 4 may also be slow to reply, so check the next line before you decide.)

Lines 5 to 8: the times stay flat. From hop 4 to hop 8 the times stay near 10 ms. Nothing else on this stretch adds much. If the time had jumped at hop 6 and stayed high, you would look at the link between hops 5 and 6 instead.

Lines 9 to 12: stars. A * means no reply came back in time. These four routers did not answer the probes. A star does not mean the path is broken. Many routers do not reply to traceroute probes, and some are set to answer very slowly. What matters is whether later hops answer. Hop 13 did, so the path works through those routers.

Line 13: the destination. The last line says "destination reached", and its time is 31.49 ms. That is the round-trip time for the whole trip in this run, the same kind of number ping gives you.

The slow hop rule. A router that is slow to answer shows a spike at its own line, and the next line is fast again. That is usually a router that deprioritises replies, not a slow link. A real slow link shows a jump that stays on every line after it. This run has no spike, so the jump at hop 4 is the one to read.

Two things a single trace cannot tell you. It cannot prove which router is at fault, because the time at hop N includes the time to get there. And it shows one path at one moment. A second run can show different routers and different times.

Worked example: a path that stops early

Suppose a trace shows hops 1 to 5 with normal times, then stars from hop 6 to hop 15, and no destination line. This is the case most people fear. Read it in two steps.

First, check whether hop 5 is the last router that answers. Then ask whether the destination answers a ping. If the destination answers ping but traceroute stops, the problem is probably the probe type. Some firewalls drop UDP probes and let ICMP echo through, so try the other probe type (the Windows tracert command sends ICMP, for example). If the destination does not answer at all, and hop 5 is the last reply, the break is likely at or just after hop 5. Run the trace again later before you report it, because routers that drop probes can change from one run to the next.

What to collect before you report a slow path

A trace on its own rarely settles an argument. If you want someone else to act on it, collect a few things first.

1. Run the trace two or three times, a few minutes apart. A hop that is slow in every run is a real finding. A hop that is slow once is noise.
2. Run ping to the destination too, and note its average time. Ping gives the total, the trace gives the steps, and you need both to say which part to look at.
3. Write down the time of each run, in your own time zone, and the name you traced. Someone else can only compare runs if they know when and where each one started.
4. Copy the exact output into your message. Do not summarise it from memory, because a line you forget is often the line that matters.

Keep the claim small. "Times jump from about 4 ms to about 10 ms at hop 4, in three runs" is something a network team can check. "The provider is slow" is not.

Common mistakes

Treating every star as a broken hop. The run above shows four stars in a row, and the path still reached the destination. A star alone is not evidence. Look for the destination line.

Using ping on a router and expecting a reply. Ping to the first router, 172.27.192.1, gave this output on the machine used to write this lesson:

PING 172.27.192.1 (172.27.192.1) 56(84) bytes of data.
 
--- 172.27.192.1 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

Yet the same router appears as hop 1 in the traceroute, with a reply time. The router does not answer ping, but it does send the "time exceeded" message that traceroute needs. So one failed ping does not prove that a router is broken. Ping the destination, and run traceroute to see the path.

Running the trace script without administrator rights. The script opens a raw socket, and that needs administrator rights. Without them, Python stops with this error:

PermissionError: [Errno 1] Operation not permitted

Run the script with sudo, or use the system's traceroute command instead.

Comparing two traces as if they were the same path. The same name can give different addresses from one run to the next. In the two runs on the machine used to write this lesson, the destination was 172.66.147.243 in the first run and 104.20.23.154 in the second. The routers in the middle also changed. Compare times and hop positions, not exact addresses, unless you ran the trace from the same place at the same time.

Reading the destination time as server speed. The time on the last line includes the trip to the server and back. It says nothing about how fast the server answers a request. For that, look at the server's response time for one request in your browser's network tab.

Practice

Exercise 1. Using the TTL model above, what happens to a packet that starts with TTL 4 on the same four-router path?

Show the answer

The packet passes the home router (TTL 3), the provider router (TTL 2), and the backbone router (TTL 1). At the far-side router the TTL would become 0, so that router drops the packet and sends back a "time exceeded" message. The packet never reaches example.com.

Exercise 2. In the traceroute output above, hop 3 shows 4.10 ms and hop 4 shows 10.57 ms. How much extra time does the step from hop 3 to hop 4 add, and what does that tell you about where to look?

Show the answer

The extra time is about 6.5 ms (10.57 minus 4.10). That is the largest step in the first eight lines, and the times stay near 10 ms afterwards, so the extra time is added on the way to hop 4. Look at the link between hop 3 and hop 4 first.

Exercise 3. Hops 9 to 12 show stars, and hop 13 replies with "destination reached". Is the path broken? Give one reason.

Show the answer

No. Hop 13 answered, so the probes reached the destination through those routers. The routers at hops 9 to 12 simply did not send replies, which is common.

Exercise 4. A ping to your first router shows 100% packet loss, but your browser loads pages without a problem. Is the first router broken? What would you run next?

Show the answer

Not necessarily. The router may drop ping requests and still send the "time exceeded" messages that traceroute uses. The browser working is good evidence that the first router forwards traffic. Run traceroute to the site to see the path and the times.

Exercise 5. Suppose a trace shows these round-trip times for hops 1 to 7: 2 ms, 3 ms, 4 ms, 180 ms, 181 ms, 180 ms, 182 ms. Which step should you investigate, and why?

Show the answer

Investigate the step from hop 3 to hop 4. The time jumps from 4 ms to 180 ms there, and it stays near 180 ms for every later hop. That pattern means the extra delay is added on the way to hop 4, most likely on the link between hop 3 and hop 4, and the rest of the path adds little.

Summary

• A router passes each packet to a neighbour that is closer to the destination. Each handoff is a hop.
• TTL is a counter that each router lowers by 1. A router that would lower it to 0 drops the packet and sends a "time exceeded" message back.
• Traceroute sends probes with TTL 1, 2, 3 and so on, and records the router that replies to each one.
• A jump in time that stays on every later line marks where the extra delay is added. A single spike that disappears on the next line usually means a router that is slow to answer probes.
• Stars mean no reply. Only a missing destination line, with no later replies, suggests a real break.
• Ping and traceroute answer different questions. A failed ping to a router does not prove the router is broken.