Embercode
✕
How the Internet WorksLesson 6 of 15

60 min read

Module assignment: MiniWeb, a pretend internet

The project

You have spent five lessons learning what happens between typing a URL and seeing a page. Now you will build it. MiniWeb is a pretend internet that runs inside one Python program. It has no real network, but every piece does the job its real counterpart does, using the same rules you studied: URLs are parsed, names are resolved through a cache, requests and responses are real HTTP text, a server routes by method and path, and a browser follows redirects, reuses cached pages and refuses a certificate that does not match the name.

Why simulate it? Because writing the parts forces you to know them exactly. You cannot write parse_request without knowing where the headers end, and you cannot write a cache without deciding when an entry expires. When you meet a real framework later, you will recognise each part as a larger version of something you wrote yourself.

You will fill in a starter file with eight pieces. A built-in self-check reports PASS or FAIL for five groups of behaviour. Then you will run a demo of your own that shows a whole page load, a redirect, a cache hit and three kinds of failure.

What you build

Everything lives in one file. Here are the pieces and exactly how each must behave.

1. parse_url(url)

Takes an absolute URL string and returns a dictionary with the keys scheme, host, port (an integer), path, query and fragment.

  • A missing path becomes /.
  • A missing port uses the default for the scheme: 80 for http and 443 for https.
  • Missing query and fragment are empty strings, and the stored values do not include the ? or #.
  • A URL without :// raises ValueError. A scheme other than http or https raises ValueError.

Example: parse_url("https://shop.example/a/b?x=1&y=2#top") returns scheme https, host shop.example, port 443, path /a/b, query x=1&y=2, fragment top.

2. DNS class

Created with a zone: a dictionary from name to (ip, ttl_seconds). It has a cache and counts how many times it had to ask the zone.

  • resolve(name, now) returns the IP address. now is a fake clock in seconds, so your output does not depend on real time.
  • If the cache holds the name and now is strictly less than the expiry time, return the cached address and do not count a lookup.
  • Otherwise look in the zone, increase upstream_lookups by one, store the address with expiry now + ttl and return it.
  • A name that is not in the zone raises LookupError("NXDOMAIN: <name>").

3. build_request and parse_request

build_request(method, host, target, body="") returns the raw request text, using \r\n line endings:

GET /search?q=shoes HTTP/1.1
Host: shop.example

The first line is the method, the target and HTTP/1.1. Then a Host header. If a body is present, a Content-Length header with the byte length of the body follows. A blank line ends the headers, then the body.

parse_request(raw) does the reverse and returns a dictionary with method, path, query (the text after ?, or empty), headers (names in lower case) and body.

4. build_response and parse_response

build_response(status, body="", headers=None) returns raw response text. The status line is HTTP/1.1 <code> <reason>. Use these reasons: 200 OK, 301 Moved Permanently, 404 Not Found, 405 Method Not Allowed and 500 Internal Server Error, and Unknown for any other code. A Content-Length header comes first, followed by the extra headers.

parse_response(raw) returns status (an integer), headers (lower-case names) and body.

5. Server class

Created with certificate_for, the host name on the server's certificate. It has a route table.

  • route(method, path, fn) registers a function that takes the parsed request and returns a raw response string.
  • handle(raw) parses the request and finds the route for its method and path.
  • If there is no route for that path with any method, return a 404 response.
  • If the path exists but not for that method, return a 405 response.
  • If the route function raises any exception, return a 500 response with the body Server error. A server must never crash because one handler failed.

6. Browser class

Created with a DNS object and a network: a dictionary from IP address to Server. It has a clock (starting at 0), a cache and a log (a list of text lines).

fetch(url, max_redirects=5) loads a page and returns the parsed response. In order, it does this:

  1. If a fresh response for exactly this URL is in the cache, log cache hit <url> and return it.
  2. Parse the URL and resolve the host through the DNS object.
  3. Find the server at the IP address. If there is none, raise ConnectionError("no server at <ip>").
  4. If the scheme is https and the server's certificate_for differs from the host, raise ConnectionError("certificate does not match <host>").
  5. Build a GET request (target is the path plus ?query when there is a query), give it to the server and parse the response. Log GET <url> -> <status> (ip <ip>).
  6. If the status is 301, build the new URL from the same scheme and host plus the Location header, and repeat from step 1.
  7. If the response has a max-age header greater than zero, store it in the cache until clock + max-age.
  8. After more than max_redirects redirects, raise RuntimeError("too many redirects").

Requirements checklist

Copy this list into your notes and tick items as the self-check passes.

  • parse_url returns all six parts and applies default ports and the default path
  • parse_url raises ValueError for a missing or unsupported scheme
  • DNS.resolve uses the cache until the TTL runs out, and counts only real lookups
  • DNS.resolve raises LookupError for an unknown name
  • build_request and parse_request agree with each other on a POST with a body
  • Content-Length counts bytes, not characters
  • build_response and parse_response agree, with lower-case header names after parsing
  • Server.handle gives 404 for unknown paths, 405 for wrong methods and 500 when a handler fails
  • Browser.fetch follows a 301 redirect and logs every hop
  • Browser.fetch serves a repeat request from the cache while max-age allows
  • Browser.fetch rejects a certificate whose name does not match the host
  • All five self-check groups print PASS
  • Your own demo prints a log with at least one redirect, one cache hit and three different errors or statuses

The starter file

Paste your work into this box and press Run. Right now every group prints FAIL, because each function raises NotImplementedError. Your job is to turn the five lines green, one piece at a time. Work in the order of the sections above, since later pieces use earlier ones.

Python · editable

When every piece is done, the output is:

PASS parse_url
PASS DNS cache
PASS messages
PASS Server
PASS Browser

Your own demo

The self-check proves the pieces work alone. The demo proves they work together. Below the self-check, add code that builds a small pretend internet and loads pages through it. Use this setup, or invent your own:

  • A server certified for shop.example with routes: GET / (200, body Welcome, header max-age: 60), GET /old (301, Location: /), GET /search (200, body results for followed by the query) and GET /boom (a handler that divides by zero).
  • A DNS zone that knows shop.example at 203.0.113.5 (TTL 300) and fake.example at 203.0.113.6 (TTL 300).
  • A second server at 203.0.113.6 whose certificate says other.example.

Then fetch these URLs in order, adding 10 to the browser clock after each: /, / again, /old, /search?q=shoes, /missing, /boom. After them fetch https://fake.example/ and https://nowhere.example/ inside try blocks and print the error each raises. Finally print the log and the number of upstream DNS lookups.

Example output

If your pieces follow the spec, the demo prints exactly this:

200 Welcome
200 Welcome
200 Welcome
200 results for q=shoes
404 Not Found
500 Server error
ConnectionError: certificate does not match fake.example
LookupError: NXDOMAIN: nowhere.example
--- log
GET https://shop.example/ -> 200 (ip 203.0.113.5)
cache hit https://shop.example/
GET https://shop.example/old -> 301 (ip 203.0.113.5)
cache hit https://shop.example/
GET https://shop.example/search?q=shoes -> 200 (ip 203.0.113.5)
GET https://shop.example/missing -> 404 (ip 203.0.113.5)
GET https://shop.example/boom -> 500 (ip 203.0.113.5)
DNS upstream lookups: 2

Read the log with the lessons in mind. The first line is a real fetch. The second request for / is a cache hit, because max-age is 60 and only 10 seconds passed. Fetching /old is a 301, and its redirect target / is again served from the cache (the clock is at 20). The search was a fresh request, and the 404 and 500 were produced by the server's own logic and not by the browser. DNS did exactly two upstream lookups: one for shop.example (every later request used the cache) and one for fake.example. The name nowhere.example failed before any lookup was counted.

Grading rubric

60 percent: it works for the normal path.

  • parse_url, the message builders and parsers, and the Server pass their checks.
  • A browser fetch of a normal page returns the right status and body.
  • Redirects and caching may be missing or flawed, and error handling may be incomplete.
  • Code is in one file and runs.

80 percent: it follows the whole spec.

  • All five self-check groups print PASS.
  • Redirects, caching with max-age, the TTL-aware DNS cache and the certificate check all work, and the demo log matches the example.
  • Errors are the right types with the right messages.
  • Functions are short, named for what they do, and free of repeated code. Parsing of requests and responses shares logic instead of copying it.

100 percent: it is careful and you can explain it.

  • Everything in 80, plus the stretch items below chosen and tested: at least two.
  • Edge cases are handled on purpose: a URL with a port but no path, a header value containing a colon, a body with a non-ASCII character whose byte length differs from its character length, and a redirect loop that ends in RuntimeError instead of running forever.
  • You wrote at least three extra tests of your own beyond the starter's.
  • You can explain, in your own words, what each of the eight pieces corresponds to in the real internet, using the module's vocabulary.

Hints

Try for a few minutes before opening each hint. They go from a nudge to nearly the answer.

Hint 1: parse_url

Peel the URL from the end. First split off the fragment at the first #. Then split off the query at the first ?. Then split the scheme off at ://. What remains is the host (maybe with a port) and the path. rest.partition("/") gives you the host and port part, and then the path (add the slash back). partition(":") on the host part gives you the port, if any. Doing the splits in this order matters because a query or fragment can itself contain a slash.

Hint 2: DNS.resolve

Check the cache first. The condition for a hit is name in self.cache and now < self.cache[name][1]. On a miss, raise LookupError if the name is not in the zone, otherwise increase the counter, store (ip, now + ttl) and return the ip. The test at time 100 with TTL 100 checks that an entry expiring at exactly 100 is already stale.

Hint 3: building and parsing messages

Build a list of lines, then join them with "\r\n", then add "\r\n\r\n" and the body. For the content length use len(body.encode()), which counts bytes. To parse, use raw.partition("\r\n\r\n") to separate head and body, split the head by "\r\n", split the first line on spaces, and split every header line on its first colon only. The status line needs split(" ", 2) so that a reason with spaces stays whole. Parse request and response with the same small helper for headers to avoid repeating yourself.

Hint 4: Server.handle

Look up (method, path) in self.routes. If it is missing, check whether any route has that path ({p for (_m, p) in self.routes}). If yes, 405, otherwise 404. Wrap the call to the route function in try and except Exception and return a 500 response in the except branch.

Hint 5: Browser.fetch

Use a for loop that runs at most max_redirects + 1 times. Inside, do the steps in order and continue after a redirect. Put the raise RuntimeError after the loop, which is reached only when every round ended in a redirect. Remember that the certificate check applies only to https, and that the redirect URL is scheme + "://" + host + location. Store a cache entry only when max-age is above zero.

Full reference solution

Look only after you have tried. This is one correct solution, with the demo at the bottom. It was run, and it prints the output shown above.

Solution
# MiniWeb: a pretend internet that runs in one program.
DEFAULT_PORTS = {"http": 80, "https": 443}
 
 
def parse_url(url):
    rest, fragment = (url.split("#", 1) + [""])[:2]
    rest, query = (rest.split("?", 1) + [""])[:2]
    if "://" not in rest:
        raise ValueError("missing scheme: " + url)
    scheme, rest = rest.split("://", 1)
    if scheme not in DEFAULT_PORTS:
        raise ValueError("unsupported scheme: " + scheme)
    host_port, slash, path = rest.partition("/")
    path = "/" + path
    host, _, port = host_port.partition(":")
    port = int(port) if port else DEFAULT_PORTS[scheme]
    return {"scheme": scheme, "host": host, "port": port,
            "path": path, "query": query, "fragment": fragment}
 
 
class DNS:
    def __init__(self, zone):
        self.zone = zone              # name -> (ip, ttl seconds)
        self.cache = {}               # name -> (ip, expires_at)
        self.upstream_lookups = 0
 
    def resolve(self, name, now):
        if name in self.cache and now < self.cache[name][1]:
            return self.cache[name][0]
        if name not in self.zone:
            raise LookupError("NXDOMAIN: " + name)
        ip, ttl = self.zone[name]
        self.upstream_lookups += 1
        self.cache[name] = (ip, now + ttl)
        return ip
 
 
def build_request(method, host, target, body=""):
    lines = [f"{method} {target} HTTP/1.1", f"Host: {host}"]
    if body:
        lines.append(f"Content-Length: {len(body.encode())}")
    return "\r\n".join(lines) + "\r\n\r\n" + body
 
 
def parse_headers(lines):
    headers = {}
    for line in lines:
        k, v = line.split(":", 1)
        headers[k.strip().lower()] = v.strip()
    return headers
 
 
def parse_request(raw):
    head, _, body = raw.partition("\r\n\r\n")
    lines = head.split("\r\n")
    method, target, _version = lines[0].split(" ")
    path, _, query = target.partition("?")
    return {"method": method, "path": path, "query": query,
            "headers": parse_headers(lines[1:]), "body": body}
 
 
def build_response(status, body="", headers=None):
    reasons = {200: "OK", 301: "Moved Permanently", 404: "Not Found",
               405: "Method Not Allowed", 500: "Internal Server Error"}
    lines = [f"HTTP/1.1 {status} {reasons.get(status, 'Unknown')}"]
    all_headers = {"Content-Length": str(len(body.encode()))}
    all_headers.update(headers or {})
    lines += [f"{k}: {v}" for k, v in all_headers.items()]
    return "\r\n".join(lines) + "\r\n\r\n" + body
 
 
def parse_response(raw):
    head, _, body = raw.partition("\r\n\r\n")
    status_line, *header_lines = head.split("\r\n")
    _version, code, _reason = status_line.split(" ", 2)
    return {"status": int(code), "headers": parse_headers(header_lines), "body": body}
 
 
class Server:
    def __init__(self, certificate_for):
        self.certificate_for = certificate_for   # host name on its certificate
        self.routes = {}                         # (method, path) -> function(request)
 
    def route(self, method, path, fn):
        self.routes[(method, path)] = fn
 
    def handle(self, raw):
        req = parse_request(raw)
        fn = self.routes.get((req["method"], req["path"]))
        if fn is None:
            paths = {p for (_m, p) in self.routes}
            if req["path"] in paths:
                return build_response(405, "Method Not Allowed")
            return build_response(404, "Not Found")
        try:
            return fn(req)
        except Exception:
            return build_response(500, "Server error")
 
 
class Browser:
    def __init__(self, dns, network):
        self.dns = dns
        self.network = network       # ip -> Server
        self.cache = {}              # url -> (response, expires_at)
        self.log = []
        self.clock = 0
 
    def fetch(self, url, max_redirects=5):
        for _ in range(max_redirects + 1):
            if url in self.cache and self.clock < self.cache[url][1]:
                self.log.append(f"cache hit {url}")
                return self.cache[url][0]
            u = parse_url(url)
            ip = self.dns.resolve(u["host"], self.clock)
            server = self.network.get(ip)
            if server is None:
                raise ConnectionError("no server at " + ip)
            if u["scheme"] == "https" and server.certificate_for != u["host"]:
                raise ConnectionError("certificate does not match " + u["host"])
            target = u["path"] + ("?" + u["query"] if u["query"] else "")
            raw = server.handle(build_request("GET", u["host"], target))
            resp = parse_response(raw)
            self.log.append(f"GET {url} -> {resp['status']} (ip {ip})")
            if resp["status"] == 301:
                url = u["scheme"] + "://" + u["host"] + resp["headers"]["location"]
                continue
            age = int(resp["headers"].get("max-age", "0"))
            if age:
                self.cache[url] = (resp, self.clock + age)
            return resp
        raise RuntimeError("too many redirects")
 
 
# ---- build the pretend internet -------------------------------------------
shop = Server("shop.example")
shop.route("GET", "/", lambda r: build_response(200, "Welcome", {"max-age": "60"}))
shop.route("GET", "/old", lambda r: build_response(301, "", {"Location": "/"}))
shop.route("GET", "/search", lambda r: build_response(200, "results for " + r["query"]))
shop.route("GET", "/boom", lambda r: 1 / 0)
 
dns = DNS({"shop.example": ("203.0.113.5", 300), "fake.example": ("203.0.113.6", 300)})
network = {"203.0.113.5": shop, "203.0.113.6": Server("other.example")}
browser = Browser(dns, network)
 
for url in ["https://shop.example/", "https://shop.example/",
            "https://shop.example/old", "https://shop.example/search?q=shoes",
            "https://shop.example/missing", "https://shop.example/boom"]:
    resp = browser.fetch(url)
    print(resp["status"], resp["body"])
    browser.clock += 10
 
for url in ["https://fake.example/", "https://nowhere.example/"]:
    try:
        browser.fetch(url)
    except Exception as e:
        print(type(e).__name__ + ":", e)
 
print("--- log")
print("\n".join(browser.log))
print("DNS upstream lookups:", dns.upstream_lookups)

Why it is built this way:

  • parse_url removes the fragment and then the query before looking at the rest, because those two parts can contain slashes and colons that would confuse later splits.
  • DNS.resolve checks now < expiry, so an entry is dead at exactly its expiry time. The counter increases only when the zone is asked.
  • parse_headers is shared by both parsers, so lower-casing and splitting on the first colon happen in one place.
  • Server.handle catches every exception from a handler. A server that crashes on one bad route takes all the others down with it.
  • Browser.fetch is a loop and not a recursion, so the redirect limit is a plain counter and a redirect loop ends with RuntimeError.

Stretch extensions

For the top band of the rubric, add at least two of these, and write a test for each.

  1. Redirect types. Handle 302, 307 and 308 as well as 301. Cache nothing for temporary redirects, and only a 301 or 308 in a separate redirect cache.
  2. Conditional requests. Give responses an ETag header. When a cached entry is stale, send If-None-Match and let the server answer 304 with no body. On a 304 the browser reuses the cached body and refreshes the expiry.
  3. Cookies. Let a server send Set-Cookie: session=abc123. The browser stores it per host and sends Cookie: session=abc123 on later requests to the same host. Add a /me route that answers Hello, <user> only when the cookie is present, and a 401 otherwise.
  4. Relative redirects and full URLs. Allow Location to be a full URL on another host, not only a path, and use urllib.parse.urljoin to resolve both kinds.
  5. Query parsing. Make parse_request return the query as a dictionary using urllib.parse.parse_qs, and add a /search route that reads q from it.
  6. A real TLS check. Replace the single certificate_for string with a chain such as the one in lesson 5: a certificate name, an issuer and a list of trusted roots. Reject self-signed and unknown issuers with different error messages.
  7. Statistics. Make the browser count cache hits, redirects and requests per host and print a summary table.
  8. A real server. With Python's http.server module, serve one of your routes on localhost:8000 and fetch it with curl -i. Compare the real response with the one your build_response makes.

How to know you are done

You are done when the five self-check lines say PASS, your demo output matches the example, every box in the requirements checklist is ticked, and you can answer these without looking: Which of your pieces is the resolver? Where does the TTL decision happen? Why is a 405 different from a 404? Why does the browser compare the certificate name with the host name? What would break if Content-Length were wrong? If you can answer them, you have the model that the rest of web development builds on.

Stuck? Ask the tutor

Ask about anything in this lesson. Answers are short and use simple words.

Sign in to ask the tutor.

← Browsers and HTTPS