Why this matters
Every web page is written in HTML (HyperText Markup Language). Many other files you will meet are written in XML (Extensible Markup Language): RSS feeds, Maven project files (pom.xml), Android screen layouts, SVG pictures and many older data exchanges. Both formats use the same kind of structure, and both are read by programs as a tree.
Once you see the tree, a lot of work becomes simple. A web scraper finds every price on a shop page. A test checks that a feed has a title. A developer finds the missing closing tag that breaks a whole file. The JSON lesson earlier in this module shows the same tree idea with braces and square brackets. The data cleaning lab later in this module uses the same kind of tree when it checks records.
This lesson covers three things. It explains tags, attributes and nesting. It shows how a file with those parts becomes a tree in memory. It ends with a short program that walks a web page's tree and pulls out the parts you need.
Tags, elements and attributes
A tag is a word inside angle brackets. Tags come in pairs. An opening tag such as <title> starts something, and a closing tag such as </title> ends it. The closing tag repeats the name with a slash in front.
An element is the whole thing: the opening tag, what sits between the tags, and the closing tag. Here is one element:
<title>Dune</title>The element's name is title. Its text content is Dune.
An attribute is extra information placed inside the opening tag, as a name and a quoted value:
<book id="b1" lang="en">This tag has two attributes. id has the value b1. lang has the value en. Attributes always sit inside the opening tag, never between the tags.
Some elements have no text and no children. They can be written as one tag ending in />:
<book id="b3"/>HTML has a list of such elements, called void elements. The tag <br> makes a line break and <img> shows a picture. HTML does not need a closing tag for these. XML does need one, or the /> form. The mistakes section shows what happens when it is missing.
A small XML file
Here is a short XML file that describes three books. Save it as library.xml:
<?xml version="1.0" encoding="UTF-8"?>
<library>
<book id="b1" lang="en">
<title>Dune</title>
<author>Frank Herbert</author>
<year>1965</year>
</book>
<book id="b2" lang="fr">
<title>L'Etranger</title>
<author>Albert Camus</author>
<year>1942</year>
</book>
</library>The first line is a declaration. It says the file uses XML version 1.0 and the UTF-8 text encoding. It is optional in many cases, but most tools expect it.
Each <book> element sits inside <library>. The title, author and year elements sit inside each <book>. The indentation is for people. The parser ignores the spaces and new lines between tags when it works out the structure.
The Python standard library reads XML with xml.etree.ElementTree. It has no third-party install step. The program below loads the file's text and prints each element with its depth, so the nesting shows:
import xml.etree.ElementTree as ET
text = """\
<?xml version="1.0" encoding="UTF-8"?>
<library>
<book id="b1" lang="en">
<title>Dune</title>
<author>Frank Herbert</author>
<year>1965</year>
</book>
<book id="b2" lang="fr">
<title>L'Etranger</title>
<author>Albert Camus</author>
<year>1942</year>
</book>
</library>
"""
root = ET.fromstring(text)
def show(node, depth=0):
attrs = "".join(f' {k}="{v}"' for k, v in node.attrib.items())
text_part = (node.text or "").strip()
label = f"{' ' * depth}<{node.tag}{attrs}>"
if text_part:
label += f" {text_part!r}"
print(label)
for child in node:
show(child, depth + 1)
show(root)
print("root tag:", root.tag, "| children:", len(root))
print("first book id:", root[0].get("id"))
print("titles:", [b.find("title").text for b in root.findall("book")])
print("french books:", [b.find("title").text for b in root.findall("book[@lang='fr']")])
print("years as numbers:", sum(int(b.findtext("year")) for b in root))Running it with Python 3.14.4 prints:
<library>
<book id="b1" lang="en">
<title> 'Dune'
<author> 'Frank Herbert'
<year> '1965'
<book id="b2" lang="fr">
<title> "L'Etranger"
<author> 'Albert Camus'
<year> '1942'
root tag: library | children: 2
first book id: b1
titles: ['Dune', "L'Etranger"]
french books: ["L'Etranger"]
years as numbers: 3907The show function is recursive: it prints one element, then calls itself for each child. Each level of the call goes one step deeper, and the printed indentation grows with it.
Every value read from XML is text. The year 1965 came back as the string '1965', so the program converts it with int() before adding.
The last three lines show three ways to pick items out of the tree. root.findall("book") returns the direct children named book. find("title") returns the first direct child named title. The text in brackets, book[@lang='fr'], is a small search pattern. It keeps only the books whose lang attribute is fr.
Nesting makes a tree
Nesting means one element sits inside another. The result is a tree, the same kind of structure a family tree or a folder tree has. A few words describe the parts:
- A parent is the element that contains another element.
bookis the parent oftitle. - A child is an element directly inside another.
title,authorandyearare the children of the firstbook. - A sibling is an element with the same parent. The two
bookelements are siblings. - A descendant is any element inside another at any depth.
titleis a descendant oflibrary, throughbook. - The root is the top element. In this file it is
library. An XML file has exactly one root.
Here is the library drawn as a text tree:
library
├── book (id=b1, lang=en)
│ ├── title: Dune
│ ├── author: Frank Herbert
│ └── year: 1965
└── book (id=b2, lang=fr)
├── title: L'Etranger
├── author: Albert Camus
└── year: 1942The lines connect each element to its children. A program reads the file into the same shape. It keeps each element as an object that points to its children, and each child points back to its parent.
Attributes or child elements?
Both attributes and child elements can carry data. Most XML files follow a loose rule: put a label or setting about the element in an attribute, and put the data in a child element or the text.
In the library file, id and lang are attributes because they describe each book. The title, author and year are child elements because they are the book's data. The rule is a habit, not a law of the format. Some files do it the other way round. Use the style the file's tool expects, and keep the style the same across the file.
Writing XML correctly
XML is strict. A parser stops at the first mistake it finds, and it does not guess what you meant. These rules are the ones you need most. A file that follows them is called well formed.
One root element. The file has exactly one top-level element. Two top-level elements are not allowed.
Every element is closed. Each opening tag needs a closing tag with the same name, or it must end in />.
Names match exactly. <Title> and <title> are different names. The closing tag must spell the name the same way as the opening tag.
Attribute values are quoted. Use double quotes or single quotes around each value. id=b1 without quotes is an error.
Some characters must be escaped. The ampersand & and the less-than sign are special in XML text. To write a plain ampersand, use &. To write a less-than sign, use <. XML defines five of these built-in escapes, and these two are the ones that break files most often.
Attribute names are not repeated. An opening tag may not contain the same attribute name twice.
Here is what the parser says when those rules are broken. The samples are short, so the line and column numbers are small, but the message text is the same in larger files:
import xml.etree.ElementTree as ET
cases = {
"unclosed tag": "<library>\n <book>\n <title>Dune</title>\n</library>\n",
"wrong closing name": "<book><title>Dune</titel></book>\n",
"unquoted attribute": "<book id=b1>Dune</book>\n",
"bare ampersand": "<book><title>Tom & Jerry</title></book>\n",
"escaped ampersand": "<book><title>Tom & Jerry</title></book>\n",
"html void tag": "<p>Line one<br>Line two</p>\n",
"two roots": "<a>1</a><b>2</b>\n",
"self closing": "<book id='b3'/>\n",
"duplicate attribute": '<book id="1" id="2"/>\n',
}
for label, src in cases.items():
print("==", label)
try:
node = ET.fromstring(src)
print(" parsed root:", node.tag, node.attrib, repr(node.text))
except ET.ParseError as err:
print(" error:", err)== unclosed tag
error: mismatched tag: line 4, column 2
== wrong closing name
error: mismatched tag: line 1, column 19
== unquoted attribute
error: not well-formed (invalid token): line 1, column 9
== bare ampersand
error: not well-formed (invalid token): line 1, column 18
== escaped ampersand
parsed root: book {} None
== html void tag
error: mismatched tag: line 1, column 25
== two roots
error: junk after document element: line 1, column 8
== self closing
parsed root: book {'id': 'b3'} None
== duplicate attribute
error: duplicate attribute: line 1, column 13The escaped ampersand parses and gives the root element book with no attributes and no text of its own, since the text sits in the child title. The self-closing example parses too. Both are valid XML.
Reading a web page as a tree
HTML uses the same tags, attributes and nesting as XML. A web page is a tree of elements, and a browser builds that tree from the HTML file before it draws anything. The tree that a browser builds in memory is called the DOM, which stands for Document Object Model. JavaScript code on the page reads and changes that same tree.
HTML has its own rules that differ from XML. Some elements, such as <br>, <img>, <meta> and <link>, have no closing tag. Some closing tags may be left out, and the browser fills in the gaps. HTML is also forgiving about mistakes that would stop an XML parser. A browser shows the page anyway and does its best to build a sensible tree.
Here is a small page. Save it as page.html:
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Shop</title>
<link rel="stylesheet" href="site.css">
</head>
<body>
<header><h1>Shop</h1><nav><a href="/cart">Cart</a> <a href="/help">Help</a></nav></header>
<main>
<ul class="products">
<li data-price="9.99">Mug <span class="tag">new</span></li>
<li data-price="4.50">Pen</li>
</ul>
<p>Free delivery over <strong>20</strong> dollars.</p>
</main>
</body>
</html>The <!doctype html> line at the top tells the browser which HTML version to use. It is not an element, so it does not appear in the tree.
Python's standard library includes html.parser. It reads HTML one tag at a time and reports each start tag, end tag and piece of text to your code. It does not build a tree by itself. The program below builds one. It keeps a pointer to the element that is currently open. A start tag adds a child to that element. An end tag moves the pointer back up to the parent of the matching open element.
from html.parser import HTMLParser
VOID = {"meta", "link", "br", "img", "input", "hr"}
class Node:
def __init__(self, tag, attrs=None, parent=None):
self.tag = tag
self.attrs = dict(attrs or [])
self.parent = parent
self.children = [] # Node objects and plain strings, in page order
def __repr__(self):
return f"<{self.tag}>"
class TreeBuilder(HTMLParser):
def __init__(self):
super().__init__()
self.root = Node("#document")
self.current = self.root
def handle_starttag(self, tag, attrs):
node = Node(tag, attrs, parent=self.current)
self.current.children.append(node)
if tag not in VOID:
self.current = node
def handle_endtag(self, tag):
# Walk up until the matching open tag. Browsers do something more
# careful here; this builder is only meant to show the idea.
node = self.current
while node is not None and node.tag != tag:
node = node.parent
if node is not None and node.parent is not None:
self.current = node.parent
def handle_data(self, data):
if data.strip():
self.current.children.append(data.strip())
def walk(node, depth=0):
if isinstance(node, str):
print(" " * depth + repr(node))
return
attrs = "".join(f' {k}="{v}"' for k, v in node.attrs.items())
print(f"{' ' * depth}<{node.tag}{attrs}>")
for child in node.children:
walk(child, depth + 1)
def find_all(node, tag, out=None):
if out is None:
out = []
if isinstance(node, str):
return out
if node.tag == tag:
out.append(node)
for child in node.children:
find_all(child, tag, out)
return out
def text_of(node):
if isinstance(node, str):
return node
return " ".join(text_of(c) for c in node.children)
page = open("page.html", encoding="utf-8").read()
builder = TreeBuilder()
builder.feed(page)
builder.close()
walk(builder.root)
print("links:", [a.attrs.get("href") for a in find_all(builder.root, "a")])
products = find_all(builder.root, "li")
print("products:", [(text_of(p), p.attrs.get("data-price")) for p in products])
main = find_all(builder.root, "main")[0]
print("main child tags:", [c.tag if not isinstance(c, str) else "(text)" for c in main.children])
print("paragraph text:", text_of(find_all(builder.root, "p")[0]))
print("parent of ul:", find_all(builder.root, "ul")[0].parent.tag)Running it prints the tree:
<#document>
<html lang="en">
<head>
<meta charset="utf-8">
<title>
'Shop'
<link rel="stylesheet" href="site.css">
<body>
<header>
<h1>
'Shop'
<nav>
<a href="/cart">
'Cart'
<a href="/help">
'Help'
<main>
<ul class="products">
<li data-price="9.99">
'Mug'
<span class="tag">
'new'
<li data-price="4.50">
'Pen'
<p>
'Free delivery over'
<strong>
'20'
'dollars.'
links: ['/cart', '/help']
products: [('Mug new', '9.99'), ('Pen', '4.50')]
main child tags: ['ul', 'p']
paragraph text: Free delivery over 20 dollars.
parent of ul: mainRead the output against the page. The li for the mug has two children: the text Mug and the span element. The paragraph has three children: a text piece, the strong element and another text piece. That is why Free delivery over 20 dollars. needs text_of to join the pieces.
The find_all function is the tree equivalent of a search. It walks every element, including those nested deep inside others, and collects the ones with the tag you asked for. The line find_all(builder.root, "li") finds the two products even though they sit inside ul inside main inside body.
Two points about this builder. It handles only the void elements listed in VOID. Real browsers follow the full HTML rules, so a page that is badly written may give a different tree in a browser. Use this program to learn the shape of a page, not as a replacement for a proper HTML parser such as the one in a browser or in a third-party library.
Why the tree matters
Suppose you want the price of each product. A flat search such as "find every line that contains data-price" is fragile. The tree gives you a more reliable path: find each li, then read its data-price attribute. The attribute belongs to that element, so it stays correct even if the page layout changes around it.
The same reasoning applies to text. If you read only the first text piece inside the paragraph, you get the wrong answer:
Here is the naive approach, using the tree from the program above:
print(find_all(builder.root, "p")[0].children[0])It prints Free delivery over, which drops the 20 inside the strong element. Joining all the pieces, as text_of does, gives the full sentence.
Class names hold several words
The class attribute is a special case in HTML. One element can have several class names, separated by spaces, such as class="item sale". A check for equality misses those elements. Using the TreeBuilder and find_all functions from the program above, here are two products, one on sale and one not:
small = TreeBuilder()
small.feed('<ul><li class="item sale">Pen</li><li class="item">Mug</li></ul>')
small.close()
items = find_all(small.root, "li")
print([i.attrs["class"] for i in items])
print([text_of(i) for i in items if i.attrs.get("class") == "item"])
print([text_of(i) for i in items if "item" in i.attrs.get("class", "").split()])['item sale', 'item']
['Mug']
['Pen', 'Mug']The first line prints the raw attribute values. The second line compares the whole string with "item", so only the plain item matches. The third line splits the value into separate words and checks each word, so both products match.
Before you scrape any site, read its terms of use and its robots.txt file, and send requests slowly enough that you do not load the server.
Common mistakes
These mistakes come from the XML rules above. Each one gives the error the parser prints.
A closing tag that does not match the open tag. The parser reports the first place where the names disagree:
<book><title>Dune</titel></book>mismatched tag: line 1, column 19The closing tag says titel, which does not match the open title. Fix the spelling, and check the name in the editor's search if the file is long.
A missing closing tag. A <title> that never closes causes the same kind of error at the next closing tag that does not match it. In the sample, the error is at the </library> line, because the <book> above it was never closed:
<library>
<book>
<title>Dune</title>
</library>mismatched tag: line 4, column 2The line number points to the closing tag that did not match, not to the tag that was left open. Look a few lines above the reported line for an element that has no closing tag.
An HTML void tag in XML. <br> is valid in HTML because HTML does not require its closing tag. In XML it is an open tag that never closes, so the parser fails at the next closing tag:
mismatched tag: line 1, column 25Write <br/> in XML. An HTML file that will be parsed as XML must follow the XML rules.
An ampersand that is not escaped. A plain & in text is read as the start of an entity reference, a code such as & that stands for one character. When no name follows it, the parser fails:
<book><title>Tom & Jerry</title></book>not well-formed (invalid token): line 1, column 18Write Tom & Jerry. The text read back by the parser is Tom & Jerry again.
An attribute value without quotes. The parser needs quotes around every attribute value:
<book id=b1>Dune</book>not well-formed (invalid token): line 1, column 9Write id="b1".
Two root elements. A file may have only one top-level element. Two side by side are rejected:
<a>1</a><b>2</b>junk after document element: line 1, column 8Wrap them in one parent element, such as <items>.
A repeated attribute. The parser refuses a second attribute with the same name inside one tag:
<book id="1" id="2"/>duplicate attribute: line 1, column 13Keep one value and remove the other.
Reading only the first text piece. This is a mistake in the program, not in the file. Any element can contain text, then a child, then more text. Taking children[0] ignores everything after the first child. Use a function that joins all the text, such as text_of above.
Practice
Exercise 1
The XML below has one error. Find the line the parser will report, and say what the error is.
<shop>
<item name="Milk" qty="2">
<item name="Eggs" qty="1"></item>
</shop>Show the answer
The parser reports mismatched tag at line 4, column 2, which is the </shop> line. The first <item> on line 2 is never closed, so when the parser reaches </shop> it finds an open item instead. The fix is to close it: <item name="Milk" qty="2"></item>.
Exercise 2
Using the library.xml text from this lesson, count how many books have lang set to fr. Write the code you would use.
Show the answer
import xml.etree.ElementTree as ET
root = ET.fromstring(text)
print(len(root.findall("book[@lang='fr']")))The answer is 1, because only b2 has lang="fr". The search pattern filters on the attribute value, so you do not need to loop and compare by hand.
Exercise 3
The HTML page in this lesson has two links. Write a line of code, using the find_all function and the builder from the program above, that prints the text of each link.
Show the answer
print([text_of(a) for a in find_all(builder.root, "a")])The output is ['Cart', 'Help']. find_all collects the two a elements, and text_of reads each one's text.
Exercise 4
Write a short XML file for a shopping list with three items. Each item is an element named item, its text is the item name, and its qty attribute holds the quantity. Parse it and print the total quantity.
Show the answer
import xml.etree.ElementTree as ET
shop = ET.fromstring('<list><item qty="2">Milk</item><item qty="1">Eggs</item><item qty="6">Apples</item></list>')
print([(i.text, int(i.get("qty"))) for i in shop])
print("total:", sum(int(i.get("qty")) for i in shop))[('Milk', 2), ('Eggs', 1), ('Apples', 6)]
total: 9The quantities are read as text, so the code converts them with int before adding.
Exercise 5
Explain why this XML fails, then write the corrected text.
<note>Tom & Jerry, the cartoon, is 1 < 2 years old.</note>Show the answer
Two characters in the text break the XML rules. The ampersand starts an escape sequence that is not complete, and the less-than sign would start a tag. Write the corrected text with the escapes:
<note>Tom & Jerry, the cartoon, is 1 < 2 years old.</note>The parser reads Tom & Jerry, the cartoon, is 1 < 2 years old. back from the escaped form.
Summary
- A tag is a name in angle brackets, and an element is an opening tag, its content and a closing tag.
- Attributes sit inside the opening tag as
name="value"pairs, and their values must be quoted. - Nesting turns a file into a tree with one root, where each element has a parent, children and siblings.
- XML must be well formed: one root, every element closed, and the ampersand and less-than sign escaped in text.
- HTML uses the same tree shape, but its parser is forgiving, and browsers build the tree in memory as the DOM.
- To pull data from a tree, find the elements you need by tag or attribute, then read their text and attributes, joining all the text pieces when an element has children.