Regex basics with the Python re module

Why this matters

Programs often need to check whether text has a certain shape. A form must reject a phone number with a letter in it. A log reader must pull the time and the error code out of each line. A search tool must find every place where a word appears. You can write a loop with indexes for each of these, but the loop gets long fast. A regular expression, usually shortened to regex, lets you describe the shape in a short pattern and let the library do the scanning.

A regex is a small language for describing text. The pattern \d{3}-\d{4} means three digits, a dash, and four digits. A program can compare that description with any string and say yes or no, find every place it appears, or copy out the parts you marked.

The power has a cost. Some patterns look short and simple but take a very long time on certain inputs. This lesson covers the common pieces of the language, how the Python re module uses them, and the cost of backtracking, which is the reason some patterns become very slow.

This lesson does not repeat the string basics from the earlier lessons in this module. It assumes you can create a string and call a method on it.

The idea: a pattern is a description of text

Start with the simplest example. The pattern cat matches the three letters c, a and t in that order. The text "cat" matches it. The text "cart" does not, because after ca the next letter is r, not t.

Now add one piece of description. The pattern c.t uses a dot, which means any single character. It matches cat, cot, c9t and c t. The dot stands for one character, whatever that character is.

A pattern is a sequence of pieces. Each piece matches some text. Pieces are read left to right, and the match succeeds only if every piece succeeds in order.

The table below lists the pieces that cover most real patterns.

Two things about the table matter most. The +, *, ? and {n,m} pieces are called quantifiers, because they set how many times the piece before them may repeat. The ^ and $ pieces are anchors, because they tie the match to the start or end of the text, not to a letter in the middle.

Raw strings for patterns

Many useful pieces start with a backslash, such as \d and \s. Python strings also use the backslash for their own special characters, such as \n for a new line. If you write the pattern as an ordinary string, you must double every backslash:

pattern = "\\d{3}-\\d{4}"

The usual way to write a pattern in Python is a raw string, with an r in front of the opening quote. A raw string keeps every backslash exactly as typed, so no doubling is needed:

pattern = r"\d{3}-\d{4}"

Both lines make the same pattern. Use raw strings for every regex in your code. If you leave out the r and write "\d", Python warns that \d is an invalid escape sequence. The string still ends up with a backslash and the regex still works, but the warning is a sign that the code is fragile. The warning reads SyntaxWarning: "\d" is an invalid escape sequence, and newer versions of Python may turn it into an error.

Worked example 1: whole match and search

The re module has two common ways to ask whether a pattern matches. The function re.fullmatch needs the whole text to match. The function re.search looks for the pattern anywhere in the text.

The output is:

matches 555-1234: True
matches a phone 555-1234 x: False
found: 555-1234

The second text is not a full match, even though it contains a phone number. fullmatch requires the pattern to cover the entire text, and the words call and now are not covered by the pattern. The search function finds the number inside the longer text.

Both functions return a match object, or None when nothing matches. That is why the code compares the result with None. The expression is not None gives True for a match and False for no match. Calling .group() on None would raise an error, so check the result first.

To find every match in a longer text, compile the pattern once with re.compile, and call finditer. Each match in the result gives its text and its position:

The output is:

found '17' at 6..8
found '204' at 24..27
found '9' at 43..44

The method group() returns the text of the current match. The methods start() and end() return the index of its first character and the index one past its last character. This is the same convention as slices: the start is included and the end is not.

The compiled pattern is an object you can keep and reuse. Compiling a pattern costs some time, so compile once, outside the loop, and call the methods on the compiled object for each text.

Worked example 2: capture groups

A group is a part of the pattern in parentheses. The parentheses do two jobs. They let a quantifier apply to several pieces at once, and they remember the text that matched the group, so you can read it later. That remembered text is called a capture.

Take an email address such as ada@example.com. The pattern below has three groups, one for each part before and after the symbols.

The output is:

user=ada domain=example tld=com

The number in group(1) counts the open parentheses from the left. Group 0 means the whole match, and group() with no number returns group 0.

Numbered groups become hard to read as a pattern grows. A named group gives the part a name. The syntax is (?P<name>...), and you read the group by name:

The output is:

year=2026 month=11 day=05

The function re.sub replaces each match. Its replacement text can refer to the captured groups with \1, \2 and so on, written as a raw string. The example below turns a date from year-month-day into day/month/year.

The output is:

swapped: 05/11/2026

Worked example 3: greedy and lazy quantifiers

A quantifier like * or + is greedy by default. It takes as much text as it can and gives back only what the rest of the pattern needs. A lazy quantifier takes as little as it can. You make a quantifier lazy by adding a question mark after it, as in *? or +?.

The text below has two bold tags. The greedy pattern takes everything from the first opening tag to the last closing tag. The lazy pattern stops at the first closing tag it can use.

The output is:

greedy: <b>one</b> and <b>two</b>
lazy first: <b>one</b>
lazy second: <b>two</b>

The greedy match covers the whole line because .* swallowed everything up to the last </b>. The lazy version stops at the first </b>, and the next match in the loop picks up the second tag.

A rule of thumb for beginners is this. If a pattern matches more text than you wanted, look for a * or + that could take too much, and make it lazy or make the characters it may take more specific. Matching HTML with a regex is fragile in general, and this example is only to show how greedy and lazy behave.

The same difference shows up in replacement. re.sub(r"<.*>", "", "<b>x</b>") removes everything from the first < to the last >, so the result is an empty string. re.sub(r"<.*?>", "", "<b>x</b>") removes only the two tags, and the result is x.

More patterns and common checks

These examples show the same pieces used on small jobs. Each one is a shape you will meet in real input.

The output is:

True False False False
word: Cat
word: car
word: COW
[a b c]
a -> 1
b -> 22
c -> x

The flag re.IGNORECASE makes the c match both capital and small letters. The \b stops the pattern from matching a c in the middle of a word. The pattern + is one or more spaces, and re.sub replaces each run with one space. In the key-value pattern, the = between the two groups is matched but not captured, because it is outside the parentheses.

The cost of backtracking

A regex engine matches a pattern by trying one path at a time. When a piece succeeds, it moves to the next piece. When a piece fails, the engine goes back to the last choice it made and tries another one. This going back is called backtracking.

Backtracking is usually fast. The trouble starts when a pattern gives the engine many ways to split the same text, and the text then fails at the end. The engine tries every split before it gives up. If the number of splits doubles with each character, the work doubles too.

Here is a short pattern that shows the effect. It uses an optional group, (a?), repeated n times, followed by n letters a. The text is n letters a followed by a b, so the match must fail. Each optional a? can match its letter or match nothing, which gives the engine two choices for each of the n groups. The number of paths is 2 to the power n.

import re
import time
 
for n in range(16, 24, 2):
    pattern = re.compile("^" + "(a?)" * n + "a" * n + "$")
    text = "a" * n + "b"
    start = time.perf_counter()
    found = pattern.search(text) is not None
    ms = (time.perf_counter() - start) * 1000
    print(f"n={n} found={found} milliseconds={ms:.0f}")

The output on the machine used to write this lesson was:

n=16 found=False milliseconds=2
n=18 found=False milliseconds=7
n=20 found=False milliseconds=29
n=22 found=False milliseconds=116

Each step of 2 in n made the time about four times longer, which is what doubling per letter looks like. The timings depend on the machine, so yours will differ. The shape is the lesson: adding a few letters to the input can turn a fast match into a wait of seconds.

A second pattern shows the same growth with a different shape. The pattern ^(a+)+$ gives the engine many ways to share the letters between the inner and outer plus signs. On the same machine, with n letters a followed by a b, its timings were:

n=16 found=False milliseconds=2
n=18 found=False milliseconds=7
n=20 found=False milliseconds=26
n=22 found=False milliseconds=108

The fix for this pattern is to remove the repeated group. The pattern ^a+$ matches the same text and has only one way to split it. On the same input with 24 letters, it finished in about 0.01 milliseconds, while ^(a+)+$ took about 450 milliseconds on the same input. Write patterns that leave the engine only one way to match each part of the text, and test a pattern on a long input of the wrong kind before you trust it with user data.

Common mistakes

Mistake 1: a backslash in an ordinary string. The pattern "\d+" is an ordinary string with an invalid escape. Python warns about it, as shown earlier, and the regex still works in this case. The fix is the raw string r"\d+". If you write "\\d+" instead, the string holds the same characters, but it is harder to read.

Mistake 2: a group that is never closed. The pattern (abc has an open parenthesis and no close. Python refuses to compile it:

The output is:

re.error: missing ), unterminated subpattern at position 0

The message gives the position of the open parenthesis. Count the parentheses in the pattern.

Mistake 3: a repetition range that is backwards. The pattern a{2,1} asks for between 2 and 1 repetitions. Python refuses it:

The output is:

re.error: min repeat greater than max repeat at position 2

Mistake 4: using a match that is None. A failed search returns None, not a match object. Calling .group() on None raises an error:

The output is:

AttributeError: 'NoneType' object has no attribute 'group'

Check that the result is not None before you read it, as the examples above do.

Mistake 5: asking for a group that does not exist. The pattern \d+ has no capture groups, so group(1) fails:

The output is:

IndexError: no such group

The same kind of error appears for a name. A named group year read with the name month raises IndexError: no such group. Count the groups in the pattern, or check the names.

Mistake 6: using fullmatch when you wanted a search. fullmatch checks the whole text. In the phone number example above, fullmatch returned False for a sentence that contained a phone number. Use search or finditer to look inside a longer text.

Mistake 7: a pattern from user input without a check. A pattern that comes from a user can be invalid, and it can be slow. Wrap re.compile in a try block when the pattern is not written by you, and use re.escape when you want the user's text taken literally. The seventh lesson of this module covers re.escape in more detail.

Variations and edge cases

Ignoring case. re.search("hello", "Say HELLO", re.IGNORECASE) finds the match, and the flag makes the letters match in either case.

Word boundaries. The pattern \bc\w* matches a word that starts with c, because \b means the place between a non-word character and a word character. The call re.findall(r"\bc\w*", "scow cow") returns ['cow'], because the c inside scow does not start a word. Without \b, the pattern would also match the c in the middle of scow.

Reusing a compiled pattern. Compiling a pattern costs some time. A loop that compiles the same pattern on every line of a large file wastes that time. Compile once, store the pattern in a variable outside the loop, and call its methods for each line. The re module also caches recent patterns, but the explicit compile makes the cost clear.

Empty matches. A pattern such as \d* can match the empty string, with no characters at all. The call re.findall(r"\d*", "a1") returns ['', '1', ''], which has an empty match before a, the match 1, and an empty match at the end. Test a pattern on an empty string before you rely on it.

Practice

Try each exercise before you open the answer. Exercises 1 and 2 are short. Exercises 3 to 5 combine several pieces.

1. Write is_zip_code(s) that returns True only when s is exactly five digits. Test it with "12345", "1234", "12345x" and the empty string.

Show the answer

import re
 
def is_zip_code(s):
    return re.fullmatch(r"\d{5}", s) is not None

The output of the four tests is True, False, False and False. The function fullmatch needs the whole text to match, so 12345x fails because of the letter. The empty string fails because \d{5} needs five characters.

2. Find every word in "Cat, dog, car and COW" that starts with the letter c, ignoring case. Print each one on its own line.

Show the answer

import re
 
for m in re.finditer(r"\bc\w*", "Cat, dog, car and COW", re.IGNORECASE):
    print("word:", m.group())

The output is word: Cat, word: car and word: COW. The \b stops the pattern from matching a c in the middle of a word. The flag makes the c match both upper and lower case.

3. Replace every run of one or more spaces in "a b c" with a single space. The output should be a b c.

Show the answer

import re
 
print("[" + re.sub(r" +", " ", "a    b  c") + "]")

The output is [a b c]. The pattern + is one or more spaces, and re.sub replaces each run with one space. The brackets in the output show that there are no spaces at the start or the end.

4. Read the pairs key=value from "a=1; b=22; c=x" and print each key and value. Use capture groups, and print a -> 1, b -> 22 and c -> x.

Show the answer

import re
 
for m in re.finditer(r"(\w+)=(\w+)", "a=1; b=22; c=x"):
    print(m.group(1), "->", m.group(2))

The first group captures the key and the second group captures the value. The = between them is matched but not captured, because it is not in parentheses. The loop visits each pair in order.

5. Predict the output of re.sub(r"<.*>", "", "<b>x</b>") and re.sub(r"<.*?>", "", "<b>x</b>"). Then run both and explain the difference.

Show the answer

The first call prints an empty string, shown as [] when you put brackets around it, and the second prints [x]. The first pattern is greedy. The greedy .* takes everything from the first < to the last >, which is the whole string, so the replacement removes all of it. The second pattern is lazy, so each <...> piece stops at the first >. The two tags are removed and the x stays.

6. Write is_simple_hex_color(s) that returns True for a # followed by exactly three or six hexadecimal digits. A hexadecimal digit is one of 0 to 9 or a to f in either case. Test it with #abc, #a1b2c3, #a1b2c3d4e and #ab.

Show the answer

import re
 
def is_simple_hex_color(s):
    return re.fullmatch(r"#(?:[0-9a-fA-F]{3}){1,2}", s) is not None

The output is True, True, False and False. The group (?:...) is a group that does not capture, so it only applies the quantifier. The quantifier {1,2} allows one or two blocks of three digits, which gives three or six digits. The nine-digit text needs three blocks and is rejected. The two-digit text is too short for one block.

Summary

• A regex describes the shape of text with pieces such as \d, [a-z], + and ^. Write each pattern as a raw string, such as r"\d+".
• re.fullmatch needs the whole text to match, while re.search and finditer look inside a longer text.
• Parentheses make a capture group that you read with group(n) or a name, and \1 in a replacement refers to the same captured text.
• Greedy quantifiers take as much as they can, and adding a question mark makes them lazy.
• A failed search returns None, so check the result before you read a match.
• Backtracking can take time that doubles with each extra character when a pattern has many ways to split the same text. Test a pattern on a long input of the wrong kind before you trust it with user data.