Secrets in code and git history

Why this matters

A secret is a value that proves who you are or gives access to something. Examples are a password, an API key (a long string that lets a program call a service as you), an access token, a private key, and a database address that contains a password. A program often needs a secret to work. The risk comes when the secret is written into the code and the code is shared.

Git is the tool most developers use to save and share code. It keeps a record of every version of a project, so a secret written into a file and saved is saved in that record too. Deleting the file later does not remove the record. Many leaks happen this way, and the developer who made the mistake often does not know it until someone else finds the key.

This lesson covers three things. It shows how a key ends up in git history, why deleting the file does not remove the leak, and the steps to rotate a key, which means replacing it with a new one and disabling the old one. The password and two-factor lessons in this module cover the accounts that these keys belong to. The phishing lesson covers how keys are sometimes taken from people instead of from code.

The idea: git keeps every version you saved

Git stores a project as a list of snapshots. Each snapshot is called a commit. A commit records the state of the files at one moment, plus a short note describing the change. The full list of commits is the history. You can look back at any commit and see the files as they were then.

Three words will come up often. A repository, or repo, is a project folder that git tracks. A commit is one saved snapshot. A push sends your commits to a shared copy of the repo, usually on a hosting site. When someone copies a repo to their own computer, the copy includes the whole history, not just the latest files.

That last point is the heart of the problem. A secret is stored in every commit where it was present, and every copy of the repo has every commit. The current version of a file is only the newest entry in the list.

Example 1: a key in a file, and how a scanner finds it

Suppose a developer writes a settings file for a shop app, and puts the payment key directly into it. The key below is a made-up value with the same shape as a test key from a payment service. The first file is the version that was committed. Save it as config.py inside a folder called project_v1.

APP_NAME = "orders"
DATABASE_URL = "postgres://app:not-a-real-password@localhost/orders"
PAYMENT_API_KEY = "sk_test_EXAMPLE0000000000000000"
DEBUG = True

Anyone who can read this file can use the key. The file is only a few lines long, which is typical. Keys often appear in config files, test files, scripts and notebooks, and the developer may not notice when they add one.

A scanner is a program that searches files for text that looks like a secret. Real scanners check many patterns. This small one checks three: an assignment to a name such as api_key, secret, token or password, a payment-style test key that starts with sk_test_, and an AWS-style access key id. Save it as scan.py.

import re
import sys
from pathlib import Path
 
PATTERNS = [
    ("assignment to a secret-looking name", re.compile(
        r"""(?i)(api_?key|secret|token|password)\s*=\s*["'][^"']+["']""")),
    ("payment-style test key", re.compile(r"sk_(test|live)_[A-Za-z0-9]+")),
    ("AWS-style access key id", re.compile(r"AKIA[0-9A-Z]{16}")),
]
 
root = Path(sys.argv[1])
found = 0
for path in sorted(root.rglob("*")):
    if not path.is_file():
        continue
    for number, line in enumerate(path.read_text().splitlines(), start=1):
        for label, pattern in PATTERNS:
            if pattern.search(line):
                found += 1
                print(f"{path.relative_to(root)}:{number}: {label}")
                print(f"    {line.strip()}")
print("matches:", found)

Run it on the folder with python3 scan.py project_v1. The output is:

config.py:3: assignment to a secret-looking name
    PAYMENT_API_KEY = "sk_test_EXAMPLE0000000000000000"
config.py:3: payment-style test key
    PAYMENT_API_KEY = "sk_test_EXAMPLE0000000000000000"
matches: 2

The scanner reports line 3 twice, once for each pattern that matched. It did not report line 2, the database address. That line holds a password, but the scanner has no pattern for database addresses. A scanner is a net with holes in it. It helps you find common mistakes, but a clean scan does not prove that a project has no secrets.

Scanners also report things that are not secrets. The folder ex3 below holds a file with three lines. The first sets a placeholder password, which is a fake value that is meant to be changed later. The second sets a time limit that is a number, not a secret. The third is a hint, not a password.

DEFAULT_PASSWORD = "changeme"
TOKEN_TTL = 3600
PASSWORD_HINT = "a pet name"

Running python3 scan.py ex3 on that folder prints:

defaults.py:1: assignment to a secret-looking name
    DEFAULT_PASSWORD = "changeme"
matches: 1

The scanner flags the placeholder, because it has the same shape as a real password. It does not flag TOKEN_TTL, because a number is not in quotes, and it does not flag PASSWORD_HINT, because the name has more text after the word password. A flagged line is a question to answer. A person decides whether the value is real.

Scanners are also useful in the other direction. Run one on a folder before you commit, and you catch the key before it enters the history. Catching it at that point costs a few seconds. Catching it later costs a lot more, as the next example shows.

Example 2: deleting the file does not remove the leak

Now suppose the developer notices the key and fixes it in two steps. First they change the code to read the key from the environment, which means from a setting that the computer or the hosting service provides. Second, they delete the file that held the key. The project then looks clean. Yet the key is still in history.

To show why, here is a simple model of snapshots. It is not git, and it does not use git's storage. It keeps three snapshots in a list, the same way git keeps commits in order. Each snapshot is a dictionary from file name to file text. Save it as history_sim.py.

# A simulation of the idea behind git history, not git itself.
# Each "commit" is a snapshot: a dict from file name to file text.
commits = []
 
commits.append({
    "app.py": "print('orders app')\n",
    "config.py": 'PAYMENT_API_KEY = "sk_test_EXAMPLE0000000000000000"\n',
})
commits.append({
    "app.py": "print('orders app')\n",
    "config.py": "import os\nPAYMENT_API_KEY = os.environ['PAYMENT_API_KEY']\n",
})
commits.append({
    "app.py": "print('orders app v2')\n",
    # config.py was deleted in this snapshot
})
 
latest = commits[-1]
print("files in the latest snapshot:", sorted(latest))
 
for number, snapshot in enumerate(commits, start=1):
    for name, text in snapshot.items():
        if "sk_test_" in text:
            print(f"snapshot {number} still contains the key in {name}")
 
print("key present in latest snapshot:", any("sk_test_" in t for t in latest.values()))
print("key present in any snapshot:", any("sk_test_" in t for s in commits for t in s.values()))

Its output is:

files in the latest snapshot: ['app.py']
snapshot 1 still contains the key in config.py
key present in latest snapshot: False
key present in any snapshot: True

The latest snapshot has no key, which is what the developer sees when they open the project. The first snapshot still has it. Deleting a file in a later snapshot only changes the list from that point on. It does not reach back into the earlier entries.

The same is true of git. In a real repo, you can search the whole history for the key text with a command such as git log -S "sk_test_" --all --oneline. That command lists each commit where the text was added or removed. You can view the old file with git show <commit>:config.py, using the commit id from the list. The output depends on your project, so none is shown here.

This also explains why a leak spreads. Every person who copied the repo before the fix has the old commits on their computer. A hosting site may also keep them. So the key must be treated as exposed, even though the current code does not contain it.

Example 3: reading the key from the environment

The fix in the code is to read the key from the environment. The environment is a set of named values that the operating system or the hosting service gives to a program when it starts. The code then holds the name of the setting, not the value. The value lives outside the repo.

Save this file as app.py in a folder with no key in it.

import os
 
key = os.environ.get("PAYMENT_API_KEY")
if key is None:
    raise SystemExit("PAYMENT_API_KEY is not set")
print("key loaded, length", len(key))

The function os.environ.get returns the value, or None when the setting does not exist. The check for None makes the program stop with a clear message, instead of failing later with a confusing error.

Run it without the setting first. The output is:

PAYMENT_API_KEY is not set

The exit code is 1, which means the program ended with an error. Now run it with the setting, using a made-up value. The name is set just for this one command, so it is not saved anywhere.

PAYMENT_API_KEY=sk_test_EXAMPLE0000000000000000 python3 app.py

The output is:

key loaded, length 31

The program prints the length of the key, not the key. Printing the key itself would put it into a log file, a terminal history or a screenshot, where other people could read it. Printing only a length is enough to check that the setting exists.

In the hosting service, or in your shell, the value is set once. It is not written into the repo. A file called .env is often used on a developer's own computer to hold such values. Add .env to the .gitignore file, which lists files that git should not track, so the file is never committed.

Removing the key from history

Once a key has been committed, there are three different actions. They do different things, and only one of them removes the key from the list of commits.

1. Deleting the file, then committing. This removes the key from the newest snapshot. The key stays in every older commit and in every copy of the repo.
2. Rewriting the history. The git documentation points to a separate tool called git-filter-repo as the replacement for the older filter-branch command. It can remove a file from every commit. It changes the commit ids, so everyone who has a copy must make a new one, and the rewritten history must be pushed over the old one. Copies that were made before the rewrite still hold the key.
3. Rotating the key. This does not remove the key from history. It makes the old key useless, by replacing it with a new one and disabling the old one. This is the step that actually protects you.

Rewriting history is worth doing for a clean repo, but it is not the fix for a leaked key. The fix is rotation. Rewriting only removes the copy that you can reach.

How to rotate a key

Rotating a key means replacing it. Follow these steps in order.

1. Confirm the exposure. Find every place the key was committed, using a scanner and a history search. Note who could read the repo. A private repo that only your team can read has a smaller risk than a public one. Treat any key pushed to a public repo as taken, because automated programs search public repos for key patterns.
2. Create a new key in the provider's dashboard or settings page. Keep the old key active for now, so the app keeps working.
3. Store the new key where the app reads it. That is the hosting service's environment settings, or a secret store, not the code.
4. Deploy or restart the app so it uses the new key. Check that the app works.
5. Revoke the old key. This is the step that makes the leak harmless. Once revoked, requests with the old key are refused.
6. Review the provider's logs for use of the old key. Look at the time from when the key was first committed until it was revoked. Use of the old key by an unknown address is a sign that someone else used it.
7. Remove the key from the current code, and record the change in a normal commit. Rewriting history is optional, and does not replace steps 2 to 5.

The order matters. If you revoke first, the app stops working before the new key is in place. If you never revoke, the old key still works for anyone who has it.

Common mistakes

Deleting the file and calling the leak fixed. A commit that removes the file does not change the history. The key is still in every older commit. Rotation is the fix.

Adding the file to .gitignore after it was committed. The .gitignore file only stops git from tracking files that are not yet tracked. A file that git already tracks keeps being tracked after it is listed there. To stop tracking it, run git rm --cached config.py, then commit. That step does not remove the old commits either.

Running the app with a value that is not set. The app from example 3 stops with a clear message when the setting is missing:

PAYMENT_API_KEY is not set

The exit code is 1. A common reaction is to paste the key directly into the code to make the error go away. Doing that puts the key back into the repo, so the same leak starts again.

Printing the whole key to check it. A key printed to a log or a terminal can be copied by anyone who sees it later. Print its length or a masked form instead, as the example does.

Trusting a scanner alone. The scan in example 1 missed the database password in the line DATABASE_URL. Scanners only find patterns they know. Use one as a first check, and also review changed files by eye before you commit.

Rotating without revoking. If the old key still works, the leak is still open. A new key only helps when the old one is turned off.

Leaving the key in test files and examples. Test data and documentation are often copied from real code, and they end up with real keys. Use fake values in these files, and check them with the same scanner.

Practice

Exercise 1. The history simulation in example 2 has three snapshots. In which snapshots does config.py contain the key? Explain why deleting the file in snapshot 3 does not change this answer.

Show the answer

Only snapshot 1 contains the key. Snapshot 2 replaced the line with a read from the environment, and snapshot 3 deleted the file. Deleting the file in snapshot 3 only changes the newest snapshot. The list of earlier snapshots is kept as it was, so the key stays in snapshot 1.

Exercise 2. Run the scanner from example 1 on a folder that holds one file, settings.py, with this line in it:

API_TOKEN = "abc123"

Which pattern matches, and which line number is reported?

Show the answer

The first pattern matches, the assignment to a secret-looking name. The name API_TOKEN contains the word token, and the scanner ignores capital letters. The scanner prints settings.py:1: assignment to a secret-looking name, followed by the line itself. The output of this check was matches: 1.

Exercise 3. Write a program that reads the setting DB_PASSWORD from the environment. If the setting is missing, it should stop with a clear message. If it is present, it should print only its length.

Show the answer

import os
import sys
 
password = os.environ.get("DB_PASSWORD")
if password is None:
    sys.exit("DB_PASSWORD is not set. Set it in your shell or your deploy settings.")
print("password loaded, length", len(password))

Run without the setting, the program prints DB_PASSWORD is not set. Set it in your shell or your deploy settings. and exits with code 1. Run with DB_PASSWORD=example-only-1234, it prints password loaded, length 17, and exits with code 0.

Exercise 4. Put these rotation steps in the right order: (a) revoke the old key, (b) create a new key, (c) deploy the app with the new key, (d) check the provider's logs for use of the old key.

Show the answer

The order is (b), (c), (a), (d). Create the new key first, so there is something to switch to. Deploy the app with it, so the app works. Revoke the old key, which stops it working for anyone who has it. Then review the logs for use of the old key. Logs cover the time before revocation, and any attempts made after it show up there too.

Exercise 5. Explain in two or three sentences why the app in example 3 prints the length of the key and not the key itself.

Show the answer

A printed value can end up in a log file, a terminal history or a screenshot, and anyone who reads those can copy it. The length is enough to check that the setting is present and not empty. A printed key stays exposed for as long as those copies exist.

Summary

• Git keeps every commit, so a secret that was committed stays in the history after the file is deleted.
• Every copy of the repo, made before or after the fix, can contain the old commits.
• Read secrets from the environment or a secret store, and keep the values out of the repo.
• Use a scanner before you commit, and do not rely on it as a full check.
• Rotate a leaked key by creating a new one, deploying it, and revoking the old one.
• Rewriting history can remove the copy you can reach, but it does not make the old key safe.