Reading a verdict: WA, TLE, RE, MLE

Why this matters when you build with AI

An AI tool that writes contest code gives you a program and a guess that it is right. The judge gives you a verdict. A verdict is a single word, such as Wrong answer, and it names a symptom. It does not name the line that caused it.

The usual mistake is to paste the verdict back to the model with no other detail and ask for a fix. The model then changes the algorithm, adds a cache, or rewrites the input reader, with no evidence about which part was wrong. Sometimes the new code passes by luck. Often it moves the bug.

This lesson gives you a checklist for each verdict and a way to reproduce it on your own machine. You will be able to say which line is wrong before you ask anyone, a person or a model, for help.

Words used in this lesson

A verdict is the judge's result for one submission. The judge runs your program on test inputs and compares the output with the expected output. Each test gets a status, and the submission gets one overall verdict.

WA means wrong answer. The program ran to the end and printed something that does not match the expected output.

TLE means time limit exceeded. The program was still running when the time limit ran out. Lesson 08 gives the exact limits used on this site.

RE means runtime error. The program stopped with an error, such as a crash, or it exited with a non-zero exit code. The exit code is the number a program returns when it ends. Zero means it finished normally. Any other number means something went wrong.

MLE means memory limit exceeded. The program used more memory than the limit allowed.

An uninitialised value is a variable that was declared but never given a value. In C++ it holds whatever bytes were in that memory before, so its value can change from run to run or from one compiler setting to another.

An out-of-bounds access reads or writes an array or vector at an index it does not own, such as index n in an array of n items. C++ does not check this on a normal build.

The stack is the memory where a function keeps its local variables while it runs. Each recursive call adds a new frame to it. The heap is the memory a program asks for while it runs, for example with new or with a vector that grows.

A sanitizer is a compiler option that adds checks to the program while it runs. It stops at the first bad memory access and prints where it happened. It makes the program slower, so you use it on your own machine, not for timing.

Address space is the range of memory addresses a process is allowed to reserve. It is larger than the memory the program actually uses. A program that reserves a large range can hit a limit that is set on address space even when most of that range is empty.

What this site reports

The judge on this site has no separate memory word. Each test gets one of these statuses: Passed, Wrong answer, Runtime error, Time limit exceeded, Judge error or Skipped. A compile failure is shown separately as Compile error. The source is judgeTests in lib/judgeCore.ts, which both the local runner and the cloud sandbox use. A test gets its status in this order.

If the program was stopped for running too long, the status is Time limit exceeded.
If the program exited with a non-zero code, or was killed, the status is Runtime error.
Otherwise the output is checked. If the problem has a custom checker, the checker decides: Passed, Wrong answer, or Judge error when the checker itself fails, which is a fault in the problem and not in your code. With no checker, the output is compared with the expected output using the problem's compare mode, and the status is Passed or Wrong answer.
When a problem has subtasks, a test is marked Skipped, without running, if every subtask it belongs to has already failed.

So a program that runs out of memory shows as Runtime error on this site. The error text is in the output panel. If the text says std::bad_alloc, the cause is memory. Check the text, not only the status word.

The status order also means that a program which prints the right answer and then crashes is reported as Runtime error, not as Passed, because the exit code is checked before the output. Passed needs an exit code of zero as well as the right output.

Other contest sites use an explicit MLE verdict. This lesson uses WA, TLE, RE and MLE as a checklist, because the causes are the same wherever you submit.

The idea: a verdict names a symptom

Think of a verdict as a doctor's first note. "Fever" tells you where to look, not what the illness is. You then run a test, and the test points to a cause.

For each verdict, the work is the same. First you form a guess about the cause. Then you make a small input or a tool run that would show the cause if the guess is right. Then you fix the line and run the same check again.

The rest of the lesson takes the four verdicts in turn. For each one you get the common causes, a local way to reproduce it, and an example with real output from the code this lesson was checked against. The output will differ on your machine in details such as process ids, memory addresses and the wording of system messages. The shape of the output will be the same.

Wrong answer: the program ran and printed something else

Wrong answer is the most common verdict. The program did not crash and did not run out of time. It printed the wrong text.

Here is an original problem for the example. The first line holds t, the number of test cases. Each test case has a line with n, then a line with n integers. For each test case, print how many of the later values are larger than the first value. Constraints: 1 <= t <= 10 and 1 <= n <= 1000, with values from 0 to 10^9.

The sample input is:

The right output is 1 and then 0. In the first test case the first value is 5, and only 7 is larger. In the second test case no value is larger than 2.

Here is a program that does not give that output.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int t;
    cin >> t;
    int count = 0;
    while (t--) {
        int n;
        cin >> n;
        vector<long long> a(n);
        for (auto &x : a) cin >> x;
        for (int i = 1; i < n; i++)
            if (a[i] > a[0]) count++;
        cout << count << "\n";
    }
}

Compiled with g++ -std=c++17 -O2 -Wall -Wextra, it prints:

1
1

The second line should be 0. The counter count is declared outside the loop, so the count from the first test case carries into the second one.

This kind of bug is not caught by a sanitizer. Nothing goes out of bounds and no memory is misused. You find it by reading the code against the statement, by running the sample, or by running a brute force program on small random inputs. The brute force method is taught in the stress testing lesson of this module, so here it is enough to know it exists.

Checklist for a wrong answer:

Run the sample input and compare by eye, then with diff.
Check every variable that should reset between test cases: counters, sums, flags and best values.
Check the boundary cases: n equal to 1, all values equal, zero and negative values when the statement allows them.
Check the output format: spaces, line breaks and the order of the answers.
Write a brute force that is slow but obviously right, and compare the two outputs on small random inputs.

Time limit exceeded: the program was too slow

A time limit verdict means the program was still running when the time was up. The cause is almost always an algorithm that does more work than the limit allows. Lesson 04 showed how to estimate that work from the constraints. The other common cause is slow input and output, covered in the fast input and output lesson.

Here is an original problem. The first line holds n and S. The second line holds n integers. Count the pairs of positions i and j with i < j and a_i + a_j = S. Constraints: 1 <= n <= 300000, values from 0 to 10^9, and S up to 2 times 10^9.

The sample input is:

4 5
1 4 2 3

The right output is 2, from the pairs (1, 4) and (2, 3).

This program checks every pair. It is correct, but it does about n times (n minus 1) divided by 2 steps.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    long long s;
    cin >> n >> s;
    vector<long long> a(n);
    for (auto &x : a) cin >> x;
    long long ans = 0;
    for (int i = 0; i < n; i++)
        for (int j = i + 1; j < n; j++)
            if (a[i] + a[j] == s) ans++;
    cout << ans << "\n";
}

To reproduce the verdict, make a large input with a generator. This Python script writes 300000 random values. It runs under python3 -I, which skips the user's site settings, so the result does not depend on your local setup.

import random
random.seed(7)
n = 300000
print(n, 1000000000)
print(" ".join(str(random.randint(0, 10**9)) for _ in range(n)))

Save it as gen.py and run python3 -I gen.py > big.txt. Then time the program with a hard stop at five seconds, which is the run limit on this site:

g++ -std=c++17 -O2 -o naive naive.cpp
timeout 5 ./naive < big.txt; echo "exit=$?"

On the machine this lesson was checked on, the output was:

exit=124

The exit code 124 comes from timeout. It means the limit was reached. The program did not print anything, which is common for TLE: the answer is only printed at the end.

The fix is to count pairs in one pass. For each value x, look up how many earlier values equal S - x, then record x. This program does that with std::map, a sorted tree that takes about log n steps per lookup, so the total is O(n log n).

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    long long s;
    cin >> n >> s;
    vector<long long> a(n);
    for (auto &x : a) cin >> x;
    map<long long, long long> seen;
    long long ans = 0;
    for (int j = 0; j < n; j++) {
        auto it = seen.find(s - a[j]);
        if (it != seen.end()) ans += it->second;
        seen[a[j]]++;
    }
    cout << ans << "\n";
}

On the same big file this version finished in 0.42 seconds. The checked version also matched a brute force program on 200 random small inputs, with no mismatches. The brute force was the naive pair loop in Python.

Checklist for a time limit:

Write the operation count from the constraints, using the budget in lesson 04.
Time the program with timeout on a large generated input, not on the sample.
Find the loop that runs the most times. Most of the time is spent there.
Check the input reader. A reader that calls cin for each value in a list of a million values can take most of the time. The fast input lesson covers this.
Check for work repeated inside a loop, such as sorting or copying a vector on every step.

Runtime error: the program stopped with an error

A runtime error means the program crashed, was killed, or returned a non-zero exit code. The most common causes are out-of-bounds access, a use of a value that was never set, a stack that grew too large, and a division by zero.

Here is an original problem. The first line holds n. The second line holds n integers. Print the sum of all values, computed with prefix sums. A prefix sum p[i] is the sum of the first i values, so p[0] is 0 and p[n] is the total.

The sample input is:

3
4 5 6

The right output is 15.

This program has an off-by-one error in its size. The vector p has n entries, but the loop writes p[n], which needs n + 1 entries.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    cin >> n;
    vector<long long> a(n);
    for (auto &x : a) cin >> x;
    vector<long long> p(n);
    p[0] = 0;
    for (int i = 1; i <= n; i++) p[i] = p[i - 1] + a[i - 1];
    cout << p[n] << "\n";
}

Built with g++ -std=c++17 -O2 and run on the sample, this program stopped with this message and exit code 134:

Fatal glibc error: malloc.c:2371 (sysmalloc): assertion failed: ...

Exit code 134 means the program stopped itself with an abort. The text after assertion failed is cut here. It means the heap was damaged. The exact wording depends on your C library version, so your message may differ. What matters is that the sample crashed.

Some out-of-bounds writes do not crash at all. The program then runs with a corrupted value and prints a wrong answer, or prints the right answer by chance. This is why you should not rely on a clean run.

The way to find this kind of bug is a sanitizer. Build the program with AddressSanitizer and run it on the same input:

g++ -std=c++17 -O1 -g -fsanitize=address,undefined -fno-omit-frame-pointer -o re_asan re.cpp
./re_asan < sample.txt

The sanitizer prints the first bad access and the line where it happened. On the machine this lesson was checked on, the important lines were:

==2808511==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x72428b9e0088 at pc 0x5d8b58d5fd97 ...
    #0 0x5d8b58d5fd96 in main .../re.cpp:12
SUMMARY: AddressSanitizer: heap-buffer-overflow .../re.cpp:12 in main

The process id, the addresses and the path will be different on your machine. The line number and the words heap-buffer-overflow point to the bad write. AddressSanitizer is a GCC feature documented in the GCC manual's instrumentation options page, and it works with -fsanitize=address.

A second common runtime error is a stack overflow. A recursive function uses stack memory for each call. If the depth is too large, the stack runs out and the program is killed by a segmentation fault, which has exit code 139.

This site runs C++ with the default stack size of the machine it runs on. The source does not raise the stack limit for C++ programs. On the shell used for this lesson, ulimit -s printed 8192, which is 8 MB. A test program that keeps about 80 bytes per call, with a depth of 10000, finished. The same program with a depth of 1000000 exited with code 139.

Checklist for a runtime error:

Check every index against the size of the array or vector. The largest valid index is size minus one.
Build with -fsanitize=address,undefined -g and run the failing input.
Check variables before their first use. Build with -Wall -Wextra to get warnings about them, as shown in the next section.
Check division and modulo by zero when a count can be zero.
For recursion, count the depth at the largest input. Turn a deep recursion into a loop, or use an explicit stack, when the depth can reach about 10^5 or more.
Remember that output that is too long goes wrong on this site: only the first 20000 characters are kept, and the local runner kills the program past 40000. The site judge lesson in this module gives the details.

Uninitialised values

An uninitialised value is one of the most common causes of wrong answers that change from run to run. The following program has one.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    cin >> n;
    long long best;
    for (int i = 0; i < n; i++) {
        long long x;
        cin >> x;
        if (i == 0 || x > best) best = x;
    }
    cout << best << "\n";
}

With n equal to 0, the variable best is never set. When compiled with g++ -std=c++17 -O2 -Wall -Wextra, the compiler warns:

warning: 'best' may be used uninitialized [-Wmaybe-uninitialized]
... note: 'best' was declared here

The exact location and wording depend on the compiler version. On the machine this lesson was checked on, the program printed 0 for n equal to 0. That was luck, not logic. The warning is the signal. Set best when it is declared, and use -Wall -Wextra on every compile you do locally.

Memory limit exceeded

On this site, the memory rule for C++ is a cap of 1 GiB on address space, which is 1073741824 bytes. The cap is set with prlimit --as. The prlimit tool shows or sets the limits a process runs under, and its help lists --as as the size of virtual memory. You can apply the same cap to your own runs.

The program below stores one long long for each cell of an n by n grid. The input is n, up to 14142 in this exercise.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    cin >> n;
    vector<long long> grid(static_cast<size_t>(n) * n, 0);
    grid[0] = 1;
    cout << grid.size() << " " << grid[0] << "\n";
}

With n equal to 2000 it uses about 32 MB and prints 4000000 1. Measured with GNU time, the maximum resident memory was about 35 MB, with /usr/bin/time -f "maxrss_kb=%M". Resident memory is the part of the address space that is actually in use, and it is the number to compare with a memory estimate.

With n equal to 14142, the program asks for about 1.6 GB. Under the judge's address cap it fails:

printf '14142\n' | prlimit --as=1073741824 --cpu=10 --fsize=10000000 -- ./mle

The output on the machine this lesson was checked on was:

terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc

On this site, that text appears in the output panel, and the status is Runtime error, not MLE. This is the case the verdict screen does not label for you. Read the message.

The fix is to store less per cell. An int takes 4 bytes instead of 8. The fixed program uses vector<int> for the same grid. With the same input and the same cap, it printed 199996164 1 and exited with code 0, because the grid takes about 800 MB.

The arithmetic is the same as in the memory section of lesson 04. Count the number of cells, multiply by the bytes per cell, and compare with the limit. Leave room for the rest of the program, because the cap covers all of it.

Checklist for memory:

Estimate the memory before you write the code: cells times bytes per cell.
Use the smallest type that holds the values. Use int when the values fit in 32 bits, as lesson 05 explains.
Measure the real maximum with /usr/bin/time -f "maxrss_kb=%M" on a large input.
Test with the judge's cap using prlimit --as=1073741824.
Do not store data you do not need. A rolling array that keeps only the last row can replace a full table.

Hands-on exercise: four broken programs

Here are four programs, one for each verdict. Each comes with a statement and a sample input. Your job is to predict the verdict on the sample, reproduce it on your machine with the tools from this lesson, and fix the program. Work through them in order, and write down each cause before you open the answer.

Program 1 is for the statement in the wrong answer section: the number of later values larger than the first one, for each test case.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int t;
    cin >> t;
    int count = 0;
    while (t--) {
        int n;
        cin >> n;
        vector<long long> a(n);
        for (auto &x : a) cin >> x;
        for (int i = 1; i < n; i++)
            if (a[i] > a[0]) count++;
        cout << count << "\n";
    }
}

Program 2 is for the statement in the time limit section: count the pairs with sum S, with the sample input 4 5 and 1 4 2 3. The program is the naive pair loop shown earlier. Use the generator to make a file with 300000 values, and run it under timeout 5.

Program 3 is for the statement in the runtime error section: the sum of values, computed with prefix sums, with the sample 3 and 4 5 6.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    cin >> n;
    vector<long long> a(n);
    for (auto &x : a) cin >> x;
    vector<long long> p(n);
    p[0] = 0;
    for (int i = 1; i <= n; i++) p[i] = p[i - 1] + a[i - 1];
    cout << p[n] << "\n";
}

Program 4 is for the memory section. The statement says: print the number of cells of an n by n grid and the value of the first cell. The constraint is 1 <= n <= 14142, and the judge gives 1 GiB of address space. The sample is 2000.

#include <bits/stdc++.h>
using namespace std;
 
int main() {
    int n;
    cin >> n;
    vector<long long> grid(static_cast<size_t>(n) * n, 0);
    grid[0] = 1;
    cout << grid.size() << " " << grid[0] << "\n";
}

For each program, answer these questions:

What is the verdict on the sample input, and what is the first line that shows the cause?
What is the verdict on the largest allowed input, and how did you reproduce it?
What is the smallest change that fixes it, and how do you know the fix is right?

Show the answer

Program 1, wrong answer. On the sample it prints 1 and then 1. The second line should be 0. The cause is that count is declared once, outside the loop, so it is never reset for the second test case. The fix is to declare int count = 0; inside the while loop, before the inner loop. The fixed program prints 1 and 0 on the sample. To check beyond the sample, write a brute force that gives the same answer and compare the two on small random inputs.

Program 2, time limit. The sample 4 5 with 1 4 2 3 gives 2, which is correct, so the sample passes. On a generated input of 300000 values, timeout 5 stops it and the exit code is 124. The cause is the pair loop, which does about 4.5 times 10^10 steps. The fix is a single pass with a map of values seen so far. The fixed version finished the same file in about 0.4 seconds. Your times will differ from these, but the exit code 124 should match when you use the same generator and the same five-second limit.

Program 3, runtime error. On the sample it crashes. The vector p has n entries, and the loop writes entry n, which is one past the end. On the machine this lesson was checked on, the program stopped with exit code 134 and a glibc message about the heap. With AddressSanitizer, the first error is a heap-buffer-overflow at the line p[i] = .... The fix is vector<long long> p(n + 1);. The fixed program prints 15.

Program 4, memory. On the sample 2000 it prints 4000000 1 and uses about 35 MB. On 14142 it needs about 1.6 GB of address space. Under prlimit --as=1073741824 it stops with std::bad_alloc. On the site it would show as Runtime error. The fix is to store one int per cell, which needs about 800 MB for 14142. The fixed program prints 199996164 1 under the same cap. Check that the rest of your program fits in the leftover memory before you submit.

Common mistakes

Reading the verdict word and not the message. Runtime error can mean a crash, a bad index, a memory failure or, on the local runner, a very long output. The error text in the output panel tells them apart. Read it before you change the code.

Testing only the sample. The sample is usually the easiest case. Most wrong answers, time limits and crashes show up only on larger or stranger inputs, and the sample will not show them. Generate inputs with your own generator.

Fixing the symptom. Adding a try block, catching a crash, or printing an extra line to get past the sample does not fix the bug. The verdict will come back on the next test.

Trusting a model's fix without a test. A model may say "this is an out-of-bounds bug" and be right, or it may change the algorithm when the bug was a missing reset. Run the failing input before and after the change, and check that the change matches the cause you found.

Using the sanitizer build for timing. Sanitizers slow the program down by a large factor. A program that is fast with -O2 alone can time out under -fsanitize. Time the program with -O2 only, and use sanitizers to find the bad access.

Forgetting that warnings are silent on the judge. A warning from -Wall is not a verdict. It is a signal you should act on before you submit, because it often points to an uninitialised value.

Counting only the main array. The judge's cap covers the whole process. Vectors, temporary strings and the output buffer all add to it.

Checklist

Read the verdict and the error text, and write down which category it is.
For WA, run the sample, check every reset, and compare with a brute force on small inputs.
For TLE, count the operations from the constraints, then time a large generated input with timeout.
For RE, check every index, build with -fsanitize=address,undefined -g, and check the recursion depth.
For MLE, estimate cells times bytes, measure with /usr/bin/time, and test under prlimit --as=1073741824.
Build with -Wall -Wextra and treat every warning about an uninitialised value as a bug until you have proved otherwise.
Make one change at a time, and run the same failing input again after each change.

Summary

A verdict is a symptom. WA means the output differs, TLE means the program ran too long, RE means it stopped with an error, and MLE means it used too much memory. On this site, memory failures appear as Runtime error, so the message text decides which one it is. For each verdict you can reproduce the cause on your machine with a sample run, a timed large input, a sanitizer build or a memory cap, and then make a small fix and check it again.

Practice

These practice items use only the tools and limits from this lesson. Work out each answer before you open it.

A learner reports three results from one program. Result A: the output is wrong on test 4, and the program printed a number with no crash. Result B: the program printed the right answer and then exited with code 1. Result C: the program ran for 8 seconds on test 7. Name the verdict on this site for each result, and give the first check you would run for each.

Show the answer

A is Wrong answer. Check the reset of every variable between test cases, then compare with a brute force on small inputs. B is Runtime error on this site, because a non-zero exit code fails the test even when the printed output matches. Check what the program does after printing, such as a read past the end of the input or an exception at the end. C is Time limit exceeded, since the run limit is 5 seconds. Count the operations from the constraints, then time the program on a large generated input.
The judge gives each C++ program 1 GiB of address space. Using 4 bytes per cell, find the largest n for which a plain vector<int> of n times n cells fits under that cap on its own, ignoring all other memory.

Show the answer

The cap is 1073741824 bytes, and each cell is 4 bytes, so the cells can use at most 268435456 bytes. The square root of 268435456 is 16384, so n can be at most 16384. In practice the answer is a little lower, because the program itself, its libraries and the output buffer also use address space. Test the largest value you plan to use with prlimit --as=1073741824 before you rely on it.
A recursive function is called with depth n, and each call keeps about 80 bytes on the stack. The machine has an 8 MB default stack. Estimate the largest depth that fits, then check it on your own machine by running the function with increasing depth and checking the exit code.

Show the answer

The stack holds about 8388608 bytes, and 8388608 divided by 80 is about 104857, so the estimate is a depth of around 100000 calls. The real limit can be lower, because the program also uses stack for main and the libraries. Measure it: run the program at depths such as 50000, 100000 and 200000, and note the first depth where the exit code is 139. Your number will depend on your machine and your compiler settings, so record your own result, not this one.