The site judge: how your code is tested here

Why this matters when you build with AI

An AI tool can write a solution in seconds, and you can submit it in seconds more. The verdict comes back, and it is only as useful as your knowledge of what the judge did. A Wrong answer may come from a formatting rule, not from the algorithm. A fast-looking solution may pass the scale check while printing wrong answers, because that check does not compare output. A Runtime error may come from a memory cap that you did not know about.

This lesson describes how this site judges code. It uses the source code of the site as the reference, so every number here can be checked in lib/judgeCore.ts (the judging loop), lib/judgeSpec.ts (per-problem limits), lib/compare.ts (output matching), lib/runner.ts (limits and commands) and lib/sandboxRunner.ts. When a model writes code for you, you can now check it against these rules: the compile command, the limits, and the output rules. That turns a guess into a test.

Words used in this lesson

A judge is the program that runs submissions and compares their output with the expected output. On this site, the judging loop is judgeTests in lib/judgeCore.ts. The local runner in lib/runner.ts and the cloud runner in lib/sandboxRunner.ts both call it.

A compile step turns source code into a program. C++, C, Java, Go, Rust, C#, Kotlin and Swift need one. Python, JavaScript, TypeScript, PHP and Ruby do not, because their programs run from source.

stdin is the input stream a program reads. The judge writes the test input into it. stdout is the output stream the program writes to. The judge reads it after the run and compares it with the expected output.

Wall-clock time is the time you would measure on a clock on the wall. It includes waiting for the operating system. CPU time is the time the processor spent on your program. The site limits both.

Address space is the range of memory addresses a process may reserve. The site caps it for most languages, as lesson 07 explained.

A sandbox is a throwaway virtual machine with no network access. The site can run code there instead of on its own server. The section on the two runner paths explains both.

A scale check is a second run, on bigger inputs, that happens only after a submission passes every test. It checks speed, not answers.

Supported languages and their commands

The runner supports 13 languages: Java, C++, Python, JavaScript, TypeScript, C, Go, Rust, C#, Kotlin, Swift, PHP and Ruby. A problem offers only the languages that have a starter file for it, so the list on each problem page is the real list for that problem.

The table below comes from the SPECS table in lib/runner.ts. The build column is the compile step. The run column is the command that runs your program. The file name is the one the judge writes your code to.

Three things in this table matter in practice.

The C++ flags are -std=c++17 and -O2. The judge does not add -Wall, so warnings are not requested. Code that needs C++20 will fail to compile. For example, the three-way comparison operator <=> is rejected with expected primary-expression, as the test later in this lesson shows.

The judge's compile step runs the compiler once. If the compile succeeds, its warnings are discarded. Only a failed compile shows its messages, and then the verdict screen shows them under the heading Compile error.

The table lists the commands in the source. The live runner must also have each compiler or runtime installed. The sandbox code checks for this before it compiles, and if the tool is missing it returns a compile error that says the language is not installed on the live runner yet. Try another language in that case.

Time and memory enforcement

These are the limits in the source code. They apply to each step separately.

A problem can set its own limits. Then limits.timeMs is the time for C++, and each language gets that time times a factor, capped at 20 seconds: 1 for C, C++ and Rust, 1.5 for Go and Swift, 2 for Java, Kotlin and C#, 3 for JavaScript and TypeScript, and 5 for PHP, Python and Ruby. A problem can override any factor. A problem that sets limits.memoryMb replaces the 1 GiB or heap cap with that many MiB, times an optional per-language factor, capped at 4096. A problem with no limits keeps the defaults in the table. The judge shows no memory verdict either way, so a memory failure is a Runtime error.

Wall-clock time is the limit that decides most time limit verdicts. A program that sleeps or waits on the disk also uses the five seconds, even if it uses little CPU time.

Address space is not the same as the memory a program uses. A runtime can reserve a large range of address space and use little of it. That is why Java, JavaScript, TypeScript, C#, and Kotlin do not get the address space cap. They get a CPU time cap and their own heap limit instead. A C++ program that reserves 1.6 GB of address space fails under the cap, which is the case in lesson 07.

The program runs in a clean environment. On the server path, the judge passes only PATH, HOME and LANG, plus the settings a language needs, so a program cannot read the server's secrets from its environment. For you, this means a program cannot rely on any other variable.

Output has two limits. Only the first 20000 characters of stdout are kept for the verdict, so a correct answer longer than that no longer matches and is Wrong answer. On the local runner, a process that writes more than 40000 characters is also killed, and the status is Runtime error with an empty output panel. The cloud sandbox runner has no 40000 kill and only cuts the output at 20000.

The test flow

The judge follows the same steps for every submission.

1. It writes your code to the file name in the table.
2. If the language has a compile step, it runs the compile step once. A failure ends the judge here, and the verdict has a compileError with the compiler's messages.
3. For each test in order, it starts a new process with the test input on stdin. Each test starts with a fresh program, so a value left in memory by test 1 cannot reach test 2.
4. It gives the test a status, using the order below.
5. It runs every test, even after a failure, so you see the full list. The one exception is a problem with subtasks, where a test is marked Skipped, without running, once every subtask it belongs to has failed.
6. If every test passed and the problem has a scale check, it runs the scale check.

The status of one test is set in this order, from judgeTests in lib/judgeCore.ts. First, if the run was stopped by the time limit, the status is Time limit exceeded. Second, if the run did not finish with exit code zero, or the process was killed, the status is Runtime error. Third, the output is checked. If the problem has a custom checker, the checker program decides: exit code 0 is Passed, exit code 1 is Wrong answer, and anything else is Judge error, a fault in the problem and not in your code. With no checker, the output is compared with the expected output and the status is Passed or Wrong answer.

This order has two effects you should remember. A program that prints the right answer and then exits with code 1 is Runtime error, because the exit code is checked before the output. And a program that crashes is never called Wrong answer, even if its output before the crash was right.

A problem with subtasks also gets a score out of 100. Each subtask lists test numbers and scores its points only if all of its tests pass. A problem still counts as solved only when every test passes.

Each problem has a list of tests. The site passes 2 as the number of visible tests, so the first two are shown: you see their input, the expected output and your output. The rest are hidden. For hidden tests, the screen shows only the status.

Output comparison

The judge compares your output with the expected output using the problem's compare mode, in lib/compare.ts. A problem with no compare mode uses the exact mode, which first does a small cleanup with the function norm. The cleanup is this.

1. Windows line endings, which are a carriage return and a line feed, become a plain line feed.
2. Spaces and tabs at the end of each line are removed.
3. Blank lines at the end of the whole output are removed.

After this cleanup, the two outputs must be equal character by character in the exact mode. Nothing else is ignored. Spaces inside a line count. Leading spaces count. Case counts. Blank lines in the middle count.

Here is what that means in practice. Five programs were run on the judge for the warm-up problem of adding two integers, with the input 2 3 and the expected output 5. Their results were these.

• A program that prints 5 with a trailing space passes. The trailing space is removed.
• A program that prints 5 and then two extra blank lines passes. The blank lines at the end are removed.
• A program that prints 05 fails with Wrong answer. The characters are different.
• A program that prints 5 with a leading space fails with Wrong answer. A leading space is not removed.
• A program that prints 1 2 with two spaces in the middle, for the made-up input 1 2 with the expected output 1 2, fails with Wrong answer. The Your output box shows two spaces, and the difference is only in the number of spaces, so you must look closely.

A problem can choose another mode. tokens compares whitespace-separated tokens, however they are laid out. sorted-tokens accepts the same tokens in any order, and sorted-lines the same lines in any order. float compares tokens, and two numbers match if the difference is at most eps times the larger of 1 and the expected value, with eps at 1e-6 by default and at most 0.1; other tokens must match exactly. Instead of a mode, a problem can use a checker. The examples above and below describe the exact mode.

The last point is important for output that the statement says is "separated by spaces". The judge does not treat two spaces as one. If the statement says one space, print one space. If you are unsure, print exactly the format of the sample output.

For local checks, the same cleanup can be written in Python. The checker below runs a command, gives it the input, applies the same cleanup to both outputs, and returns the status. It reads the input and the expected output from two files.

import subprocess
 
 
def norm(text):
    lines = text.replace("\r\n", "\n").split("\n")
    return "\n".join(line.rstrip() for line in lines).rstrip("\n")
 
 
def check(cmd, input_path, expected_path):
    with open(input_path) as f:
        data = f.read()
    with open(expected_path) as f:
        expected = f.read()
    try:
        result = subprocess.run(cmd, input=data, capture_output=True, text=True, timeout=5)
    except subprocess.TimeoutExpired:
        return "Time limit exceeded"
    if result.returncode != 0:
        return "Runtime error"
    return "Passed" if norm(result.stdout) == norm(expected) else "Wrong answer"

Call it with the command as a list, for example check(["python3", "-I", "sol.py"], "in.txt", "out.txt"). It returns one of four statuses. It is a rough local copy of the exact mode, not the site's judge, so it does not apply the memory and CPU limits. Its job is to catch the output rules early.

The scale check

After a submission passes every test, a problem with a scale check runs bigger inputs. The sizes are listed in the problem data. The runner starts with the smallest size and moves up. It stops at the first run that times out or exits with a non-zero code.

The scale check measures time and crashes only. It does not compare output. The code in runScale checks the run's ok flag, which is true only for exit code zero and no time-out. This is the most important fact in this section, so here is a real test that shows it.

The problem counts pairs of positions i less than j with a_i + a_j equal to S. Its tests use only small inputs, with n up to 25. The scale sizes are 1000, 4000, 16000 and 64000. Three programs were submitted to the judge, with these results.

The first program checks every pair. It passed all six tests. Its scale runs took 50 ms at size 1000, and 512 ms at size 4000. At size 16000 it ran past 5 seconds and was stopped. The judge's note was:

Correct, but it ran out of time on an input of size 16,000. The best known approach is O(n). Try for a faster approach.

The second program counts pairs with a dictionary, in one pass. It passed all six tests, and its scale runs took 20 ms, 25 ms, 26 ms and 49 ms. The judge's note was:

Correct and fast. Its time grew about linearly. The best known approach is O(n).

The third program is wrong on large inputs. When n is above 100 it prints 0, and it is correct on the small tests. It passed all six tests and all four scale sizes, because the scale check does not look at its output. The judge's note was the same as the second program's, which says "Correct and fast". That note is true about the speed and false about the answer.

This is why a passing scale check does not prove your answer is right. Use the scale check as a timing signal, and test the answers on your own large inputs with a brute force and a random generator.

The note is built from the time of each run and the best known complexity written in the problem data. The best known approach is shown only when the problem has one. The note's first sentence changes with the result, so read the whole note.

The verdict screen

After a run, the verdict screen shows the following, in this order.

1. An error line, if the request failed. For example, a language that is not offered for the problem returns an error.
2. A Compile error heading with the compiler's messages, if the compile failed.
3. A summary line: "All tests passed.", or the count such as "2 of 4 tests passed."
4. A speed note, when a scale check ran.
5. A list with one line per test: Test 1: Passed, Test 2: Wrong answer, and so on. Hidden tests carry the word hidden.
6. For each visible test that did not pass, three boxes: Input, Expected and Your output. For a Runtime error, the Your output box holds the error text from the program's error stream.

The screen does not show the time of each test. Only the scale check gives timings, and it reports the time of each size, not of each test. For the time of your own tests, measure locally with time or with the timeout command, as lesson 07 showed.

Some of the screen depends on the plan. The source says that on the free plan, JavaScript and Python run only the examples, in the browser, and the screen says so. Hidden tests, the speed note and XP for solving are part of the Practice plan and above. The AI review of a solution is part of Job-ready and above. The plans are named Starter, Practice, Job-ready and Interview-ready in lib/plans.ts. Check the current prices and included features on the plans page, because they change.

The two runner paths

The code has two ways to run a submission. The choice is made by the site operator with an environment setting, and a learner does not choose it.

On the sandbox path, each submission runs in a throwaway virtual machine with the network denied. The judge creates it, writes the files, compiles and runs the tests, and then stops the machine. When the operator turns this path on, signed-in learners' runs are counted against a daily limit that resets at midnight India time. When the limit is used up, the site says so and does not run the code.

On the server path, the code runs on the server that serves the site. The source says in its comment that this runs untrusted code on the host, so the server must be a container or a virtual machine with no secrets. On a Vercel deployment this path is refused unless the sandbox is on. In that case the site shows that running code is not enabled on this deployment.

On the sandbox path, a passing submission is also saved. The judge compares it with other saved solutions for the same problem and language, and close matches are recorded for review. The saving happens after the verdict is shown, and it does not change the verdict. Write your own code. Using someone else's solution is a way to lose a contest and it is not learning.

A site operator can also pause running code for everyone with a feature flag. In that case the site returns a message that running code is paused, and you can try again later.

Hands-on exercise: submit a warm-up

The module has six warm-up problems. They are easy on purpose. Their job is to show you the verdict screen and the output rules, so you can read a verdict without guessing. The statements are below. Submit your solution to the matching problem on the site when that problem is open to you. If the problem page does not show it yet, run the local checker on the sample input and the sample output shown with each warm-up.

Each warm-up has a sample input and the output a correct program prints for it. Read the statement, write a solution in the language you use, run it on the sample, and then submit. Read the verdict line by line before you change anything.

Warm-up 1, sum. One line holds two integers a and b, with -10^9 <= a, b <= 10^9. Print a + b.

Sample input:

2 3

Sample output:

5

Warm-up 2, evens. The first line holds n, with 1 <= n <= 100. The second line holds n integers, each from -1000 to 1000. Print how many of them are even. A negative even number counts as even.

Sample input:

5
1 2 3 4 6

Sample output:

3

Warm-up 3, reverse. One line holds a word of lowercase letters, with a length from 1 to 50. Print the word with its letters in reverse order.

Sample input:

lesson

Sample output:

nossel

Warm-up 4, maximum. The first line holds n, with 1 <= n <= 100. The second line holds n integers, each from -1000 to 1000. Print the largest value.

Sample input:

4
3 -7 12 5

Sample output:

12

Warm-up 5, palindrome. One line holds a word of lowercase letters, with a length from 1 to 50. Print YES if the word reads the same forwards and backwards, and NO if it does not.

Sample input:

level

Sample output:

YES

Warm-up 6, distinct. The first line holds n, with 1 <= n <= 100. The second line holds n integers, each from -5 to 5. Print the number of different values. Equal values count once.

Sample input:

5
2 2 3 -1 3

Sample output:

3

For each warm-up, the solution below is one correct Python version. It was tested on the judge with five or six generated tests, and every test passed. Write your own version first, then compare.

Show the answer

Warm-up 1:

a, b = map(int, input().split())
print(a + b)

Warm-up 2:

n = int(input())
a = list(map(int, input().split()))
print(sum(1 for x in a if x % 2 == 0))

Warm-up 3:

print(input().strip()[::-1])

Warm-up 4:

n = int(input())
print(max(map(int, input().split())))

Warm-up 5:

s = input().strip()
print("YES" if s == s[::-1] else "NO")

Warm-up 6:

n = int(input())
print(len(set(map(int, input().split()))))

Common mistakes

Printing extra text. A debug line such as print("here") adds a line to the output, and the judge compares it. Remove every debug print before you submit. Writing to stderr is safer during debugging, because the judge shows stderr only for a failed run.

Changing the spacing of the format. Two spaces where the sample has one is a Wrong answer. The judge does not collapse spaces, so match the sample exactly.

Reading all input with one line function. A program that calls input() once, when the statement has two lines, reads the wrong values. The verdict is often Runtime error or Wrong answer, depending on the code after the read. Read the input in the order the statement gives.

Trusting a passing scale check. The scale check does not compare output. A program can be fast and wrong. Test answers on your own large inputs.

Ignoring a Runtime error. A Runtime error has a message in the output panel. Read it. A memory message points to the memory cap, a stack message points to recursion depth, and a Python traceback names the line.

Using C++20 features. The judge compiles C++ with -std=c++17. A feature from a later standard fails to compile, and the screen shows a Compile error. Check the feature against C++17 before you use it.

Assuming the judge matches your local compiler. Your machine may use a different compiler version or flags. Compile locally with the judge's exact flags, from the table, before you submit.

Checklist

• Choose the language the problem offers, and check its file name and compile command in the table.
• Compile locally with the same flags, and read every warning.
• Run the sample and compare the output with the sample, character by character.
• Remove debug output from stdout before you submit.
• Read the verdict line for each test, then the Input, Expected and Your output boxes for the visible tests.
• For a Runtime error, read the message before changing anything.
• For a time limit, check the operation count and time a large input with timeout 5.
• After a scale check, read the whole note. It does not prove the answers are right.

Summary

This site compiles each submission with a fixed command for the language, runs every test in a new process with a 5 second wall-clock limit, caps memory by address space for most languages and by heap settings for managed runtimes, and compares output, by default after removing line-ending differences and trailing spaces. A test is Passed only when the exit code is zero and the output matches. The scale check runs only after every test passes, and it measures time and crashes, not answers. The verdict screen shows the status of each test, with the first two tests in full.

Practice

These practice items use only the rules and the table from this lesson. Work out each answer before you open it.

1. A program prints 7 with a trailing space on a line, and the expected output is 7. A second program prints 7 and then a blank line, and a third prints 7 with a leading space. Say which of the three passes, and give the reason from the output rules.

   Show the answer

   The first passes, because trailing spaces on a line are removed. The second passes, because blank lines at the end of the whole output are removed. The third fails with Wrong answer, because a leading space is not removed, so the two outputs differ.

2. A problem has a scale check with sizes 1000, 10000 and 100000. The runs at 1000 and 10000 finish, and the run at 100000 is stopped at the 5 second limit. Name the scale check's outcome, and say which size the note will name.

   Show the answer

   The scale check stops at the run that timed out. The note says the program is correct but ran out of time on an input of size 100,000, and it adds the best known approach if the problem data lists one. The answers at sizes 1000 and 10000 are not checked by the scale check, so the note does not confirm them.

3. A C++ program crashes on test 2 with a segmentation fault, and the exit code is 139. Tests 1 and 3 pass. Name the status for test 2, the status shown on the summary line, and the first local check you would run.

   Show the answer

   Test 2 shows Runtime error. The summary line says "2 of 3 tests passed.", and it does not say all tests passed. The first local check is to build with -fsanitize=address,undefined -g and run the input for test 2, which points to the bad access. If the input is large, check the recursion depth as well, as lesson 07 showed with the 8 MB stack.

4. You want a solution to run on the Java run step from the table. Explain why a memory test under prlimit --as=1073741824 does not tell you what the judge will do for Java, and say what limit does apply to Java in the source.

   Show the answer

   The judge does not apply the address space cap to Java. It applies a CPU time cap of 10 seconds, and it starts the Java process with -Xmx256m, which sets a heap limit of 256 MB. A test under the 1 GiB address space cap would measure a limit that Java does not get, and it could fail a program that would run fine with the heap limit. For Java, test the heap use with a large input and the -Xmx256m flag.

5. The scale check passed a solution that prints the wrong answer for large inputs. Describe a test you can run on your own machine that would catch the error before you submit, and say what it must compare.

   Show the answer

   Write a brute force that is slow but obviously right, and a generator that makes inputs of a size where the brute force still finishes, such as a few thousand values. Run both programs on the same inputs and compare their outputs with the cleanup from this lesson, which removes line-ending differences and trailing spaces but keeps everything else. The check must compare the answers, not the timings. Repeat it on sizes from small to large, because the wrong branch in the example only starts above a threshold.