Local judging with official test data

The module on testing already covers building your own tests: a hand-made case, an edge case, and a maximum-size case, each one you build and check yourself. For a past CCC problem, there is a further kind of test available. It is the exact input and output files CEMC itself used to judge the problem, published once the contest is over.

What the official test data is

For every past contest, CEMC's past-contests page makes the official inputs and expected outputs available alongside the problem statements. Every input the grader judged your solution against is there. The exact output it expected in return sits alongside it, grouped by problem. This is not a substitute sample, and not a guess at what the grader might have used. It is the real data, for a problem whose contest window has already closed.

Keeping the files for one problem together, in a folder named after that problem, mirrors the way the past-contests page already groups them by year and by problem. The script below expects every input file and its matching output file to sit side by side. It only works cleanly once the files are organized this way. Doing that sorting once, when the files first arrive, saves having to work out which output belongs to which input every time the check is rerun later.

Why this data is worth running against

A hand-made test and an edge-case test both rely on you noticing which cases matter, then working out the correct answer yourself. The official test data removes both of those steps. It already covers every subtask boundary the problem's author intended, and its expected outputs are already correct by definition, since they are what the real grader used. Running a solution against every one of these files mirrors what the grader's per-subtask checking does. It is the closest offline equivalent there is, for a problem you can no longer submit to for a real verdict.

A mismatch on one of these files tells you something a sample alone cannot: which specific case your solution gets wrong, not merely that some case, somewhere, eventually would. If every small file matches but a large one does not, that points at a bound your solution handles correctly only up to a point. If one particular file fails while every other one, including larger ones, passes, that points at a specific case your plan overlooked, rather than a general problem with its approach.

Comparing exactly, against every file at once

The module on testing already covers comparing two outputs character by character, rather than by eye. Checking a whole problem's official test data means doing that same exact comparison once per file, rather than once for a single sample. A short Python script can loop over every input file for a problem, run your solution against each one in turn, and compare the result against that file's official output:

1import os2import subprocess3
4test_dir = "tests"5names = sorted(name[:-4] for name in os.listdir(test_dir) if name.endswith(".txt"))6
7for base in names:8    with open(f"{test_dir}/{base}.txt") as f:9        official_input = f.read()10    with open(f"{test_dir}/{base}.out") as f:11        official_output = f.read()12
13    result = subprocess.run(14        ["pypy3", "sol.py"],15        input=official_input,16        capture_output=True,17        text=True,18        check=False,19    )20
21    if result.stdout == official_output:22        print(base, "matches")23    else:24        print(base, "does NOT match")

For each input file, the script reads the matching official output file, runs the solution as a separate program, and checks whether what it printed matches that official output exactly. Reporting a plain "matches" or "does NOT match" for every file, rather than stopping at the first mismatch, is worth keeping. A problem can fail on more than one file at once. Seeing every result together shows whether the failures cluster around one case or are scattered across many.

Timing is only a hint here too

The module on complexity analysis already covers why a run on your own machine is not proof a plan is fast enough. Nothing about running against the official test data changes that. Your machine is not the grader's machine. Running your solution with plain CPython instead of PyPy makes it slower still, often by ten to fifty times, the same estimate the module on complexity analysis gives. A local timing only ever gives a rough sense of whether a solution sits in the right neighbourhood of fast enough, on the larger official files. Running every official test and confirming every output matches exactly is a strong local check on correctness. Treat any timing you notice while doing it as a hint worth following up on, never as the grader's own answer.

Recap

CEMC publishes the official input and expected-output files for past contests on its past-contests page, alongside each year's statements. Running a solution against every one of these files, and comparing each output exactly, checks that solution against real subtask boundaries and real expected answers. That goes well beyond what a hand-made test or a single sample can confirm on its own, and a mismatch points at exactly which case failed. A short script can automate running every file and reporting which ones match. Any timing noticed this way stays only a local hint, never the grader's own measurement.