Skip to content

Testset#

The testset contains the tests that will be used to judge a problem, and can be decomposed into two components:

  • Its skeleton: which are tests are samples? how other tests are grouped?
  • How to generate tests for each group? Are the tests manually defined? Are they generated by a script?

The test skeleton can be specified in the problem.rbx.yml file through the testcases field. This field is a list of TestcaseGroup objects, which describe a group of tests.

Every test group should have an unique name. The name will be used to identify the group of tests when running rbx commands. There's a special reserved name, samples, which will be used to identify the group of samples.

Below, an example of a very simple, ICPC-style test plan: just two groups, one secret, with tests hidden to the user and one with the samples.

problem.rbx.yml
# ...
testcases:
  - name: 'samples'
  - name: 'secret'

Of course, we have to add tests to these groups. The rest of this section will be devoted to this topic.

There are 5 different ways of adding tests to group:

Method

Field

Description

List of testcases

testcases

A list of Testcase objects that manually
defines a few testcases.

A testcase glob

testcaseGlob

A path glob (string) that matches a set of testcase
inputs (.in files).

A list of generator calls

generators

A list of GeneratorCall objects,
which describe how to generate the tests for the group.

A static generator script
(aka a testplan)

generatorScript

A path to a .txt file -- each of its lines is a
generator call for a testcase.

A dynamic generator script

generatorScript

A path to a code or script that, when called, will generate
a static generator script.

In this section, we'll talk about the two most recommended approaches: using a testcase glob and using a generator script.

Defining the testset#

Testcase glob#

Testcase globbing is the simplest way of adding manually defined tests to a group.

problem.rbx.yml
testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'

In the example above, we define a group of samples which will contains tests matching the glob tests/*.in.

Thus, if there are 3 files in the tests directory, tests/01.in, tests/02.in and tests/03.in, all these three will be added to the samples group.

Test ordering

The order of the tests will be the lexicographical order of the files.

Be careful to not define tests as 1.in, 2.in, ..., 10.in as this will lead to a test set where test 10.in is executed before test 2.in.

Instead, define the tests as 01.in, 02.in, ..., 10.in, using leading zeroes.

Generator script#

If you haven't read the Generators section yet, you should read it before proceeding.

A generator script is a script that will be used to generate tests for a group.

It can be either a static script (in which case we also call it a testplan) or a dynamic script, and can be specified through the generatorScript field of a test group.

Static generator script (aka testplan)#

A static generator script (or a testplan) is a .txt file containing a list of line-separated generator calls.

A generator call is simply a pair of <generator-name> <generator-args...>, where <generator-args> is a list of space-separated arguments to pass to the generator.

Testplan can also have lines starting with a #, denoting this line is a comment and should be ignored, or even empty lines.

Below there's an example of a testplan for a problem that has two generators, random and small, and how to define it in the problem.rbx.yml file.

# Two random tests
random 10 100
random 10 1000

# Two small tests
small 50
small 100
testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
  - name: 'secret'
    generatorScript:
      path: 'testplan.txt'

Dynamic generator script#

A dynamic generator script is a code that produces a testplan. Think of a code (in Python, or even in C++) that produces a testplan file as its output.

Below, there's an example of a dynamic generator script for a problem that has a random generator.

for i in range(10):
    print(f"random {i}")
testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
  - name: 'secret'
    generatorScript:
      path: 'testplan.py'

The script spits a testplan with exactly 10 random tests, each one generated from a different argument between 0 and 9.

Advanced testplan features#

The static generator script (testplan) supports a few advanced features that can be useful in some scenarios.

Manual tests (@input)#

You can define manual tests directly in the testplan using the @input directive. This is useful when you want to add some edge cases that are hard to generate with a generator.

You can specify the input content using a quoted string (single or double quotes) or a block of braced content.

# Single line input (not raw)
@input "10 20\n"

# Multi-line input (triple quotes, raw)
@input """10
20
30"""

# Multi-line input (braces, raw, lines around the input are stripped)
# Recommended approach.
@input {
10
20
30
}
Copying tests (@copy)#

You can include existing test files in the testplan using the @copy directive. This is useful when you have some manual tests in a directory and you want to include them in the testplan.

@copy tests/manual/01.in
@copy tests/manual/02.in
Grouping tests (@testgroup)#

You can group tests using the @testgroup directive. This is useful when you want to add the specified tests only to testgroups that have this name. This is useful when different testgroups import the same testplan, but you want to create separate tests for each group.

# This test will be part of any testgroup that imports this testplan.
random 5

@testgroup subtask1 {
    # These tests will be part only of testgroups that have the name 'subtask1' and
    # import this testplan.
    random 10
    random 20
    @input "5"
}

@testgroup subtask2 {
    # These tests will be part only of testgroups that have the name 'subtask2' and
    # import this testplan.
    random 100
}
A shared, problem-level generator script#

When a single testplan -- usually partitioned with @testgroup blocks like above -- drives generation for every group, you don't have to repeat the generatorScript field on each one. Set it once at the problem level, and any test group (or subgroup) that does not define its own test parameters (testcases, testcaseGlob, generators, generatorScript) and has no subgroups of its own will inherit it as a default.

# problem.rbx.yml
generatorScript:
  path: "testplan/main.txt"

testcases:
  - name: "subtask1"   # inherits the problem-level generatorScript
  - name: "subtask2"   # inherits the problem-level generatorScript
  - name: "manual"
    testcaseGlob: "tests/*.in"   # has its own parameters, so it does NOT inherit

A group that sets any of its own test parameters always overrides the default.

What about the outputs?#

Until now, we've just generated the inputs of our testcases. What about the outputs? Where they come from?

By default, rbx will use the model solution to generate the outputs of the testcases. This will be done when building the testset.

The model solution is the topmost accepted solution in the solutions field of the problem.rbx.yml file.

In some cases, though, it's useful to specify a different output than the one generated by the model solution. You can create an .ans file in the same directory as the .in file to specify the answer for the problem, and override the one provided by the model solution (if it exists).

Notice this will affect both the testset itself, and the sample output rendered in the statement.

There are times, though, where you only want to modify the statement output without affecting the testset (think of cases where the model solution output reveals too much about the intended solution). You can create an .out file in the same directory as the .in file to specify the output for the problem.

You can only create manually crafted outputs for testcases you've defined manually (with a testcase glob, for instance).

Let's look at the file tree above, and assume we have a testcase glob for samples such as tests/*.in.

- tests/
  - 01.in
  - 01.out  # This will override the output in the statement.
  - 02.in
  - 02.ans  # This will override the answer both in the statement AND in the testset.

Outputs that aren't answers#

An .out file still has to be a correct answer: rbx runs the checker over it when you build with validation on, so a typo there fails the build instead of shipping.

Sometimes that's not what you want -- the output you'd like to show is deliberately not a checkable answer (an abridged listing, a hand-written illustration of the format). Name the file .out.statement instead of .out and rbx takes it at face value: it goes into the statement, and neither the checker nor the output validators ever see it. The same works on the input side with .in.statement, which replaces the input shown in the statement without touching the test that is actually judged.

- tests/
  - 01.in
  - 01.in.statement   # Shown in the statement; 01.in is what gets judged.
  - 01.out.statement  # Shown in the statement; never checked.

Taking a file at face value is a real risk, though: nothing catches a .out.statement whose format drifts away from what the problem actually specifies. Set validateStatementFiles on the group to get the format guarantee back without the checker:

testcases:
  - name: "samples"
    testcaseGlob: "tests/*.in"
    validateStatementFiles: true

With this on, the group's validators run over every .in.statement file and its output validators run over every .out.statement file. The checker still stays out of it -- skipping it is the whole point of the extension. This is mostly worth setting on samples, which is where .statement files usually live.

What about interactive problems?#

For interactive problems, you can create a .interaction file in the very same path (and with the very same name) as the .in files you've manually defined.

This file should contain the interaction between the solution and the interactor. Each line of interaction should start with either < or > to denote whether this was the output of the interactor or the output of the solution, respectively. One space after the marker is optional: rbx drops it, so < 3 and <3 mean the same line. It keeps any further spaces.

< this is an interactor line
< followed by another line
> this is the solution

Like .in.statement and .out.statement, an .interaction file is statement-only. It takes the place, in the statement only, of the interaction rbx captures when it runs the model solution against the interactor. The test itself doesn't change: rbx still runs the model solution on it, and still judges every solution against it.

So you can show a hand-written exchange that a correct solution would never produce, and the statement builds before you've written any solution at all.

Building the testset#

The command below can be used to build the testset.

rbx build

This command will build the testset, using the generator scripts to generate the tests for each group. All tests will be written to the build/tests directory, which you can inspect manually in our file system.

This command also accepts an extra verification flag (-v), which you can use to control whether validators will be run after generating the tests or not. The flag defaults to -v0, which means no verification will be done.

rbx build -v1

You can read more about the verification level flag in the verification section and about validation in the Validators section.

Allowing duplicate tests#

While it builds, rbx compares every test with the ones before it, across all groups, and warns when two of them are the same. rbx run reports each pair as a duplicate-test issue, and other commands, such as rbx build, list them when they finish. Each one gives both tests and why they match:

test main/3 (testplan/main.txt:4) is generated by the same call as main/1 (testplan/main.txt:2) (random 5)
test main/4 (random 7) has the same input as main/2 (random 3)

The first kind means the same generator call makes both tests. The second means their inputs are byte-identical. A duplicate is usually a slip, such as a generator that ignores its seed or a line pasted twice, and each one costs judging time without testing anything new.

Sometimes, though, you want the same test twice. A subtask may reuse a small case from an earlier one, or a random generator may produce the same tiny input more than once. You can mark one test, or a whole group, as allowed to repeat another test. rbx only warns about a pair of duplicates when neither test allows it, so marking one side of the pair is enough.

We recommend marking tests one by one. A group-wide flag also hides the accidental duplicates the warning exists to catch.

To mark one test, end its testplan line with @allow-duplicate. It works on generator calls, @copy lines and @input lines:

random 10 @allow-duplicate
@copy tests/manual/01.in @allow-duplicate
@input "5" @allow-duplicate
@input {
10
20
} @allow-duplicate

Each of these four tests may now repeat another test without a warning. rbx reads the annotation only as the last word of the line, and removes it before calling the generator, so random still receives 10. On a multi-line @input, put it right after the closing } or """.

Warning

Don't follow @allow-duplicate with a comment. A generator call or a @copy line takes everything after its first word as arguments or as the path, comments included, so random 10 @allow-duplicate # reused passes @allow-duplicate # reused to the generator. Put the comment on a line of its own.

To mark a whole group, set allowDuplicate on it:

problem.rbx.yml
testcases:
  - name: "subtask2"
    generatorScript:
      path: "testplan/subtask2.txt"
    allowDuplicate: true

Every test of subtask2 may now repeat another test without a warning. A group can also hold subgroups, each with tests of its own. A subgroup takes the allowDuplicate of its group, unless it sets its own.

Promoting tests#

Tests don't always stay where you first put them.

A generated test can be special enough to freeze into a file exactly as it is. An input you typed while debugging can deserve a place in the testset. Promoting a test puts it into a test group, usually moving it there, and rbx testcases promote does the editing for you.

# Freeze the test on line 3 of the testplan into a @copy, on the same line.
rbx testcases promote testplan.txt:3 --to secret --as copy

# Move the fourth test of `secret` into the samples.
rbx testcases promote secret/3 --to samples

# Turn an input file into a test of `secret`.
rbx testcases promote file:debug.in --to secret

Each command starts with the tests to promote, and --to says which group they go into. A selector says which test to promote:

Selector The test
testplan.txt:3 the one whose statement starts on line 3 of that testplan
secret/3 the fourth test of the secret group, counting from 0 as rbx build does
finding:2 finding 2 of the last stress test
file:debug.in an input file, and file:debug.in:debug.ans to keep an answer with it

Leave out the tests, --to, or both, and rbx asks for them. To promote into a group that doesn't exist yet, pass --new-manual NAME --glob GLOB or --new-script PATH instead of --to; see rbx testcases promote in the reference.

Choosing where a test goes#

A test can go into two kinds of group, and each holds it differently:

  • A group defined by a testcase glob gets a copy of the test's input, numbered after the group's last file, so that it sorts last: after tests/03.in comes tests/04.in. --name picks the name yourself.
  • A group backed by a testplan gets a line: the generator call that makes the test, when it has one, or a @copy of its input otherwise. --as recipe and --as copy pick one.

A @copy needs its input somewhere on disk, and rbx follows the @copy lines the group already has. If secret holds @copy tests/secret/edge-001.in and @copy tests/secret/edge-002.in, the new file is tests/secret/edge-003.in. A group with no @copy lines yet keeps its files in tests/<group>/.

Pass --path to choose yourself, either a file or a pattern whose * rbx fills with the next free name.

When several groups share one testplan, an untagged line would belong to every one of them. So rbx writes the line inside the group's @testgroup block, adding a @testgroup <group> block if the group has none yet.

Moving, freezing, and copying#

By default, a promoted test moves: rbx deletes its line from the testplan, or its files from a testcase glob folder. A @copy file goes too, unless another test still reads it. Pass --keep-source to copy the test instead.

Stress findings and input files are always copied. Some tests can't be moved, because rbx can't remove them where they are: those a group lists one by one in its own testcases field or makes with its generators field, and those made by a dynamic generator script. Promoting one of them fails unless you pass --keep-source.

A test that is already in the group's testplan changes in place: promoting a generator call as a @copy freezes it on the same line, so no other test moves. With --keep-source, the new line goes at the end of the group instead, as anything else does.

An answer is copied along with its input. A glob test's .ans file goes with it, and the answer of a file:IN:ANS is written next to the new input, where testcase globs and @copy both find it.

Previewing a promotion

--dry-run prints what a promotion would change without changing it, and --list-sinks lists the groups a test can go into. With --json, both print their result as JSON, which is what the VS Code extension reads to show you a promotion before making it.

Visualizing the testset#

You can use the rbx ui to visualize the testcases that were built through the rbx build command.

rbx ui

This command will start an interactive UI in your terminal which you can use to browse the testset.