Testset#

The testset contains the tests that will be used to judge a problem, and can be decomposed into two components:

Its skeleton: which are tests are samples? how other tests are grouped?
How to generate tests for each group? Are the tests manually defined? Are they generated by a script?

The test skeleton can be specified in the problem.rbx.yml file through the testcases field. This field is a list of TestcaseGroup objects, which describe a group of tests.

Every test group should have an unique name. The name will be used to identify the group of tests when running rbx commands. There's a special reserved name, samples, which will be used to identify the group of samples.

Below, an example of a very simple, ICPC-style test plan: just two groups, one secret, with tests hidden to the user and one with the samples.

problem.rbx.yml

# ...
testcases:
  - name: 'samples'
  - name: 'secret'

Of course, we have to add tests to these groups. The rest of this section will be devoted to this topic.

There are 5 different ways of adding tests to group:

Method	Field	Description
List of testcases	`testcases`	A list of `Testcase` objects that manually defines a few testcases.
A testcase glob	`testcaseGlob`	A path glob (string) that matches a set of testcase inputs (`.in` files).
A list of generator calls	`generators`	A list of `GeneratorCall` objects, which describe how to generate the tests for the group.
A static generator script (aka a testplan)	`generatorScript`	A path to a `.txt` file -- each of its lines is a generator call for a testcase.
A dynamic generator script	`generatorScript`	A path to a code or script that, when called, will generate a static generator script.

In this section, we'll talk about the two most recommended approaches: using a testcase glob and using a generator script.

Defining the testset#

Testcase glob#

Testcase globbing is the simplest way of adding manually defined tests to a group.

problem.rbx.yml

testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'

In the example above, we define a group of samples which will contains tests matching the glob tests/*.in.

Thus, if there are 3 files in the tests directory, tests/01.in, tests/02.in and tests/03.in, all these three will be added to the samples group.

Test ordering

The order of the tests will be the lexicographical order of the files.

Be careful to not define tests as 1.in, 2.in, ..., 10.in as this will lead to a test set where test 10.in is executed before test 2.in.

Instead, define the tests as 01.in, 02.in, ..., 10.in, using leading zeroes.

Generator script#

If you haven't read the Generators section yet, you should read it before proceeding.

A generator script is a script that will be used to generate tests for a group.

It can be either a static script (in which case we also call it a testplan) or a dynamic script, and can be specified through the generatorScript field of a test group.

Static generator script (aka testplan)#

A static generator script (or a testplan) is a .txt file containing a list of line-separated generator calls.

A generator call is simply a pair of <generator-name> <generator-args...>, where <generator-args> is a list of space-separated arguments to pass to the generator.

Testplan can also have lines starting with a #, denoting this line is a comment and should be ignored, or even empty lines.

Below there's an example of a testplan for a problem that has two generators, random and small, and how to define it in the problem.rbx.yml file.

testplan.txtproblem.rbx.yml

# Two random tests
random 10 100
random 10 1000

# Two small tests
small 50
small 100

testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
  - name: 'secret'
    generatorScript: 'testplan.txt'

Dynamic generator script#

A dynamic generator script is a code that produces a testplan. Think of a code (in Python, or even in C++) that produces a testplan file as its output.

Below, there's an example of a dynamic generator script for a problem that has a random generator.

testplan.pyproblem.rbx.yml

for i in range(10):
    print(f"random {i}")

testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
  - name: 'secret'
    generatorScript: 'testplan.py'

The script spits a testplan with exactly 10 random tests, each one generated from a different argument between 0 and 9.

What about the outputs?#

Until now, we've just generated the inputs of our testcases. What about the outputs? Where they come from?

By default, rbx will use the model solution to generate the outputs of the testcases. This will be done when building the testset.

The model solution is the topmost accepted solution in the solutions field of the problem.rbx.yml file.

In some cases, though, it's useful to specify a different output than the one generated by the model solution. Think of cases where there are multiple possible correct outputs, but the one given by the model solution reveals too much about the intended solution.

In these cases, you can create a .out file in the very same path (and with the very same name) as the .in files you've manually defined. You can only create manually crafted outputs for testcases you've defined manually (with a testcase glob, for instance).

Let's look at the file tree above, and assume we have a testcase glob for samples such as tests/*.in.

- tests/
  - 01.in
  - 01.out
  - 02.in

Building the testset#

The command below can be used to build the testset.

rbx build

This command will build the testset, using the generator scripts to generate the tests for each group. All tests will be written to the build/tests directory, which you can inspect manually in our file system.

This command also accepts an extra verification flag (-v), which you can use to control whether validators will be run after generating the tests or not. The flag defaults to -v0, which means no verification will be done.

rbx build -v1

You can read more about the verification level flag in the verification section and about validation in the Validators section.

Visualizing the testset#

You can use the rbx ui to visualize the testcases that were built through the rbx build command.

rbx ui

This command will start an interactive UI in your terminal which you can use to browse the testset.