Testset#

The testset contains the tests that will be used to judge a problem, and can be decomposed into two components:

Its skeleton: which are tests are samples? how other tests are grouped?
How to generate tests for each group? Are the tests manually defined? Are they generated by a script?

The test skeleton can be specified in the problem.rbx.yml file through the testcases field. This field is a list of TestcaseGroup objects, which describe a group of tests.

Every test group should have an unique name. The name will be used to identify the group of tests when running rbx commands. There's a special reserved name, samples, which will be used to identify the group of samples.

Below, an example of a very simple, ICPC-style test plan: just two groups, one secret, with tests hidden to the user and one with the samples.

problem.rbx.yml

# ...
testcases:
  - name: 'samples'
  - name: 'secret'

Of course, we have to add tests to these groups. The rest of this section will be devoted to this topic.

There are 5 different ways of adding tests to group:

Method	Field	Description
List of testcases	`testcases`	A list of `Testcase` objects that manually defines a few testcases.
A testcase glob	`testcaseGlob`	A path glob (string) that matches a set of testcase inputs (`.in` files).
A list of generator calls	`generators`	A list of `GeneratorCall` objects, which describe how to generate the tests for the group.
A static generator script (aka a testplan)	`generatorScript`	A path to a `.txt` file -- each of its lines is a generator call for a testcase.
A dynamic generator script	`generatorScript`	A path to a code or script that, when called, will generate a static generator script.

In this section, we'll talk about the two most recommended approaches: using a testcase glob and using a generator script.

Defining the testset#

Testcase glob#

Testcase globbing is the simplest way of adding manually defined tests to a group.

problem.rbx.yml

testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'

In the example above, we define a group of samples which will contains tests matching the glob tests/*.in.

Thus, if there are 3 files in the tests directory, tests/01.in, tests/02.in and tests/03.in, all these three will be added to the samples group.

Test ordering

The order of the tests will be the lexicographical order of the files.

Be careful to not define tests as 1.in, 2.in, ..., 10.in as this will lead to a test set where test 10.in is executed before test 2.in.

Instead, define the tests as 01.in, 02.in, ..., 10.in, using leading zeroes.

Generator script#

If you haven't read the Generators section yet, you should read it before proceeding.

A generator script is a script that will be used to generate tests for a group.

It can be either a static script (in which case we also call it a testplan) or a dynamic script, and can be specified through the generatorScript field of a test group.

Static generator script (aka testplan)#

A static generator script (or a testplan) is a .txt file containing a list of line-separated generator calls.

A generator call is simply a pair of <generator-name> <generator-args...>, where <generator-args> is a list of space-separated arguments to pass to the generator.

Testplan can also have lines starting with a #, denoting this line is a comment and should be ignored, or even empty lines.

Below there's an example of a testplan for a problem that has two generators, random and small, and how to define it in the problem.rbx.yml file.

testplan.txtproblem.rbx.yml

# Two random tests
random 10 100
random 10 1000

# Two small tests
small 50
small 100

testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
  - name: 'secret'
    generatorScript:
      path: 'testplan.txt'

Dynamic generator script#

A dynamic generator script is a code that produces a testplan. Think of a code (in Python, or even in C++) that produces a testplan file as its output.

Below, there's an example of a dynamic generator script for a problem that has a random generator.

testplan.pyproblem.rbx.yml

for i in range(10):
    print(f"random {i}")

testcases:
  - name: 'samples'
    testcaseGlob: 'tests/*.in'
  - name: 'secret'
    generatorScript:
      path: 'testplan.py'

The script spits a testplan with exactly 10 random tests, each one generated from a different argument between 0 and 9.

Advanced testplan features#

The static generator script (testplan) supports a few advanced features that can be useful in some scenarios.

Manual tests (`@input`)#

You can define manual tests directly in the testplan using the @input directive. This is useful when you want to add some edge cases that are hard to generate with a generator.

You can specify the input content using a quoted string (single or double quotes) or a block of braced content.

# Single line input (not raw)
@input "10 20\n"

# Multi-line input (triple quotes, raw)
@input """10
20
30"""

# Multi-line input (braces, raw, lines around the input are stripped)
# Recommended approach.
@input {
10
20
30
}

Copying tests (`@copy`)#

You can include existing test files in the testplan using the @copy directive. This is useful when you have some manual tests in a directory and you want to include them in the testplan.

@copy tests/manual/01.in
@copy tests/manual/02.in

Grouping tests (`@testgroup`)#

You can group tests using the @testgroup directive. This is useful when you want to add the specified tests only to testgroups that have this name. This is useful when different testgroups import the same testplan, but you want to create separate tests for each group.

# This test will be part of any testgroup that imports this testplan.
random 5

@testgroup subtask1 {
    # These tests will be part only of testgroups that have the name 'subtask1' and
    # import this testplan.
    random 10
    random 20
    @input "5"
}

@testgroup subtask2 {
    # These tests will be part only of testgroups that have the name 'subtask2' and
    # import this testplan.
    random 100
}

What about the outputs?#

Until now, we've just generated the inputs of our testcases. What about the outputs? Where they come from?

By default, rbx will use the model solution to generate the outputs of the testcases. This will be done when building the testset.

The model solution is the topmost accepted solution in the solutions field of the problem.rbx.yml file.

In some cases, though, it's useful to specify a different output than the one generated by the model solution. Think of cases where there are multiple possible correct outputs, but the one given by the model solution reveals too much about the intended solution.

In these cases, you can create a .ans file in the very same path (and with the very same name) as the .in files you've manually defined. You can only create manually crafted outputs for testcases you've defined manually (with a testcase glob, for instance).

Let's look at the file tree above, and assume we have a testcase glob for samples such as tests/*.in.

- tests/
  - 01.in
  - 01.ans
  - 02.in

What about interactive problems?#

For interactive problems, you can create a .interaction file in the very same path (and with the very same name) as the .in files you've manually defined.

This file should contain the interaction between the solution and the interactor. Each line of interaction should start with either < or > to denote whether this was the output of the interactor or the output of the solution, respectively.

< this is an interactor line
< followed by another line
> this is the solution

Building the testset#

The command below can be used to build the testset.

rbx build

This command will build the testset, using the generator scripts to generate the tests for each group. All tests will be written to the build/tests directory, which you can inspect manually in our file system.

This command also accepts an extra verification flag (-v), which you can use to control whether validators will be run after generating the tests or not. The flag defaults to -v0, which means no verification will be done.

rbx build -v1

You can read more about the verification level flag in the verification section and about validation in the Validators section.

Visualizing the testset#

You can use the rbx ui to visualize the testcases that were built through the rbx build command.

rbx ui

This command will start an interactive UI in your terminal which you can use to browse the testset.