Problem package#

This documentation goes over how each field (or group of fields) in problem.rbx.yml works.

Problem definition#

The three main required fields in a problem.rbx.yml package are:

name: supports anything matching the regex ^[a-zA-Z0-9\-]+$
timeLimit: the time limit of your problem, in milliseconds.
memoryLimit: the memory limit of your problem, in megabytes.

A barebones package would look something like:

name: "test-problem"
timeLimit: 1000  # 1 second TL
memoryLimit: 256  # 256 MB ML

Language Modifiers#

You can add extra language-based limit modifiers to problems.

name: "test-problem"
timeLimit: 1000  # 1 second TL
memoryLimit: 256  # 256 MB ML
modifiers:
  java:
    time: 2000  # 2 second TL for Java
    memory: 512  # 512 MB ML for Java

Task Type#

Field: type Schema: rbx.box.schema.TaskType

You can use this (optional) field to specify the type of your problem. If you're building an interactive problem, for instance, you should set this to COMMUNICATION, otherwise the default BATCH will be used.

name: "interactive-problem"
type: COMMUNICATION
timeLimit: 1000  # 1 second TL
memoryLimit: 256  # 256 MB ML

Variables#

Field: vars Schema: Dict[str, Union[str, int, float, bool]]

In the package definition you can define variables that can be referenced in statements, validators, checkers, interactors and stress tests.

This is useful to maintain consistency when changing constraints of your problem. A common mistake this field tries to solve is changing constraints in the statement, but not updating them in the other components.

vars:
  MAX_N: 10000
  MAX_A: 100000
  MOD: py`10**9+7` # Backticks force the var to be evaluated as a Python expression.

rbx automatically generates an rbx.h header file in the root of your package that you can include in your code to read these variables.

#include "rbx.h"

int main() {
  int MAX_N = getVar<int>("MAX_N");
  // Rest of your code...
}

If you're not using C++ for your components, consider doing so as the rbx experience is tightly integrated with testlib.

If not, you can still use the variables in validators, see the Validators section for more information.

Warning

Refrain from using variables in generators. Although it is tempting to use rbx.h in there, it is not recommended to do so.

Checker#

Field: checker Schema: rbx.box.schema.CodeItem

Checkers are also a very important part of your problem package, and define how the output of a solution will be judged.

The checker is controlled through the top-level parameter checker, and is optional. In the case it is not specified, rbx falls back to testlib's wcmp.cpp, which is a token-based checker that compares the output of the solution with the expected answer. In case of interactive problems, a checker is not required.

rbx is tightly integrated with testlib, and thus you can either specify:

The name of any checker in defined in testlib's checkers folder. The checker will be automatically downloaded when running solutions and building the package.
```
checker:
  path: "yesno.cpp"
```
Tip

If you want to explicitly download a checker from testlib, you can run rbx download checker [name-of-the-checker].
A custom testlib-based checker of your own, placed in the folder (or any sub-folder) of your package.
```
checker:
  path: "my-testlib-based-checker.cpp"
```
Tip

rbx automatically places testlib.h together with your code when compiling it, but you can explicitly download it with rbx download testlib if you want.

Recommended

This is usually the recommended solution when building a custom checker, as rbx provides a clear integration with testlib and you can use include rbx.h in your checker to read variables.
A custom checker (not necessarily using testlib). It can even be in other language, in which case we suggest specifying the language property.
```
checker:
  path: "my-custom-checker.py"
  language: "python"
```
Warning

Currently, it is not possible to use variables in custom checkers not written in C++.

Note

Although this is not a testlib-based checker, we still expect checker programs to follow the same command line structure as testlib, which is receiving and reading three file paths from the command line.

Thus, your checker program will be called as <program> <input-file> <output-file> <expected-answer-file>.

Interactor#

Field: interactor Schema: rbx.box.schema.Interactor

You can specify an interactor for your problem, or leave this field empty for non-interactive problems.

interactor:
  path: "my-interactor.cpp"

According to the ICPC package specification, used by the ICPC World Finals, interactors cannot be specified together with a checker, and an error will be thrown if both are present.

rbx enforces this rule, but supports interactive problems with a checker if you set the legacy flag in the interactor definition.

interactor:
  path: "my-interactor.cpp"
  legacy: true

Danger

rbx currently only supports testlib-based interactors. If you do not use a testlib-based interactor, rbx will not complain but might behave unexpectedly.

Warning

Notice that a few judges do not support interactors with a checker, and an error will be thrown if you try to build a package for an unsupported judge with both components specified.

Generators#

Field: generators Schema: List[Generator]

You can also specify a set of testcase generators that can be re-used in different places in your package (testgroup generation, stress tests to name a couple).

Again, we encourage the use of testlib-based generators, but give you the flexibility to develop however you like.

After implementing a generator, you can define it in problem.rbx.yml similar to how you define a checker, but you also have to name it to be able to reference it down the line.

generators:
  - name: "gen"
    path: "my-gen.cpp"

Notice also how the generators field is a list, and as such you can define multiple generators.

Danger

Refrain from using Variables in generators. Although it is tempting to depend on them, it is not a good practice.

This will make your generator sensible to the changes you make to the vars field, meaning if you found a testcase that breaks a specific solution, but it depends on a vars entry and you change it, you might end up with a totally different test. Instead, you should be able to describe all your generator parameters through static variables.

Solutions#

Field: solutions Schema: List[Solution]

You can specify multiple solutions to your problem, including incorrect ones to check that your testset is strong enough.

You can define them similarly to generators and checkers, but you also have to provide an expected outcome for them.

solutions:
  - path: "sols/main.cpp"
    outcome: accepted
  - path: "sols/slow.cpp"
    outcome: tle
  - path: "sols/wa.cpp"
    outcome: wa

Also, you have to define an accepted solution. The first accepted solution in this list will be considered the main solution to generate answers for the testcases.

For a full list of expected outcomes, see here.

Testcase groups#

Field: testcases Schema: List[TestcaseGroup]

You can define multiple testgroups for you problem. For each testgroup, you can define tests for it in five (5) different ways:

Specifying a sequence of manually defined testcases present in your package's directory with the testcases field.
```
testcases:
  - name: "group-1"
    testcases:
      - inputPath: "manual-tests/1.in"
        outputPath: "manual-tests/1.ans"
      - inputPath: "manual-tests/1.in"
```
Note

Notice how the outputPath is optional. If it is not defined, the main solution will be used to generate an output. This is the recommended approach.
Specifying a glob of input paths with the testcaseGlob field.
```
testcases:
  - name: "group-1"
    testcaseGlob: "tests/*.in" # (1)!
```
1. Pick all files ending in .in inside the tests/ folder as test inputs for the testgroup. The files are taken in lexicographically increasing order.

Specifying a list of generator calls with the generators field.

testcases:
  - name: "generated"
    generators:
      - name: "gen1"
        args: "1000 30"
      - name: "gen1"
        args: "1000 42"

Specifying a static generator script with the generatorScript field:
problem.rbx.ymlscript.txt
testcases: - name: "generated" generatorScript: path: "script.txt"
gen 1000 30 gen 1000 42 # ...other tests
Note

The .txt extension is necessary for rbx to identify this is a static script.

Recommended

This is usually the recommended approach to generate a few pre-defined testcases. Prefer this over the generators call to keep problem.rbx.yml cleaner.

Specifying a dynamic generator script with the generatorScript field:

problem.rbx.ymlscript.py

testcases:
  - name: "generated"
    generatorScript:
      path: "script.py"

# Generates 10 different cases with different parameters.
for i in range(10):
  print(f'gen 1000 {i}')

Recommended

This is usually the recommended approach to generate multiple random testcases.

Warning

If the platform you package the problem for does not support testgroups, tests will be flattened into a single group, and the tests will be executed in the order the groups were defined.

Samples#

You can also specify samples to be included in your statement by defining a testgroup named samples. This testgroup has to be the first one defined, otherwise an error will be raised.

Validators#

Field: validator, testcases.validator Schema: rbx.box.schema.CodeItem

You can specify testlib validators that check whether both manual and generated testcases are in the correct format.

You can specify validators in two places:

In the package definition, through the validator field.
```
validator:
  path: "validator.cpp"
```

In the testgroup definition, through the validator field, in which case this validator will be executed only for this group, and the package one will be ignored.

validator:
  path: "validator.cpp"
testcases:
  - name: "group-with-usual-constraints"
    # ...other testgroup definitions
  - name: "group-with-different-constraints"
    # ...other testgroup definitions
    validator:
      path: "validator-alternative.cpp"

You can pass variables to validators in two different ways.

(C++ only) Include the rbx.h header and using the getVar accessor.

#include "testlib.h"
#include "rbx.h"

using namespace std;

int main(int argc, char *argv[]) {
    registerValidation(argc, argv);

    int MAX_N = getVar<int>("MAX_N"); // Read from package vars.

    inf.readInt(1, MAX_N, "N");
    inf.readEoln();
    inf.readEof();
}

This header is automatically generated at your problem's root directory by rbx.

Recommended

This is the recommended approach of passing variables to validators.

Read the variables from the command line.

rbx passes all the variables defined in the vars section to the validator through --{name}={value} parameters. You are responsible for writing code to parse them.

import sys

variables = {}

for arg in sys.argv[1:]:
  if arg.startswith('--') and '=' in arg:
    name, value = arg[2:].split('=', 1)
    variables[name] = value

# Use variables...

Extra validators#

You can also specify extra validators to be used in your testcases, both at the problem-level and testgroup-level.

validator:
  path: "validator.cpp"
testcases:
  - name: "group-1"
    # ...other testgroup definitions
  - name: "group-2-without-solution"
    # a group whose testcases don't have a solution
    extraValidators:
      - path: "validator-no-solution.cpp"

Stress tests#

Field: stresses Schema: List[Stress]

You can pre-define stress tests with a few applications in mind, such as:

Finding testcases that break incorrect solutions;
Ensuring solutions you expect to be correct cannot be broken.

Let's break down each field in the stress test below:

stresses:
  - name: "stress1"
    generator:
      name: "gen"
      args: "--MAX_N=[1..1000] @"
    finder: "[sols/wa.cpp] ~ INCORRECT"

name: a name for the stress test. Useful when calling this test through rbx stress [name].
generator: a generator expression to be repeatedly evaluated in this stress test.
name: the name of the generator
args: args pattern to be passed to generator.
- You can pass a random integer by writing something like [1..1000].
- You can pass variables defined in the vars section with something like <MAX_N>.
- You can pass a random choice by writing something like (a|b|c).
- You can pass a random hex string by passing @.
finder: a finder expression that, when evaluated to true, consider the given generated test as a match

Unit tests#

Field: unitTests Schema: UnitTests

You can specify unit tests for your validator and your checker through the unitTests field.

unitTests:
  validator:
    - glob: "unit/validator/valid_*.in"  # (1)!
      outcome: VALID
    - glob: "unit/validator/invalid_*.in"  # (2)!
      outcome: INVALID
  checker: 
    - glob: "unit/checker/ac*"  # (3)!
      outcome: ACCEPTED
    - glob: "unit/checker/wa*"
      outcome: WRONG_ANSWER

Specify a glob to match manually crafted input files that when validated should be considered valid.
Specify a glob to match manually crafted input files that when validated should be considered INVALID.
Specify a glob to match files named unit/checker/ac*(.in|.out|.ans) that when checked should be considered ACCEPTED.

Validator globs are really simple: they should match .in files relative to the problem root directory. Those files will be passed to the validator program and validated.

Checker globs are a bit more complex, since they accept three different parameters:

A .in file, the input for the testcase;
A .out file, the output of the participant's program;
A .ans file, the output of the main solution.

Thus, to test a checker, you should provide a subset of these three (3) files. The checker unit test definition expects a glob that matches the names of these three files. Thus, the glob unit/checker/ac* will match, for instance, unit/checker/ac.in, unit/checker/ac.out and unit/checker/ac.ans.

Not all three files must exist, only those required by the checker.