Validating model parameters#

When running your model, you always want to be sure only valid parameters have been passed, since for invalid parameters the model will either break or show unexpected behaviour. For instance, certain parameters may need to be integers or probabilities, or you may want string parameters to be one of only a few permissible options.

Utopia allows to specify validity constraints on your model’s parameters and have them automatically checked before a simulation run and without the need to include those checks into your model implementation.

You can define a parameter’s default value, its data type, range limits, or pass a set of accepted values. In case of any errors, a concise message is logged and the run terminated. Additionally, you can directly add a clear name and descriptive text to your parameter, making that information parseable and reducing the need for comment strings.

For certain frequently used parameter types (such as probabilities), a convenient shorthand notation is available.

Note

This feature currently only supports validating scalar nodes. You cannot validate sequences or the structure of mappings.

Basic syntax#

Validation parameters can be optionally specified on all scalar parameters in your default model configuration file, i.e. inside the <your_model_name>_cfg.yml file. The rationale behind specifying it in that file is to have all information on model parameters in one place.

If you wish to have a model parameter validated, simply add a !param tag after the parameter name, and specify the default value and any validity constraints in an indented block underneath. A default value is required, though the validity restrictions are optional.

Refer to the documentation below and to the examples to learn more about possible keys.

Parameter.__init__(*, default: Any, name: Optional[str] = None, description: Optional[str] = None, is_any_of: Optional[Sequence[Any]] = None, limits: Optional[Tuple[Union[None, float], Union[None, float]]] = None, limits_mode: str = '[]', dtype: Optional[Union[str, type]] = None)[source]

Creates a new Parameter object, which holds a default value as well as some constraints on the possible values this parameter can assume.

Parameters

  • default (Any) – the default value of the parameter.

  • name (str, optional) – the name of the parameter.

  • description (str, optional) – a description of this parameter or its effects.

  • is_any_of (Sequence[Any], optional) – a sequence of possible values this parameter can assume. If this parameter is given, limits cannot be used.

  • limits (Tuple[Union[None, float], Union[None, float]], optional) – the upper and lower bounds of the parameter (only applicable to scalar numerals). If None, the bound is assumed to be negative or positive infinity, respectively. Whether boundary values are included into the interval is controlled by the limits_mode argument. This argument is mutually exclusive with is_any_of!

  • limits_mode (str, optional) – whether to interpret the limits as an open, closed, or semi-closed interval. Possible values: '[]' (closed, default), '()' (open), '[)', and '(]'.

  • dtype (Union[str, type], optional) – expected data type of this parameter. Accepts all strings that are accepted by numpy.dtype , eg. int, float, uint16, string.

Raises

  • TypeError – On a limits argument that was not tuple-like or if a limits argument was given but the default was a

  • ValueError – if an invalid limits_mode is passed, if limits and is_any_of are both passed, or if the limits argument did not have length 2.

Warning

!param can only be specified inside your default model configuration, e.g. MyFancyModel_cfg.yml. Specifying it elsewhere will lead to errors!

Examples#

# # The default configuation file for MyModel
# ---
  some_int: !param
    name: my integer
    description: some integer parameter
    default: 3
    dtype: int

  another_int: !param
    name: my negative integer
    description: an integer less than zero
    default: -2
    limits: [~, 0]
    limits_mode: ()

  some_unsigned_int: !param
    default: 0
    dtype: uint8  # only allows integer values in [0, 255]

  a_float: !param
    default: 4.20
    dtype: float

  a_string: !param
    default: foo
    is_any_of: [foo, bar, baz]

  some:
    nested:
      parameter: !param
        default: spam
        is_any_of: [spam, fish]
        dtype: str

Hint

Use YAML’s ~ (i.e., None) for specifying ±inf in limits.

Shorthands#

For frequently used parameter types, shorthand tags are available to spare you the trouble of having to specify limits or data types. Simply add the corresponding YAML tag and the default value directly:

# # The default configuation file for MyModel
# ---
  # ...

  # a probability value, i.e. a float in [0, 1]
  some_probability: !is-probability 0.3

  # an integer
  some_int: !is-int 3

  # a positive value (strictly greater than 0)
  some_positive_value: !is-positive 5

  # a boolean
  some_bool: !is-bool true

Available shorthands are:

  • is-probability (a float in range [0, 1])

  • is-int

  • is-bool

  • is-string

  • is-positive/is-negative: strictly greater/less than zero

  • is-unsigned: greater or equal to zero and data type uint

  • is-positive-int/is-negative-int: strictly greater/less than zero and data type int

See also the Forest Fire model for an integrated demo of this feature.

Validation procedure#

Parameter validation happens as part of the Multiverse initialization and before any simulations are started.

The validation is restricted to the parameter_space entry of the meta configuration, i.e. all values that will be made available to the model executable. If parameter_space actually is a parameter space, validation will always occur for the full parameter space and including the default point, independent of whether a sweep will actually be performed.

For all specified validation parameters, the given value is checked against the constraints using the Parameter‘s validate() method:

Parameter.validate(value: Any, *, raise_exc: bool = True) → bool[source]

Checks whether the given value would be a valid parameter.

The checks for the corresponding arguments are carried out in the following order:

  1. is_any_of

  2. dtype

  3. limits

The data type is checked according to the numpy type hierarchy, see docs. To reduce strictness, the following additional compatibilities are taken into account:

  • for unsigned integer dtype, a signed integer-type value is compatible if value >= 0

  • for floating-point dtype, integer-type value are always considered compatible

  • for floating-point dtype, value of all floating-point- types are considered compatible, even if they have a lower precision (note the coercion test below, though)

Additionally, it is checked whether value is representable as the target data type. This is done by coercing value to dtype and then checking for equality (using np.isclose).

Parameters

  • value (Any) – The value to test.

  • raise_exc (bool, optional) – Whether to raise an exception or not.

Returns

Whether or not the given value is a valid parameter.

Return type

bool

Raises

ValidationError – If validation failed or is impossible (for instance due to ambiguous validity parameters). This error message contains further information on why validation failed.

Hint

For very large sweeps (parameter space volume > 10k), parameter validation may take a long time and can thus be optionally disabled.

To skip parameter validation, set the following entry on the top-level of the run configuration:

perform_validation: false

Alternatively, you can specify --skip-validation via the command line interface, see utopia run --help.