Validating model parameters
Contents
Validating model parameters#
When running your model, you always want to be sure only valid parameters have been passed, since for invalid parameters the model will either break or show unexpected behaviour. For instance, certain parameters may need to be integers or probabilities, or you may want string parameters to be one of only a few permissible options.
Utopia allows to specify validity constraints on your model’s parameters and have them automatically checked before a simulation run and without the need to include those checks into your model implementation.
You can define a parameter’s default value, its data type, range limits, or pass a set of accepted values. In case of any errors, a concise message is logged and the run terminated. Additionally, you can directly add a clear name and descriptive text to your parameter, making that information parseable and reducing the need for comment strings.
For certain frequently used parameter types (such as probabilities), a convenient shorthand notation is available.
Note
This feature currently only supports validating scalar nodes. You cannot validate sequences or the structure of mappings.
Basic syntax#
Validation parameters can be optionally specified on all scalar parameters in your default model configuration file, i.e. inside the <your_model_name>_cfg.yml
file.
The rationale behind specifying it in that file is to have all information on model parameters in one place.
If you wish to have a model parameter validated, simply add a !param
tag after the parameter name, and specify the default value and any validity constraints in an indented block underneath.
A default
value is required, though the validity restrictions are optional.
Refer to the documentation below and to the examples to learn more about possible keys.
- Parameter.__init__(*, default: Any, name: Optional[str] = None, description: Optional[str] = None, is_any_of: Optional[Sequence[Any]] = None, limits: Optional[Tuple[Union[None, float], Union[None, float]]] = None, limits_mode: str = '[]', dtype: Optional[Union[str, type]] = None)[source]
Creates a new Parameter object, which holds a default value as well as some constraints on the possible values this parameter can assume.
- Parameters
default (Any) – the default value of the parameter.
name (str, optional) – the name of the parameter.
description (str, optional) – a description of this parameter or its effects.
is_any_of (Sequence[Any], optional) – a sequence of possible values this parameter can assume. If this parameter is given,
limits
cannot be used.limits (Tuple[Union[None, float], Union[None, float]], optional) – the upper and lower bounds of the parameter (only applicable to scalar numerals). If None, the bound is assumed to be negative or positive infinity, respectively. Whether boundary values are included into the interval is controlled by the
limits_mode
argument. This argument is mutually exclusive withis_any_of
!limits_mode (str, optional) – whether to interpret the limits as an open, closed, or semi-closed interval. Possible values:
'[]'
(closed, default),'()'
(open),'[)'
, and'(]'
.dtype (Union[str, type], optional) – expected data type of this parameter. Accepts all strings that are accepted by numpy.dtype , eg.
int
,float
,uint16
,string
.
- Raises
TypeError – On a
limits
argument that was not tuple-like or if alimits
argument was given but thedefault
was aValueError – if an invalid
limits_mode
is passed, iflimits
andis_any_of
are both passed, or if thelimits
argument did not have length 2.
Warning
!param
can only be specified inside your default model configuration, e.g. MyFancyModel_cfg.yml
.
Specifying it elsewhere will lead to errors!
Examples#
# # The default configuation file for MyModel
# ---
some_int: !param
name: my integer
description: some integer parameter
default: 3
dtype: int
another_int: !param
name: my negative integer
description: an integer less than zero
default: -2
limits: [~, 0]
limits_mode: ()
some_unsigned_int: !param
default: 0
dtype: uint8 # only allows integer values in [0, 255]
a_float: !param
default: 4.20
dtype: float
a_string: !param
default: foo
is_any_of: [foo, bar, baz]
some:
nested:
parameter: !param
default: spam
is_any_of: [spam, fish]
dtype: str
Hint
Use YAML’s ~
(i.e., None
) for specifying ±inf in limits
.
Shorthands#
For frequently used parameter types, shorthand tags are available to spare you the trouble of having to specify limits or data types. Simply add the corresponding YAML tag and the default value directly:
# # The default configuation file for MyModel
# ---
# ...
# a probability value, i.e. a float in [0, 1]
some_probability: !is-probability 0.3
# an integer
some_int: !is-int 3
# a positive value (strictly greater than 0)
some_positive_value: !is-positive 5
# a boolean
some_bool: !is-bool true
Available shorthands are:
is-probability
(afloat
in range[0, 1]
)is-int
is-bool
is-string
is-positive
/is-negative
: strictly greater/less than zerois-unsigned
: greater or equal to zero and data typeuint
is-positive-int
/is-negative-int
: strictly greater/less than zero and data typeint
See also the Forest Fire model for an integrated demo of this feature.
Validation procedure#
Parameter validation happens as part of the Multiverse
initialization and before any simulations are started.
The validation is restricted to the parameter_space
entry of the meta configuration, i.e. all values that will be made available to the model executable.
If parameter_space
actually is a parameter space, validation will always occur for the full parameter space and including the default point, independent of whether a sweep will actually be performed.
For all specified validation parameters, the given value is checked against the constraints using the Parameter
‘s validate()
method:
- Parameter.validate(value: Any, *, raise_exc: bool = True) bool [source]
Checks whether the given value would be a valid parameter.
The checks for the corresponding arguments are carried out in the following order:
is_any_of
dtype
limits
The data type is checked according to the numpy type hierarchy, see docs. To reduce strictness, the following additional compatibilities are taken into account:
for unsigned integer
dtype
, a signed integer-typevalue
is compatible ifvalue >= 0
for floating-point
dtype
, integer-typevalue
are always considered compatiblefor floating-point
dtype
,value
of all floating-point- types are considered compatible, even if they have a lower precision (note the coercion test below, though)
Additionally, it is checked whether
value
is representable as the target data type. This is done by coercingvalue
todtype
and then checking for equality (using np.isclose).- Parameters
value (Any) – The value to test.
raise_exc (bool, optional) – Whether to raise an exception or not.
- Returns
Whether or not the given value is a valid parameter.
- Return type
bool
- Raises
ValidationError – If validation failed or is impossible (for instance due to ambiguous validity parameters). This error message contains further information on why validation failed.
Hint
For very large sweeps (parameter space volume > 10k), parameter validation may take a long time and can thus be optionally disabled.
To skip parameter validation, set the following entry on the top-level of the run configuration:
perform_validation: false
Alternatively, you can specify --skip-validation
via the command line interface, see utopia run --help
.