utopya.model_registry package

This submodule implements a registry of Utopia models, which is used to provide the required model information to the frontend and use it throughout.

Submodules

utopya.model_registry._exceptions module

All model registry related error types

exception utopya.model_registry._exceptions.ModelRegistryError

Bases: ValueError

Raised on errors with model registry

__module__ = 'utopya.model_registry._exceptions'

__weakref__

list of weak references to the object (if defined)

exception utopya.model_registry._exceptions.BundleExistsError

Bases: utopya.model_registry._exceptions.ModelRegistryError

Raised when a bundle that compared equal already exists.

__module__ = 'utopya.model_registry._exceptions'

utopya.model_registry.entry module

This module implements the ModelRegistryEntry class, which contains a set of ModelInfoBundle objects. These can be combined in a labelled and unlabelled fashion.

class utopya.model_registry.entry.ModelRegistryEntry(model_name: str, *, registry_dir: str)

Bases: object

Holds model config bundles for a single model.

Instances of this class are associated with a file in the model registry directory. Upon change, they update that file.

Initialize a model registry entry from a registry directory.

Expects a file named <model_name>.yml in the registry directory. If it does not exist, creates an empty one.

__init__(model_name: str, *, registry_dir: str)

Initialize a model registry entry from a registry directory.

Expects a file named <model_name>.yml in the registry directory. If it does not exist, creates an empty one.

property model_name: str

property registry_dir: str

property registry_file_path: str

__len__() → int

Returns total number of bundles, labelled and unlabelled.

__contains__(obj) → bool

Checks if the given _object_ is part of this registry entry.

Note that this does not check for keys!

__str__() → str

Return str(self).

__eq__(other) → bool

Check for equality by inspecting stored bundles and model name

__getitem__(key: Union[str, int]) → utopya.model_registry.info_bundle.ModelInfoBundle

Return a bundle for the given label or index. If None, returns the tries to return the item.

item() → utopya.model_registry.info_bundle.ModelInfoBundle

Retrieve the single bundle for this model, if not ambiguous.

keys() → Iterator[Union[str, int]]

Returns legitimate keys for item access, i.e.: all registered keys. Starts with labelled bundle names, continues with indices of unlabelled bundles.

values() → Generator[utopya.model_registry.info_bundle.ModelInfoBundle, None, None]

Returns stored model config bundles, starting with labelled ones

items() → Generator[Tuple[Union[str, int], utopya.model_registry.info_bundle.ModelInfoBundle], None, None]

Returns keys and registered bundles, starting with labelled ones

add_bundle(*, label: Optional[str] = None, overwrite_label: bool = False, update_registry_file: bool = True, **bundle_kwargs) → utopya.model_registry.info_bundle.ModelInfoBundle

Add a new configuration bundle to this registry entry.

This makes sure that the added bundle does not compare equal to an already existing bundle.

Parameters

• label (str, optional) – The label under which to add it. If None, will be added as unlabelled.

• overwrite_label (bool, optional) – If True, overwrites an existing bundle under the same label. This option does not affect unlabelled bundles or registry file updates.

• update_registry_file (bool, optional) – Whether to write changes directly to the registry file.

• **bundle_kwargs – Passed on to construct the ModelInfoBundle that is to be stored.

Raises

ModelRegistryError – If exist_ok is False and such a bundle already exists in this registry entry.

pop(key: Union[str, int], *, update_registry_file: bool = True) → utopya.model_registry.info_bundle.ModelInfoBundle

Pop a configuration bundle from this entry.

clear(*, update_registry_file: bool = True)

Removes all configuration bundles from this entry.

classmethod to_yaml(representer, node)

Creates a YAML representation of the data stored in this entry. As the data is a combination of a dict and a sequence, instances of this class are also represented as such.

Parameters

• representer (ruamel.yaml.representer) – The representer module

• node (type(self)) – The node to represent, i.e. an instance of this class.

Returns

A YAML representation of the given instance of this class.

__dict__ = mappingproxy({'__module__': 'utopya.model_registry.entry', '__doc__': 'Holds model config bundles for a single model.\n\n    Instances of this class are associated with a file in the model registry\n    directory. Upon change, they update that file.\n    ', '__init__': <function ModelRegistryEntry.__init__>, 'model_name': <property object>, 'registry_dir': <property object>, 'registry_file_path': <property object>, '__len__': <function ModelRegistryEntry.__len__>, '__contains__': <function ModelRegistryEntry.__contains__>, '__str__': <function ModelRegistryEntry.__str__>, '__eq__': <function ModelRegistryEntry.__eq__>, '__getitem__': <function ModelRegistryEntry.__getitem__>, 'item': <function ModelRegistryEntry.item>, 'keys': <function ModelRegistryEntry.keys>, 'values': <function ModelRegistryEntry.values>, 'items': <function ModelRegistryEntry.items>, 'add_bundle': <function ModelRegistryEntry.add_bundle>, 'pop': <function ModelRegistryEntry.pop>, 'clear': <function ModelRegistryEntry.clear>, '_load_from_registry': <function ModelRegistryEntry._load_from_registry>, '_update_registry_file': <function ModelRegistryEntry._update_registry_file>, '_as_dict': <classmethod object>, 'to_yaml': <classmethod object>, '__dict__': <attribute '__dict__' of 'ModelRegistryEntry' objects>, '__weakref__': <attribute '__weakref__' of 'ModelRegistryEntry' objects>, '__hash__': None, '__annotations__': {}})

__hash__ = None

__module__ = 'utopya.model_registry.entry'

__weakref__

list of weak references to the object (if defined)

utopya.model_registry.info_bundle module

Implements the ModelInfoBundle class, which holds a single bundle of information about a model.

class utopya.model_registry.info_bundle.ModelInfoBundle(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', **additional_kwargs)

Bases: object

A bundle of model information; behaves like a read-only dict

Initialize a ModelInfoBundle

PATH_KEYS = (('binary', <class 'str'>, True), ('source_dir', <class 'str'>), ('default_cfg', <class 'str'>, True), ('default_plots', <class 'str'>), ('base_plots', <class 'str'>), ('python_model_tests_dir', <class 'str'>), ('python_model_plots_dir', <class 'str'>))

PATH_KEYS_REL_TO_SRC = ('default_cfg', 'default_plots', 'base_plots')

SRC_DIR_SEARCH_PATHS = (('default_cfg', '{}_cfg.yml'), ('default_plots', '{}_plots.yml'), ('base_plots', '{}_base_plots.yml'))

METADATA_KEYS = (('version', <class 'str'>), ('long_name', <class 'str'>), ('description', <class 'str'>), ('author', <class 'str'>), ('email', <class 'str'>), ('website', <class 'str'>), ('dependencies', <class 'list'>), ('utopya_compatibility', <class 'str'>), ('misc', <class 'dict'>))

__init__(*, model_name: str, paths: dict, metadata: Optional[dict] = None, project_name: Optional[str] = None, registration_time: Optional[str] = None, missing_path_action: str = 'log', **additional_kwargs)

Initialize a ModelInfoBundle

__eq__(other) → bool

Compares equality by comparing the stored configuration. Only if another ModelInfoBundle is compared against does the model name also take part in the comparison.

__str__() → str

Return str(self).

property model_name: str

property registration_time: str

Registration time string of this bundle

property as_dict: dict

Returns a deep copy of all bundle data. This does NOT include the model name and the registration time.

__getitem__(key: str)

Direct access to the full bundle data

property paths: dict

Access to the paths information of the bundle

property metadata: dict

Access to the metadata information of the bundle

property project_name: str

Access to the Utopia project name information of the bundle

property missing_paths: dict

Returns those paths where os.path.exists did not evaluate to True

classmethod to_yaml(representer, node)

Creates a YAML representation of the data stored in this bundle.

Parameters

• representer (ruamel.yaml.representer) – The representer module

• node (type(self)) – The node to represent, i.e. an instance of this class.

Returns

A YAML representation of the given instance of this class.

__dict__ = mappingproxy({'__module__': 'utopya.model_registry.info_bundle', '__doc__': 'A bundle of model information; behaves like a read-only dict', 'PATH_KEYS': (('binary', <class 'str'>, True), ('source_dir', <class 'str'>), ('default_cfg', <class 'str'>, True), ('default_plots', <class 'str'>), ('base_plots', <class 'str'>), ('python_model_tests_dir', <class 'str'>), ('python_model_plots_dir', <class 'str'>)), 'PATH_KEYS_REL_TO_SRC': ('default_cfg', 'default_plots', 'base_plots'), 'SRC_DIR_SEARCH_PATHS': (('default_cfg', '{}_cfg.yml'), ('default_plots', '{}_plots.yml'), ('base_plots', '{}_base_plots.yml')), 'METADATA_KEYS': (('version', <class 'str'>), ('long_name', <class 'str'>), ('description', <class 'str'>), ('author', <class 'str'>), ('email', <class 'str'>), ('website', <class 'str'>), ('dependencies', <class 'list'>), ('utopya_compatibility', <class 'str'>), ('misc', <class 'dict'>)), '__init__': <function ModelInfoBundle.__init__>, '__eq__': <function ModelInfoBundle.__eq__>, '__str__': <function ModelInfoBundle.__str__>, 'model_name': <property object>, 'registration_time': <property object>, 'as_dict': <property object>, '__getitem__': <function ModelInfoBundle.__getitem__>, 'paths': <property object>, 'metadata': <property object>, 'project_name': <property object>, 'missing_paths': <property object>, '_parse_paths': <function ModelInfoBundle._parse_paths>, 'to_yaml': <classmethod object>, '__dict__': <attribute '__dict__' of 'ModelInfoBundle' objects>, '__weakref__': <attribute '__weakref__' of 'ModelInfoBundle' objects>, '__hash__': None, '__annotations__': {}})

__hash__ = None

__module__ = 'utopya.model_registry.info_bundle'

__weakref__

list of weak references to the object (if defined)

utopya.model_registry.registry module

This module implements the ModelRegistry, which combines ModelRegistryEntry objects and makes it possible to register new models.

class utopya.model_registry.registry.KeyOrderedDict(*args, key: Optional[Callable] = None, **kwds)

Bases: dantro.utils.ordereddict.KeyOrderedDict

A key-ordered dict that expects string keys and sorts by the lower-case representation of keys.

Initialize a KeyOrderedDict, which maintains key ordering. If no custom ordering function is given, orders by simple “smaller than” comparison.

Apart from that, the interface is the same as for regular dictionaries.

Parameters

• *args – a single sequence of (key, value) pairs to insert

• key (Callable, optional) – The callable used to

• **kwds – Passed on to update method

Raises

TypeError – on len(args) > 1

DEFAULT_KEY_COMPARATOR(k)

__module__ = 'utopya.model_registry.registry'

class utopya.model_registry.registry.ModelRegistry(utopia_cfg_dir_path: str = '/root/.config/utopia')

Bases: object

The ModelRegistry class takes care of providing model information to the rest of the utopya package and, at the same time, maintains the model registry it is associated with.

It provides a dict-like interface to access the stored registry entries under their model name. Via reigster_model_info, a model can be registered and information can be added to it.

Additionally, there are some functions that provide an overview over the registered models and with which information they

# TODO Consider making this class a singleton?

Loads the utopia model registry from the configuration at the given path.

Parameters

utopia_cfg_dir_path (str, optional) – The path to store the model registry folder in.

__init__(utopia_cfg_dir_path: str = '/root/.config/utopia')

Loads the utopia model registry from the configuration at the given path.

Parameters

utopia_cfg_dir_path (str, optional) – The path to store the model registry folder in.

property registry_dir: str

The model registry directory path

__len__() → int

__str__() → str

Return str(self).

property info_str: str

property info_str_detailed: str

keys()

values()

items()

__contains__(model_name: str) → bool

Whether an entry for the given model name exists in the registry

__getitem__(model_name: str) → utopya.model_registry.entry.ModelRegistryEntry

Retrieve a deep copy of a model registration entry for the given model name.

register_model_info(model_name: str, *, exists_action: Optional[str] = None, **bundle_kwargs) → utopya.model_registry.entry.ModelRegistryEntry

Register information for a single model. This method also allows to create a new entry if a model does not exist.

However, it will raise an error if the model was already registered and neither the skip nor the remove options were explicitly specified.

Parameters

• model_name (str) – The name of the model to register

• exists_action (str, optional) –

  The action to take when a model of the given name already exists. Possible values:

  • None: Continue with registering the info bundle; note

    that this might still lead to errors if the exact same bundle already exists.

  • skip: Skip bundle registration

  • raise: Raise an error

  • clear: Clear all existing bundles

    from the corresponding model registry entry.

  • validate: Makes sure the bundle that will be created

    from the given kwargs is part of the registry entry.

• **bundle_kwargs – Passed on to ModelRegistryEntry.add_bundle

Returns

The registry entry for this model.

Return type

ModelRegistryEntry

Raises

ValueError – On exists_action == 'raise' and model already existing.

remove_entry(model_name: str)

Removes a registry entry and deletes the associated registry file

__dict__ = mappingproxy({'__module__': 'utopya.model_registry.registry', '__doc__': 'The ModelRegistry class takes care of providing model information to the\n    rest of the utopya package and, at the same time, maintains the model\n    registry it is associated with.\n\n    It provides a dict-like interface to access the stored registry entries\n    under their model name. Via ``reigster_model_info``, a model can be\n    registered and information can be added to it.\n\n    Additionally, there are some functions that provide an overview over the\n    registered models and with which information they\n\n    # TODO Consider making this class a singleton?\n    ', '__init__': <function ModelRegistry.__init__>, 'registry_dir': <property object>, '__len__': <function ModelRegistry.__len__>, '__str__': <function ModelRegistry.__str__>, 'info_str': <property object>, 'info_str_detailed': <property object>, 'keys': <function ModelRegistry.keys>, 'values': <function ModelRegistry.values>, 'items': <function ModelRegistry.items>, '__contains__': <function ModelRegistry.__contains__>, '__getitem__': <function ModelRegistry.__getitem__>, 'register_model_info': <function ModelRegistry.register_model_info>, 'remove_entry': <function ModelRegistry.remove_entry>, '_add_entry': <function ModelRegistry._add_entry>, '_load_from_registry_dir': <function ModelRegistry._load_from_registry_dir>, '__dict__': <attribute '__dict__' of 'ModelRegistry' objects>, '__weakref__': <attribute '__weakref__' of 'ModelRegistry' objects>, '__annotations__': {}})

__module__ = 'utopya.model_registry.registry'

__weakref__

list of weak references to the object (if defined)

utopya.model_registry.utils module

Utility functions that work on the already initialized model registry.

utopya.model_registry.utils.get_info_bundle(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, bundle_key: Optional[Union[str, int]] = None) → utopya.model_registry.info_bundle.ModelInfoBundle

Determine the model info bundle in cases where both a model name and an info bundle are allowed as arguments.

Parameters

• model_name (str, optional) – The model name.

• info_bundle (ModelInfoBundle, optional) – The info bundle object. If given, will directly return this object again.

• bundle_key (Union[str, int], optional) – In cases where only the model name is given, the bundle_key can be used for item access, e.g. in cases where more than one bundle is available and access would be ambiguous.

Returns

The selected info bundle item

Return type

ModelInfoBundle

Raises

ValueError – If neither or both model_name and info_bundle were None

utopya.model_registry.utils.load_model_cfg(**get_info_bundle_kwargs) → Tuple[dict, str, dict]

Loads the default model configuration file for the given model name, using the path specified in the info bundle.

Furthermore, extract_validation_objects() is called to extract any Parameter objects that require validation, replace them with their default values, and gather the Parameter class objects into a separate dict.

Parameters

**get_info_bundle_kwargs – Passed on to get_info_bundle

Returns

The corresponding model configuration, the path

to the model configuration file, and the Parameter class objects requiring validation.

Return type

Tuple[dict, str, dict]

Raises

• FileNotFoundError – On missing file

• ValueError – On missing model