Working Interactively with Utopia¶

Via utopya, you also have full access to all of Utopia in an interactive fashion, e.g. in an IPython session or in a Jupyter Notebook. This is especially useful for exploring some model parameters and digging through the data generated by your model.

Setting up an Interactive Session
Basic Concepts
Examples

Setting up an Interactive Session ¶

Note

All of the below needs to happen inside the virtual environment that utopya was installed into.

With IPython ¶

After installing IPython into the virtual environment, you can enter an interactive session simply via:

python -m IPython

Unlike the ipython command, this ensures that it uses the python binary of the virtual environment, which has access to the utopya installation.

With Jupyter Notebook ¶

You can do the same with Jupyter Notebooks. For this, you just have to take care to select the correct kernel, i.e.: the one provided by the utopia-env and the corresponding packages.

To do so, use pip to install jupyter, if you haven’t already done so. Then, install ipykernel and manually add a kernel to jupyter:

ipython kernel install --user --name=projectname

When running jupyter notebook, you should now be able to select a kernel from the dropdown menu in the top right-hand corner.

With the Docker Container ¶

You can also work interactively with Utopia when using the Utopia docker container. As part of the ccees/utopia docker image, Jupyter is already pre-installed and a notebook server is started alongside the container. The idea is that you connect to that notebook server with the browser on your host system.

Follow the detailed instructions on the corresponding Docker Hub page to find out more.

Basic Concepts ¶

This section aims to convey the basic concepts of the interactive utopya interface.

The `Model` class ¶

Objects of this class are your main tool when working with Utopia interactively. When you want to run a certain model, say ForestFire, the first thing to do is to create an instance of the Model class.

import utopya

ffm = utopya.Model(name='ForestFire')

This object now provides an interface to perform one or many simulation runs, load the data, and continue working with it. It basically manages the model invocation and loading, and has the same capabilities the CLI has; in fact, the CLI is just a wrapper around the Model class.

For example, to run a model, you can simply call:

mv, dm = ffm.create_run_load()

This will create a Multiverse (see below), run a simulation (here: with the default parameters), and then load the data. It returns the Multiverse object and the corresponding DataManager, for convenience.

Hint

You can perform multiple runs with one single Model instance, so it suffices to create one such instance for a model in the beginning of the interactive session.

Importantly, the Model is associated with one specific ModelInfoBundle, which contains all the information that is required to run the selected model. By using the Model class and its interface, you don’t need to worry about passing this info bundle around.

For more information on the available interface, have a look at excerpts from its API:

class utopya.Model(*, name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, base_dir: Optional[str] = None, sim_errors: Optional[str] = None, use_tmpdir: bool = False)[source]

Bases: object

A class to work with Utopia models interactively.

It attaches to a certain model and makes it easy to load config files, create a Multiverse from them, run it, and work with it further…

Initialize the ModelTest for the given model name

Parameters

name (str, optional) – Name of the model to attach to. If not given, need to pass info_bundle.
info_bundle (ModelInfoBundle, optional) – The required information to work with this model. If not given, will attempt to find the model in the model registry via name.
base_dir (str, optional) – For convenience, can specify this path which will be seen as the base path for config files; if set, arguments that allow specifying configuration files can specify them relative to this directory.
sim_errors (str, optional) – Whether to raise errors from Multiverse
use_tmpdir (bool, optional) – Whether to use a temporary directory to write data to. The default value can be set here; but the flag can be overwritten in the create_mv and create_run_load methods. For false, the regular model output directory is used.

Raises

ValueError – Upon bad base_dir

__init__(*, name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, base_dir: Optional[str] = None, sim_errors: Optional[str] = None, use_tmpdir: bool = False)[source]¶

Initialize the ModelTest for the given model name

Parameters

name (str, optional) – Name of the model to attach to. If not given, need to pass info_bundle.
info_bundle (ModelInfoBundle, optional) – The required information to work with this model. If not given, will attempt to find the model in the model registry via name.
base_dir (str, optional) – For convenience, can specify this path which will be seen as the base path for config files; if set, arguments that allow specifying configuration files can specify them relative to this directory.
sim_errors (str, optional) – Whether to raise errors from Multiverse
use_tmpdir (bool, optional) – Whether to use a temporary directory to write data to. The default value can be set here; but the flag can be overwritten in the create_mv and create_run_load methods. For false, the regular model output directory is used.

Raises

ValueError – Upon bad base_dir

__init__(*, name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, base_dir: Optional[str] = None, sim_errors: Optional[str] = None, use_tmpdir: bool = False)[source]

Initialize the ModelTest for the given model name

Parameters

name (str, optional) – Name of the model to attach to. If not given, need to pass info_bundle.
info_bundle (ModelInfoBundle, optional) – The required information to work with this model. If not given, will attempt to find the model in the model registry via name.
base_dir (str, optional) – For convenience, can specify this path which will be seen as the base path for config files; if set, arguments that allow specifying configuration files can specify them relative to this directory.
sim_errors (str, optional) – Whether to raise errors from Multiverse
use_tmpdir (bool, optional) – Whether to use a temporary directory to write data to. The default value can be set here; but the flag can be overwritten in the create_mv and create_run_load methods. For false, the regular model output directory is used.

Raises

ValueError – Upon bad base_dir

__str__() → str[source]: Returns an informative string for this Model instance

property info_bundle: utopya.model_registry.info_bundle.ModelInfoBundle: The model info bundle

property name: str: The name of this Model object, which is at the same time the name of the attached model.

property base_dir: str

Returns the path to the base directory, if set during init.

This is the path to a directory from which config files can be loaded using relative paths.

property default_model_cfg: dict: Returns the default model configuration by loading it from the file specified in the info bundle.

property default_config_set_search_dirs: List[str]: Returns the default config set search directories for this model in the order of precedence.

Note

These may be relative paths.

property default_config_sets: Dict[str, dict]

Config sets at the default search locations.

To retrieve an individual config set, consider using get_config_set() instead of this property.

For more information, see Configuration sets.

create_mv(*, from_cfg: Optional[str] = None, from_cfg_set: Optional[str] = None, run_cfg_path: Optional[str] = None, use_tmpdir: Optional[bool] = None, **update_meta_cfg) → utopya.multiverse.Multiverse[source]

Creates a utopya.multiverse.Multiverse for this model, optionally loading a configuration from a file and updating it with further keys.

Parameters

from_cfg (str, optional) – The name of the config file (relative to the base directory) to be used.
from_cfg_set (str, optional) – Name of the config set to retrieve the run config from. Mutually exclusive with from_cfg and run_cfg_path.
run_cfg_path (str, optional) – The path of the run config to use. Can not be passed if from_cfg or from_cfg_set arguments were given.
use_tmpdir (bool, optional) – Whether to use a temporary directory to write the data to. If not given, uses default value set at initialization.
**update_meta_cfg – Can be used to update the meta configuration

Returns

The created Multiverse object

Return type

Multiverse

Raises

ValueError – If more than one of the run config selecting arguments (from_cfg, from_cfg_set, run_cfg_path) were given.

create_frozen_mv(**fmv_kwargs) → utopya.multiverse.FrozenMultiverse[source]

Create a utopya.multiverse.FrozenMultiverse, coupling it to a run directory.

Use this method if you want to load an existing simulation run.

Parameters: **fmv_kwargs – Passed on to FrozenMultiverse.__init__

create_run_load(*, from_cfg: Optional[str] = None, run_cfg_path: Optional[str] = None, from_cfg_set: Optional[str] = None, use_tmpdir: Optional[bool] = None, print_tree: bool = True, **update_meta_cfg) → Tuple[utopya.multiverse.Multiverse, utopya.datamanager.DataManager][source]

Chains the create_mv, mv.run, and mv.dm.load_from_cfg methods together and returns a (Multiverse, DataManager) tuple.

Parameters

from_cfg (str, optional) – The name of the config file (relative to the base directory) to be used.
from_cfg_set (str, optional) – Name of the config set to retrieve the run config from. Mutually exclusive with from_cfg and run_cfg_path.
run_cfg_path (str, optional) – The path of the run config to use. Can not be passed if from_cfg or from_cfg_set arguments were given.
use_tmpdir (bool, optional) – Whether to use a temporary directory to write the data to. If not given, uses default value set at initialization.
print_tree (bool, optional) – Whether to print the loaded data tree
**update_meta_cfg – Arguments passed to the create_mv function

Return type

Tuple[Multiverse, DataManager]

get_config_set(name: Optional[str] = None) → Dict[str, str][source]

Returns a configuration set: a dict containing paths to run and/or eval configuration files. These are accessible via the keys run and eval.

Config sets are retrieved from multiple locations:

The cfgs directory in the model’s source directory

The user-specified lookup directories, specified in the utopya configuration as config_set_search_dirs

If name is an absolute or relative path, and a directory exists at the specified location, the parent directory is interpreted as a search path.

This uses get_config_sets() to retrieve all available configuration sets from the above paths and then selects the one with the given name. Config sets that are found later overwrite those with the same name found in previous searches and log a warning message (which can be controlled with the warn argument); sets are not merged.

For more information, see Configuration sets.

Parameters: name (str, optional) – The name of the config set to retrieve. This may also be a local path, which is looked up prior to the default search directories.

get_config_sets(*, search_dirs: Optional[List[str]] = None, warn: bool = True, cfg_sets: Optional[dict] = None) → Dict[str, dict][source]

Searches for all available configuration sets in the given search directories, aggregating them into one dict.

The search is done in reverse order of the paths given in search_dirs, i.e. starting from those directories with the lowest precedence. If configuration sets with the same name are encountered, warnings are emitted, but the one with higher precedence (appearing more towards the front of search_dirs, i.e. the later-searched one) will take precedence.

Note

This will not merge configuration sets from different search directories, e.g. if one contained only an eval configuration and the other contained only a run configuration, a warning will be emitted but the entry from the later-searched directory will be used.

Parameters

search_dirs (List[str], optional) – The directories to search sequentially for config sets. If not given, will use the default config set search directories, see default_config_set_search_dirs.
warn (bool, optional) – Whether to warn (via log message), if the search yields a config set with a name that already existed.
cfg_sets (dict, optional) – If given, aggregate newly found config sets into this dict. Otherwise, start with an empty one.

__annotations__ = {}

__dict__ = mappingproxy({'__module__': 'utopya.model', '__doc__': 'A class to work with Utopia models interactively.\n\n It attaches to a certain model and makes it easy to load config files,\n create a Multiverse from them, run it, and work with it further...\n ', '__init__': <function Model.__init__>, '__str__': <function Model.__str__>, 'info_bundle': <property object>, 'name': <property object>, 'base_dir': <property object>, 'default_model_cfg': <property object>, 'default_config_set_search_dirs': <property object>, 'default_config_sets': <property object>, 'create_mv': <function Model.create_mv>, 'create_frozen_mv': <function Model.create_frozen_mv>, 'create_run_load': <function Model.create_run_load>, 'get_config_set': <function Model.get_config_set>, 'get_config_sets': <function Model.get_config_sets>, '_store_mv': <function Model._store_mv>, '_create_tmpdir': <function Model._create_tmpdir>, '_find_config_sets': <function Model._find_config_sets>, '__dict__': <attribute '__dict__' of 'Model' objects>, '__weakref__': <attribute '__weakref__' of 'Model' objects>, '__annotations__': {}})

__module__ = 'utopya.model'

__weakref__: list of weak references to the object (if defined)

The `Multiverse` class ¶

This class actually carries out the simulation run, i.e. running multiple so-called “universes” (independent simulations with different parameters). Upon completion, it provides the interface to an associated DataManager and PlotManager, which let you load the data and create plots, respectively. If you want to create a Multiverse object from your existing Model object, use the Model.create_mv method (see above for documentation).

Note

Each Multiverse can only run once.

class utopya.Multiverse(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, run_cfg_path: Optional[str] = None, user_cfg_path: Optional[str] = None, _shared_worker_manager: Optional[utopya.workermanager.WorkerManager] = None, **update_meta_cfg)[source]

Bases: object

The Multiverse is where a single simulation run is orchestrated from.

It spawns multiple universes, each of which represents a single simulation of the selected model with the parameters specified by the meta configuration.

The WorkerManager takes care to perform these simulations in parallel, the DataManager allows loading the created data, and the PlotManager handles plotting of that data.

Initialize the Multiverse.

Parameters

model_name (str, optional) – The name of the model to run
info_bundle (ModelInfoBundle, optional) – The model information bundle that includes information about the binary path etc. If not given, will attempt to read it from the model registry.
run_cfg_path (str, optional) – The path to the run configuration.
user_cfg_path (str, optional) – If given, this is used to update the base configuration. If None, will look for it in the default path, see Multiverse.USER_CFG_SEARCH_PATH.
_shared_worker_manager (WorkerManager, optional) –
If given, this already existing WorkerManager instance (and its reporter) will be used instead of initializing new instances.

Warning

This argument is only exposed for internal purposes. It should not be used for production code and behavior of this argument may change at any time.
**update_meta_cfg – Can be used to update the meta configuration generated from the previous configuration levels

BASE_META_CFG_PATH = '/builds/utopia-project/docs/utopia/python/utopya/utopya/cfg/base_cfg.yml'

USER_CFG_SEARCH_PATH = '/root/.config/utopia/user_cfg.yml'

RUN_DIR_TIME_FSTR = '%y%m%d-%H%M%S'

UTOPYA_BASE_PLOTS_PATH = '/builds/utopia-project/docs/utopia/python/utopya/utopya/plot_funcs/base_plots.yml'

__init__(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, run_cfg_path: Optional[str] = None, user_cfg_path: Optional[str] = None, _shared_worker_manager: Optional[utopya.workermanager.WorkerManager] = None, **update_meta_cfg)[source]

Initialize the Multiverse.

Parameters

model_name (str, optional) – The name of the model to run
info_bundle (ModelInfoBundle, optional) – The model information bundle that includes information about the binary path etc. If not given, will attempt to read it from the model registry.
run_cfg_path (str, optional) – The path to the run configuration.
user_cfg_path (str, optional) – If given, this is used to update the base configuration. If None, will look for it in the default path, see Multiverse.USER_CFG_SEARCH_PATH.
_shared_worker_manager (WorkerManager, optional) –
If given, this already existing WorkerManager instance (and its reporter) will be used instead of initializing new instances.

Warning

This argument is only exposed for internal purposes. It should not be used for production code and behavior of this argument may change at any time.
**update_meta_cfg – Can be used to update the meta configuration generated from the previous configuration levels

property info_bundle: utopya.model_registry.info_bundle.ModelInfoBundle: The model info bundle for this Multiverse

property model_name: str: The model name associated with this Multiverse

property model_binpath: str: The path to this model’s binary

property meta_cfg: dict: The meta configuration.

property dirs: dict: Information on managed directories.

property cluster_mode: bool: Whether the Multiverse should run in cluster mode

property cluster_params: dict: Returns a copy of the cluster mode configuration parameters

property resolved_cluster_params: dict: Returns a copy of the cluster configuration with all parameters resolved. This makes some additional keys available on the top level.

property dm: utopya.datamanager.DataManager: The Multiverse’s DataManager.

property wm: utopya.workermanager.WorkerManager: The Multiverse’s WorkerManager.

property pm: utopya.plotting.PlotManager: The Multiverse’s PlotManager.

run(*, sweep: Optional[bool] = None)[source]

Starts a Utopia simulation run.

Specifically, this method adds simulation tasks to the associated WorkerManager, locks its task list, and then invokes the start_working() method which performs all the simulation tasks.

If cluster mode is enabled, this will split up the parameter space into (ideally) equally sized parts and only run one of these parts, depending on the cluster node this Multiverse is being invoked on.

Note

As this method locks the task list of the WorkerManager, no further tasks can be added henceforth. This means, that each Multiverse instance can only perform a single simulation run.

Parameters: sweep (bool, optional) – Whether to perform a sweep or not. If None, the value will be read from the perform_sweep key of the meta-configuration.

run_single()[source]

Runs a single simulation using the parameter space’s default value.

See run() for more information.

run_sweep()[source]

Runs a parameter sweep.

See run() for more information.

renew_plot_manager(**update_kwargs)[source]

Tries to set up a new PlotManager. If this succeeds, the old one is discarded and the new one is associated with this Multiverse.

Parameters: **update_kwargs – Passed on to PlotManager.__init__

__annotations__ = {}

__dict__ = mappingproxy({'__module__': 'utopya.multiverse', '__doc__': 'The Multiverse is where a single simulation run is orchestrated from.\n\n It spawns multiple universes, each of which represents a single simulation\n of the selected model with the parameters specified by the meta\n configuration.\n\n The WorkerManager takes care to perform these simulations in parallel, the\n DataManager allows loading the created data, and the PlotManager handles\n plotting of that data.\n ', 'BASE_META_CFG_PATH': '/builds/utopia-project/docs/utopia/python/utopya/utopya/cfg/base_cfg.yml', 'USER_CFG_SEARCH_PATH': '/root/.config/utopia/user_cfg.yml', 'RUN_DIR_TIME_FSTR': '%y%m%d-%H%M%S', 'UTOPYA_BASE_PLOTS_PATH': '/builds/utopia-project/docs/utopia/python/utopya/utopya/plot_funcs/base_plots.yml', '__init__': <function Multiverse.__init__>, 'info_bundle': <property object>, 'model_name': <property object>, 'model_binpath': <property object>, 'meta_cfg': <property object>, 'dirs': <property object>, 'cluster_mode': <property object>, 'cluster_params': <property object>, 'resolved_cluster_params': <property object>, 'dm': <property object>, 'wm': <property object>, 'pm': <property object>, 'run': <function Multiverse.run>, 'run_single': <function Multiverse.run_single>, 'run_sweep': <function Multiverse.run_sweep>, 'renew_plot_manager': <function Multiverse.renew_plot_manager>, '_create_meta_cfg': <function Multiverse._create_meta_cfg>, '_create_run_dir': <function Multiverse._create_run_dir>, '_setup_pm': <function Multiverse._setup_pm>, '_perform_backup': <function Multiverse._perform_backup>, '_perform_pspace_backup': <function Multiverse._perform_pspace_backup>, '_prepare_executable': <function Multiverse._prepare_executable>, '_resolve_cluster_params': <function Multiverse._resolve_cluster_params>, '_add_sim_task': <function Multiverse._add_sim_task>, '_add_sim_tasks': <function Multiverse._add_sim_tasks>, '_validate_meta_cfg': <function Multiverse._validate_meta_cfg>, '__dict__': <attribute '__dict__' of 'Multiverse' objects>, '__weakref__': <attribute '__weakref__' of 'Multiverse' objects>, '__annotations__': {}})

__module__ = 'utopya.multiverse'

__weakref__: list of weak references to the object (if defined)

The `FrozenMultiverse` class ¶

If you have already performed a run and only want to load and work with its data, you don’t need a whole Multiverse with all its simulation capabilities. Instead, you want a frozen state of it, after the simulation.

The FrozenMultiverse is just what you need: it couples to some output directory and then lets you continue working with the DataManager and PlotManager, just as you would directly after a simulation run. After initialization, the API is the same as for the Multiverse.

To create such an instance from the Model class, use the Model.create_frozen_mv method (see above for documentation).

class utopya.multiverse.FrozenMultiverse(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, run_dir: Optional[str] = None, run_cfg_path: Optional[str] = None, user_cfg_path: Optional[str] = None, use_meta_cfg_from_run_dir: bool = False, **update_meta_cfg)[source]

Bases: utopya.multiverse.Multiverse

A frozen Multiverse is like a Multiverse, but frozen.

It is initialized from a finished Multiverse run and re-creates all the attributes from that data, e.g.: the meta configuration, a DataManager, and a PlotManager.

Note that it is no longer able to perform any simulations.

Initializes the FrozenMultiverse from a model name and the name of a run directory.

Note that this also takes arguments to specify the run configuration to use.

Parameters

model_name (str) – The name of the model to load. From this, the model output directory is determined and the run_dir will be seen as relative to that directory.
info_bundle (ModelInfoBundle, optional) – The model information bundle that includes information about the binary path etc. If not given, will attempt to read it from the model registry.
run_dir (str, optional) – The run directory to load. Can be a path relative to the current working directory, an absolute path, or the timestamp of the run directory. If not given, will use the most recent timestamp.
run_cfg_path (str, optional) – The path to the run configuration.
user_cfg_path (str, optional) – If given, this is used to update the base configuration. If None, will look for it in the default path, see Multiverse.USER_CFG_SEARCH_PATH.
use_meta_cfg_from_run_dir (bool, optional) – If True, will load the meta configuration from the given run directory; only works for absolute run directories.
**update_meta_cfg – Can be used to update the meta configuration generated from the previous configuration levels

__init__(*, model_name: Optional[str] = None, info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, run_dir: Optional[str] = None, run_cfg_path: Optional[str] = None, user_cfg_path: Optional[str] = None, use_meta_cfg_from_run_dir: bool = False, **update_meta_cfg)[source]

Initializes the FrozenMultiverse from a model name and the name of a run directory.

Note that this also takes arguments to specify the run configuration to use.

Parameters

model_name (str) – The name of the model to load. From this, the model output directory is determined and the run_dir will be seen as relative to that directory.
info_bundle (ModelInfoBundle, optional) – The model information bundle that includes information about the binary path etc. If not given, will attempt to read it from the model registry.
run_dir (str, optional) – The run directory to load. Can be a path relative to the current working directory, an absolute path, or the timestamp of the run directory. If not given, will use the most recent timestamp.
run_cfg_path (str, optional) – The path to the run configuration.
user_cfg_path (str, optional) – If given, this is used to update the base configuration. If None, will look for it in the default path, see Multiverse.USER_CFG_SEARCH_PATH.
use_meta_cfg_from_run_dir (bool, optional) – If True, will load the meta configuration from the given run directory; only works for absolute run directories.
**update_meta_cfg – Can be used to update the meta configuration generated from the previous configuration levels

__annotations__ = {}

__module__ = 'utopya.multiverse'

The `DataManager`¶

This is the home of all your simulation data for a simulation run. It provides a dict-like interface to navigate through the data tree, consisting of data groups (branching points of the tree) and data containers (leaves of the tree).

When created by a multiverse instance, it already has the required default configuration available.

class utopya.DataManager(data_dir: str, *, name: Optional[str] = None, load_cfg: Optional[Union[dict, str]] = None, out_dir: Union[str, bool] = '_output/{timestamp:}', out_dir_kwargs: Optional[dict] = None, create_groups: Optional[List[Union[str, dict]]] = None, condensed_tree_params: Optional[dict] = None, default_tree_cache_path: Optional[str] = None)[source]

Bases: dantro.data_loaders.AllAvailableLoadersMixin, dantro.data_mngr.DataManager

This class manages the data that is written out by Utopia simulations.

It is based on the dantro.DataManager class and adds the functionality for specific loader functions that are needed in Utopia: Hdf5 and Yaml.

Furthermore, to enable file caching via the DAG framework, all available data loaders are included here.

Initializes a DataManager for the specified data directory.

Parameters

data_dir (str) – the directory the data can be found in. If this is a relative path, it is considered relative to the current working directory.
name (str, optional) – which name to give to the DataManager. If no name is given, the data directories basename will be used
load_cfg (Union[dict, str], optional) – The base configuration used for loading data. If a string is given, assumes it to be the path to a YAML file and loads it using the load_yml() function. If None is given, it can still be supplied to the load() method later on.
out_dir (Union[str, bool], optional) – where output is written to. If this is given as a relative path, it is considered relative to the data_dir. A formatting operation with the keys timestamp and name is performed on this, where the latter is the name of the data manager. If set to False, no output directory is created.
out_dir_kwargs (dict, optional) – Additional arguments that affect how the output directory is created.
create_groups (List[Union[str, dict]], optional) – If given, these groups will be created after initialization. If the list entries are strings, the default group class will be used; if they are dicts, the name key specifies the name of the group and the Cls key specifies the type. If a string is given instead of a type, the lookup happens from the _DATA_GROUP_CLASSES variable.
condensed_tree_params (dict, optional) – If given, will set the parameters used for the condensed tree representation. Available options: max_level and condense_thresh, where the latter may be a callable. See dantro.base.BaseDataGroup._tree_repr() for more information.
default_tree_cache_path (str, optional) – The path to the default tree cache file. If not given, uses the value from the class variable _DEFAULT_TREE_CACHE_PATH. Whichever value was chosen is then prepared using the _parse_file_path() method, which regards relative paths as being relative to the associated data directory.

__abstractmethods__ = frozenset({})

__annotations__ = {}

__module__ = 'utopya.datamanager'

The `PlotManager`¶

The PlotManager – well, manages the plotting.

class utopya.plotting.PlotManager(*args, _model_info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, **kwargs)[source]

Bases: dantro.plot_mngr.PlotManager

This is the Utopia-specific version of the dantro PlotManager class

It registers the Utopia-specific plot creators and allows for custom interface specifications.

Sets up a PlotManager.

This additionally stores some Utopia-specific metadata about the model this PlotManager is used with. That information is then used to load some additional model-specific information once a creator is invoked.

CREATORS = {'multiverse': <class 'utopya.plotting.MultiversePlotCreator'>, 'universe': <class 'utopya.plotting.UniversePlotCreator'>}

__init__(*args, _model_info_bundle: Optional[utopya.model_registry.info_bundle.ModelInfoBundle] = None, **kwargs)[source]

Sets up a PlotManager.

This additionally stores some Utopia-specific metadata about the model this PlotManager is used with. That information is then used to load some additional model-specific information once a creator is invoked.

property common_out_dir: str

The common output directory of all plots that were created with this plot manager instance. This uses the plot output paths stored in the plot information dict, specifically the target_dir entry.

If there was no plot information yet, the return value will be empty.

plot_from_cfg(*args, plots_cfg: Optional[Union[dict, str]] = None, **kwargs)[source]: Thin wrapper around parent method that shows which plot configuration file will be used.

__annotations__ = {}

__module__ = 'utopya.plotting'

Examples ¶

Running a model ¶

The simplest way was already demonstrated above. Let’s do something more elaborate here:

# Run the default configuration for 100 steps
mv = ffm.create_mv(parameter_space=dict(num_steps=100))
mv.run()

# Use a run configuration file, but update it
mv = ffm.create_mv(run_cfg_path='/absolute/path/to/run_cfg.yml',
                   parameter_space=dict(seed=42, write_every=7))
mv.run_sweep()

With Utopia working so heavily with configuration files, it would be tedious to specify an absolute path for configuration files. Instead, create_mv also allows the from_cfg argument, where a path can be given relative to a base directory. The base directory needs to be specified at initialization of the Model instance:

import os

# An FFM model instance that has the current working directory as its base
ffm = utopya.Model(name='ForestFire', base_dir=os.getcwd())

mv = ffm.create_mv(from_cfg="some/run_cfg.yml")
mv.run_sweep()

Note

Although work happens interactively, the data is still stored in the regular output directory. You don’t have to worry that the simulation data won’t be saved.

If, on the other hand, you do not want the simulation data saved, the use_tmpdir argument might be of interest to you (see the utopya.model module documentation in the utopya API ). The argument is available on all methods that produce a Multiverse and leads to the creation of a temporary directory, which is automatically deleted when the Model instance goes out of scope. The default for this can be set when initializing the Model, but can also be specified separately for each call.

Working with an already finished run ¶

If you want to work with an already executed simulation, you will need to instantiate a FrozenMultiverse. You can do so via Model.create_frozen_mv, which then returns such an object.

If you do not specify any other parameters, this will inspect the output directory of the chosen model and couple to the run with the most recent timestamp (see FrozenMultiverse.__init__ for available arguments.)

Loading and navigating the data ¶

After instantiating some multiverse object, you can grab its DataManager and ask it to load the data from the output directory:

mv = ffm.create_mv(from_cfg="my_sweep_cfg.yml")
mv.run_sweep()

# Load the data and print a tree of the loaded data
dm = mv.dm
dm.load_from_cfg(print_tree=True)

The DataManager instance provides a dict-like interface with which you can navigate through the tree and access the simulation data as well as all the configuration files.

For example, all the multiverse data is stored under the multiverse path. To perform an operation for each of the simulated universes, you can just iterate over the MultiverseGroup:

for uni in dm['multiverse'].values():
    # Get the universe configuration
    cfg = uni['cfg']

    # Get some data
    tree_density = uni['data/ForestFire/tree_density']

    # Do something with the data
    # ...

Note

Utopia requires working with multi-dimensional data. As numpy can be very confusing in this regard, the xarray package is used for almost all of the data handling. We highly recommended having a look at how to work with labelled dimensions and coordinates; it will make your life much easier!

Plotting ¶

To create plots, access the PlotManager via the pm property of your current multiverse object.

You can then create the default plots using the plot_from_cfg method. To create a single plot, use the plot method instead. See the API reference for more information.

Note

The PlotManager will store the plot directly in the output directory. There currently is no easy option to view the plot directly.

Todo

Expand this section.

Working Interactively with Utopia

Contents

Working Interactively with Utopia¶