Working Interactively with Utopia

Working Interactively with Utopia#

Via utopya, you also have full access to all of Utopia in an interactive fashion, e.g. in an IPython session or in a Jupyter Notebook. This is especially useful for exploring some model parameters and digging through the data generated by your model.

Setting up an Interactive Session#

Note

All of the below needs to happen inside the virtual environment that utopya was installed into.

With IPython#

After installing IPython into the virtual environment, you can enter an interactive session simply via:

python -m IPython

Unlike the ipython command, this ensures that it uses the python binary of the virtual environment, which has access to the utopya installation.

With Jupyter Notebook#

You can do the same with Jupyter Notebooks. For this, you just have to take care to select the correct kernel, i.e.: the one provided by the utopia-env and the corresponding packages.

To do so, use pip to install jupyter, if you haven’t already done so. Then, install ipykernel and manually add a kernel to jupyter:

ipython kernel install --user --name=projectname

When running jupyter notebook, you should now be able to select a kernel from the dropdown menu in the top right-hand corner.

With the Docker Container#

You can also work interactively with Utopia when using the Utopia docker container. As part of the ccees/utopia docker image, Jupyter is already pre-installed and a notebook server is started alongside the container. The idea is that you connect to that notebook server with the browser on your host system.

Follow the detailed instructions on the corresponding Docker Hub page to find out more.

Basic Concepts#

This section aims to convey the basic concepts of the interactive utopya interface.

The Model class#

Objects of this class are your main tool when working with Utopia interactively.

Note

The frontend’s Python Model class should not to be confused with the C++ library’s Model base class.

When you want to run a certain model, say ForestFire, the first thing to do is to create an instance of the Model class.

import utopya

ffm = utopya.Model(name="ForestFire")

This object now provides an interface to perform one or many simulation runs, load the data, and continue working with it. It basically manages the model invocation and loading, and has the same capabilities the CLI has; in fact, the CLI is just a wrapper around the Model class.

For example, to run a model, you can simply call:

mv, dm = ffm.create_run_load()

This will create a Multiverse (see below), run a simulation (here: with the default parameters), and then load the data. It returns the Multiverse object and the corresponding DataManager, for convenience.

Hint

You can perform multiple runs with one single Model instance, so it suffices to create one such instance for a model in the beginning of your interactive session.

Note

The Model is associated with one specific ModelInfoBundle, which contains all the information that is required to run the selected model. By using the Model class and its interface, you don’t need to worry about passing this info bundle around.

For more information on the available interface, have a look at the utopya.model.Model class and the following methods:

The Multiverse class#

This class actually carries out the simulation run, i.e. running multiple so-called “universes” (independent simulations with different parameters). Upon completion, it provides the interface to an associated utopya DataManager and PlotManager, which let you load the data and create plots, respectively. If you want to create a Multiverse object from your existing Model object, use its create_mv() method.

Note

Each Multiverse can only invoke its run() method once.

The FrozenMultiverse class#

If you have already performed a run and only want to load and work with its data, you don’t need a whole Multiverse with all its simulation capabilities. Instead, you want a frozen state of it, after the simulation.

The FrozenMultiverse is just what you need: it couples to some output directory and then lets you continue working with the DataManager and PlotManager, just as you would directly after a simulation run. After initialization, the API is the same as for the Multiverse.

To create such an instance from the Model class, use the Model.create_frozen_mv method (see above for documentation).

The DataManager#

This is the home of all your simulation data for a simulation run. It provides a dict-like interface to navigate through the data tree, consisting of data groups (branching points of the tree) and data containers (leaves of the tree).

When created by a Multiverse instance, it already has the required default configuration available (data_manager key of the meta configuration). The most important methods are implemented in the dantro.data_mngr module:

The PlotManager#

The PlotManager – well, manages the plotting.

The most important methods are implemented in the dantro.plot_mngr module:

Examples#

Running a model#

The simplest way was already demonstrated above. Let’s do something more elaborate here:

# Run the default configuration for 100 steps
mv = ffm.create_mv(parameter_space=dict(num_steps=100))
mv.run()

# Use a run configuration file, but update it
mv = ffm.create_mv(run_cfg_path='/absolute/path/to/run_cfg.yml',
                   parameter_space=dict(seed=42, write_every=7))
mv.run_sweep()

With Utopia working so heavily with configuration files, it would be tedious to specify an absolute path for configuration files. Instead, create_mv() also allows the from_cfg argument, where a path can be given relative to a base directory. The base directory needs to be specified at initialization of the Model instance:

import os

# An FFM model instance that has the current working directory as its base
ffm = utopya.Model(name='ForestFire', base_dir=os.getcwd())

mv = ffm.create_mv(from_cfg="some/run_cfg.yml")
mv.run_sweep()

Note

Although work happens interactively, the data is still stored in the regular output directory. You don’t have to worry that the simulation data won’t be saved.

If, on the other hand, you do not want the simulation data saved, the use_tmpdir argument might be of interest to you, see Model. The argument is available on all methods that produce a Multiverse and leads to the creation of a temporary directory, which is automatically deleted when the Model instance goes out of scope. The default for this can be set when initializing the Model, but can also be specified separately for each call.

Working with an already finished run#

If you want to work with an already executed simulation, you will need to instantiate a FrozenMultiverse. You can do so via create_frozen_mv(), which then returns such an object.

If you do not specify any other parameters, this will inspect the output directory of the chosen model and couple to the run with the most recent timestamp (see FrozenMultiverse for available arguments).

Loading and navigating the data#

After instantiating some multiverse object, you can grab its DataManager and ask it to load the data from the output directory:

mv = ffm.create_mv(from_cfg="my_sweep_cfg.yml")
mv.run_sweep()

# Load the data and print a tree of the loaded data
dm = mv.dm
dm.load_from_cfg(print_tree=True)

The DataManager instance provides a dict-like interface with which you can navigate through the tree and access the simulation data as well as all the configuration files.

For example, all the multiverse data is stored under the Multiverse path. To perform an operation for each of the simulated universes, you can just iterate over the MultiverseGroup:

for uni in dm['multiverse'].values():
    # Get the universe configuration
    cfg = uni['cfg']

    # Get some data
    tree_density = uni['data/ForestFire/tree_density']

    # Do something with the data
    # ...

Note

Utopia requires working with multi-dimensional data. As numpy can be very confusing in this regard, the xarray package is used for almost all of the data handling. We highly recommended having a look at how to work with labelled dimensions and coordinates; it will make your life much easier!

Plotting#

To create plots, access the PlotManager via the pm property of your current Multiverse object.

You can then create the default plots using the plot_from_cfg() method. To create a single plot, use the plot() method instead. See the API reference for more information.

Note

The PlotManager will store the plot directly in the output directory. There currently is no easy option to view the plot directly.

Todo

Expand this section.