.. _utopya_interactive: Working Interactively with Utopia ================================= Via ``utopya``, you also have full access to all of Utopia in an interactive fashion, e.g. in an `IPython `_ session or in a `Jupyter `_ Notebook. This is especially useful for exploring some model parameters and digging through the data generated by your model. .. contents:: :local: :depth: 2 ---- Setting up an Interactive Session --------------------------------- .. note:: All of the below needs to happen inside the :ref:`virtual environment ` that ``utopya`` was installed into. With IPython ^^^^^^^^^^^^ After installing `IPython `_ into the virtual environment, you can enter an interactive session simply via: .. code-block:: console python -m IPython Unlike the ``ipython`` command, this ensures that it uses the python binary *of the virtual environment*, which has access to the ``utopya`` installation. With Jupyter Notebook ^^^^^^^^^^^^^^^^^^^^^ You can do the same with `Jupyter Notebooks `_. For this, you just have to take care to select the correct kernel, i.e.: the one provided by the ``utopia-env`` and the corresponding packages. To do so, use ``pip`` to install ``jupyter``, if you haven't already done so. Then, install ``ipykernel`` and manually add a kernel to jupyter: .. code-block:: console ipython kernel install --user --name=projectname When running ``jupyter notebook``, you should now be able to select a kernel from the dropdown menu in the top right-hand corner. With the Docker Container ^^^^^^^^^^^^^^^^^^^^^^^^^ You can also work interactively with Utopia when using the `Utopia docker container `_. As part of the ``ccees/utopia`` docker image, Jupyter is already pre-installed and a notebook server is started alongside the container. The idea is that you connect to that notebook server with the browser on your *host* system. Follow the `detailed instructions on the corresponding Docker Hub page `_ to find out more. Basic Concepts -------------- This section aims to convey the basic concepts of the interactive :py:mod:`utopya` interface. The :py:class:`~utopya.model.Model` class ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Objects of this class are your main tool when working with Utopia interactively. .. note:: The frontend's Python :py:class:`~utopya.model.Model` class should not to be confused with the C++ library's ``Model`` base class. When you want to run a certain model, say ``ForestFire``, the first thing to do is to create an instance of the :py:class:`~utopya.model.Model` class. .. testcode:: Model_run_interactively import utopya ffm = utopya.Model(name="ForestFire") This object now provides an interface to perform one or many simulation runs, load the data, and continue working with it. It basically manages the model invocation and loading, and has the same capabilities the CLI has; in fact, the CLI is just a wrapper around the :py:class:`~utopya.model.Model` class. For example, to run a model, you can simply call: .. testcode:: Model_run_interactively mv, dm = ffm.create_run_load() .. testoutput:: Model_run_interactively :hide: :options: +ELLIPSIS ... Initialized PseudoParent ... This will create a :py:class:`~utopya.multiverse.Multiverse` (see below), run a simulation (here: with the default parameters), and then load the data. It returns the :py:class:`~utopya.multiverse.Multiverse` object and the corresponding :py:class:`~utopya.eval.datamanager.DataManager`, for convenience. .. hint:: You can perform multiple runs with one single :py:class:`~utopya.model.Model` instance, so it suffices to create one such instance for a model in the beginning of your interactive session. .. note:: The :py:class:`~utopya.model.Model` is associated with one specific :py:class:`~utopya.model_registry.info_bundle.ModelInfoBundle`, which contains all the information that is required to run the selected model. By using the :py:class:`~utopya.model.Model` class and its interface, you don't need to worry about passing this info bundle around. For more information on the available interface, have a look at the :py:class:`utopya.model.Model` class and the following methods: - :py:meth:`~utopya.model.Model.create_mv` - :py:meth:`~utopya.model.Model.create_run_load` - :py:meth:`~utopya.model.Model.create_frozen_mv` - :py:attr:`~utopya.model.Model.info_bundle` - :py:attr:`~utopya.model.Model.default_model_cfg` The :py:class:`~utopya.multiverse.Multiverse` class ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This class actually carries out the simulation run, i.e. running multiple so-called "universes" (independent simulations with different parameters). Upon completion, it provides the interface to an associated :py:mod:`utopya` :py:class:`~utopya.eval.datamanager.DataManager` and :py:class:`~utopya.eval.plotmanager.PlotManager`, which let you load the data and create plots, respectively. If you want to create a :py:class:`~utopya.multiverse.Multiverse` object from your existing :py:class:`~utopya.model.Model` object, use its :py:meth:`~utopya.model.Model.create_mv` method. .. note:: Each :py:class:`~utopya.multiverse.Multiverse` can only invoke its :py:meth:`~utopya.multiverse.Multiverse.run` method once. The :py:class:`~utopya.multiverse.FrozenMultiverse` class ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you have already performed a run and only want to load and work with its data, you don't need a whole :py:class:`~utopya.multiverse.Multiverse` with all its simulation capabilities. Instead, you want a frozen state of it, after the simulation. The :py:class:`~utopya.multiverse.FrozenMultiverse` is just what you need: it couples to some output directory and then lets you continue working with the :py:class:`~utopya.eval.datamanager.DataManager` and :py:class:`~utopya.eval.plotmanager.PlotManager`, just as you would directly after a simulation run. After initialization, the API is the same as for the :py:class:`~utopya.multiverse.Multiverse`. To create such an instance from the :py:class:`~utopya.model.Model` class, use the ``Model.create_frozen_mv`` method (see above for documentation). The :py:class:`~utopya.eval.datamanager.DataManager` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This is the home of all your simulation data for a simulation run. It provides a dict-like interface to navigate through the data tree, consisting of data groups (branching points of the tree) and data containers (leaves of the tree). When created by a :py:class:`~utopya.multiverse.Multiverse` instance, it already has the required default configuration available (``data_manager`` key of the :ref:`meta configuration `). The most important methods are implemented in the :py:mod:`dantro.data_mngr` module: - :py:meth:`~dantro.data_mngr.DataManager.load_from_cfg` - :py:meth:`~dantro.data_mngr.DataManager.load` The :py:class:`~utopya.eval.plotmanager.PlotManager` ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The :py:class:`~utopya.eval.plotmanager.PlotManager` – well, manages the plotting. The most important methods are implemented in the :py:mod:`dantro.plot_mngr` module: - :py:meth:`~dantro.plot_mngr.PlotManager.plot_from_cfg` - :py:meth:`~dantro.plot_mngr.PlotManager.plot` Examples -------- Running a model ^^^^^^^^^^^^^^^ The simplest way was already demonstrated above. Let's do something more elaborate here: .. code-block:: python # Run the default configuration for 100 steps mv = ffm.create_mv(parameter_space=dict(num_steps=100)) mv.run() # Use a run configuration file, but update it mv = ffm.create_mv(run_cfg_path='/absolute/path/to/run_cfg.yml', parameter_space=dict(seed=42, write_every=7)) mv.run_sweep() With Utopia working so heavily with configuration files, it would be tedious to specify an absolute path for configuration files. Instead, :py:meth:`~utopya.model.Model.create_mv` also allows the ``from_cfg`` argument, where a path can be given *relative* to a base directory. The base directory needs to be specified at initialization of the :py:class:`~utopya.model.Model` instance: .. code-block:: python import os # An FFM model instance that has the current working directory as its base ffm = utopya.Model(name='ForestFire', base_dir=os.getcwd()) mv = ffm.create_mv(from_cfg="some/run_cfg.yml") mv.run_sweep() .. note:: Although work happens interactively, the data is still stored in the regular output directory. You don't have to worry that the simulation data won't be saved. If, on the other hand, you do not *want* the simulation data saved, the ``use_tmpdir`` argument might be of interest to you, see :py:class:`~utopya.model.Model`. The argument is available on all methods that produce a :py:class:`~utopya.multiverse.Multiverse` and leads to the creation of a temporary directory, which is automatically deleted when the :py:class:`~utopya.model.Model` instance goes out of scope. The default for this can be set when initializing the :py:class:`~utopya.model.Model`, but can also be specified separately for each call. Working with an already finished run ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you want to work with an already executed simulation, you will need to instantiate a :py:class:`~utopya.multiverse.FrozenMultiverse`. You can do so via :py:meth:`~utopya.model.Model.create_frozen_mv`, which then returns such an object. If you do not specify any other parameters, this will inspect the output directory of the chosen model and couple to the run with the most recent timestamp (see :py:class:`~utopya.multiverse.FrozenMultiverse` for available arguments). Loading and navigating the data ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ After instantiating some multiverse object, you can grab its :py:class:`~utopya.eval.datamanager.DataManager` and ask it to load the data from the output directory: .. code-block:: python mv = ffm.create_mv(from_cfg="my_sweep_cfg.yml") mv.run_sweep() # Load the data and print a tree of the loaded data dm = mv.dm dm.load_from_cfg(print_tree=True) The :py:class:`~utopya.eval.datamanager.DataManager` instance provides a dict-like interface with which you can navigate through the tree and access the simulation data as well as all the configuration files. For example, all the multiverse data is stored under the :py:class:`~utopya.multiverse.Multiverse` path. To perform an operation for each of the simulated universes, you can just iterate over the :py:class:`~utopya.eval.groups.MultiverseGroup`: .. code-block:: python for uni in dm['multiverse'].values(): # Get the universe configuration cfg = uni['cfg'] # Get some data tree_density = uni['data/ForestFire/tree_density'] # Do something with the data # ... .. note:: Utopia requires working with multi-dimensional data. As numpy can be very confusing in this regard, the `xarray `_ package is used for almost all of the data handling. We highly recommended having a look at how to work with labelled dimensions and coordinates; it will make your life much easier! Plotting ^^^^^^^^ To create plots, access the :py:class:`~utopya.eval.plotmanager.PlotManager` via the ``pm`` property of your current :py:class:`~utopya.multiverse.Multiverse` object. You can then create the default plots using the :py:meth:`~dantro.plot_mngr.PlotManager.plot_from_cfg` method. To create a single plot, use the :py:meth:`~dantro.plot_mngr.PlotManager.plot` method instead. See the API reference for more information. .. note:: The :py:class:`~utopya.eval.plotmanager.PlotManager` will store the plot directly in the output directory. There currently is no easy option to view the plot directly. .. todo:: Expand this section.