Plotting

The Utopia plotting framework is an extension of the dantro plotting framework which adds specializations for the data structures used by Utopia. Furthermore, it extends the available plotting capabilities by providing a basic set of plotting functions.

Note

While this page and the linked documents aim to give an overview over the plot framework in the context of Utopia, the dantro documentation provides the full API reference and more detailed information. To access it, visit the project page or the online dantro documentation.


Nomenclature

In the following, the basic vocabulary to understand the plotting framework is established. Relevant conceps and structures are:

  • The plot configuration, which contains all the parameters required to make one or multiple plots.

  • The plot creator, which creates the actual plot. Given some plot configuration, it outputs the plot.

  • The PlotManager, which orchestrates the plotting procedure by feeding the relevant plot configuration to a specific plot creator.

Plot Configuration

The plotting framework uses a configuration-based approach with the aim of keeping as much information about how a plot should be created in the configuration, rather than in a specific implementation. This makes plotting much simpler and facilitates automation.

In this documentation, plot configurations are usually given in YAML, as this is how configurations are tpyically defined. The standard plot configuration structure is:

---
my_plot:
  creator: some_creator
  # ... plot configuration parameters here ...

my_other_plot:
  creator: another_creator
  # ... plot configuration parameters for this plot ...

This leads to the PlotManager instantiating a plot creator some_creator, which is instructed to create a plot called my_plot. The additional parameters are passed to the plot creator, which then uses these for its own purposes. The same happens for the my_other_plot plot, which uses another_creator. For more information on the PlotManager, refer to the dantro documentation.

Plot configuration entries can also make use of parameter sweeps. Simply add the !pspace tag to the top-level entry:

---
my_plot: !pspace
  some_param: !sweep
    default: foo
    values: [foo, bar, baz]

This will automatically create a separate file for each plot and include the value of the parameter into the file or folder name.

Note

Usually, the term plot configuration refers to the set of parameters required to create a single plot.

Many individual plot configurations can be stored in a YAML file. The top level of that file then denotes the names of the plots. In the above example, these would be my_plot and my_other_plot.

Hint

Plot configuration entries starting with an underscore are ignored by the plot manager:

---
_foobar:        # This entry is ignored
  # ...

my_plot:        # -> creates my_plot
  # ...

my_other_plot:  # -> creates my_other_plot
  # ...

This can be useful when defining YAML anchors that are used in the actual configuration entries.

Plot Configuration Inheritance

New plot configurations can be based on existing ones. This makes it very easy to define various plot functions without copy-pasting the plot configurations. To do so, add the based_on key to your plot configuration. As arguments, you can provide either a string or a sequence of strings, where the strings have to refer to names of so-called “base plot configuration entries”. These are configuration entries that utopya and the models already provide.

For example, the following lines suffice to generate a simple line plot based on the plot function configured as .basic_uni.lineplot:

tree_density:
  based_on: .basic_uni.lineplot

  model_name: ForestFire
  path_to_data: tree_density

Here, the configuration for .basic_uni.lineplot is first loaded and then recursively updated with those keys that are specified below the based_on. When providing a sequence, e.g. based_on: [foo, bar, baz], the first configuration is used as the base and is subsequently recursively updated with those that follow.

For a list of all base plot configurations provided by utopya, see Multiverse Base Configuration.

This feature is completely implemented in dantro; see the plot configuration inheritance entry in the linked documentation for details. For Utopia, the following base configuration pools are made available:

  • The utopya base configuration pool, see Multiverse Base Configuration

  • The {model_name}_base configuration pool for the currently selected model, if available

Configuration sets

Same as run configurations, plot configurations can also be included in Configuration Sets, simply by adding an eval.yml file to the configuration set directory. This allows to define plot configurations for a specific simulation run, directly alongside it.

Hint

To avoid excessive duplication of plot configurations when adding config sets, Plot Configuration Inheritance can be helpful:

  • put shared definitions into the base configuration

  • in the config set, only specify those options that deviate from the default or that should better be explicitly specified

The ExternalPlotCreator

In Utopia, the ExternalPlotCreator has a central role, as it forms the basis of several, more specialized plot creators. The “external” refers to is abiliy to invoke some plot function from an external module or file. Such a plot function can essentially be arbitrary. However, the ExternalPlotCreator has some specialized functionality for working with matplotlib which aims to make plotting more convenient: the style option and the PlotHelper framework. Furthermore, it has access to dantro’s data transformation framework.

In practice, the ExternalPlotCreator itself is hardly used in Utopia, but it is the base class of the UniversePlotCreator and the MultiversePlotCreator. Thus, the following information is valid for both these specializations and is important to understand before looking at the other creators. More detail on the specializations themselves is given later.

Specifying which plotting function to use

Let’s assume we have a plotting function defined somewhere and want to communicate to the ExternalPlotCreator that this function should be used for some plot. For the moment, the exact definition of the function is irrelevant. You can read more about it in below.

Importing a plotting function from a module

To import a plot function, the module and plot_func entries are required. The following example shows a plot that uses a plot function from utopya.plot_funcs and another plot that uses some (importable) package from which the module and the plot function are imported:

---
my_plot:
  # Import some module from utopya.plot_funcs (note the leading dot)
  module: .distribution

  # Use the function with the following name from that module
  plot_func: my_plot_func

  # ... all other arguments

my_other_plot:
  # Import a module from any installed package
  module: my_installed_plotting_package.some_module
  plot_func: my_plot_func

  # ... all other arguments

Importing a plotting function from a file

There are plenty of plot function implementations provided both by utopya and the various Utopia models. However, you might also want to implement a plot function of your own design. This can be achieved by specifying the module_file key instead of the module key in the plot configuration. The python module is then loaded from file and the plot_func key is used to retrieve the plotting function:

---
my_plot:
  # Load the following file as a python module
  module_file: ~/path/to/my/python/script.py

  # Use the function with the following name from that module
  plot_func: my_plot_func

  # ... all other arguments (as usual)

Adjusting a plot’s style

All matplotlib-based plots can profit from this feature. Using the style keyword, matplotlib parameters can be configured fully via the plot configuration; no need to touch the code. Basically, this sets the matplotlib.rcParams and makes the matplotlib stylesheets available. The following example illustrates the usage:

---
my_plot:
  # ...

  # Configure the plot style
  style:
    base_style: ~        # optional, name of a matplotlib style to use
    rc_file: ~           # optional, path to YAML file to load params from
    # ... all further parameters are interpreted directly as rcParams

In the following example, the ggplot style is used and subsequently adjusted by setting the linewidth, marker size and label sizes.

---
my_ggplot:
  # ...

  style:
    base_style: ggplot
    lines.linewidth : 3
    lines.markersize : 10
    xtick.labelsize : 16
    ytick.labelsize : 16

For the base_style entry, choose the name of a matplotlib stylesheet. For valid RC parameters, see the matplotlib customization documentation.

The PlotHelper

The aim of the PlotHelper framework is to let the plot functions focus on what cannot easily be automated: being the bridge between some selected data and its visualization. The plot function should not have to concern itself with plot aesthetics, as these can be easily automated. The PlotHelper framework can make your life significantly easier, as it already takes care of most of the plot aesthetics by making large parts of the matplotlib interface accessible via the plot configuration. That way, you don’t need to touch Python code for trivial tasks like changing the plot limits. It also takes care of setting up a figure and storing it in the appropriate location. Most importantly, it will make your plots future-proof and let them profit from upcoming features. For available plot helpers, have a look at the PlotHelper API reference.

As an example, the following plot configuration sets the title of the plot as well as the labels and limits of the axes:

my_plot:
  # ...

  # Configure the plot helpers
  helpers:
    set_title:
      title: This is My Fancy Plot
    set_labels:
      x: $A$
      y: Counts $N_A$
    set_limits:
      x: [0, max]
      y: [1.0, ~]

Furthermore, notice how you can combine the capabilities of the plot helper framework with the ability to set the plot style.

The data transformation framework

As part of dantro, a data selection and transformation framework based on a directed, acyclic graph (DAG) of operations is provided. This is a powerful tool, especially when combined with the plotting framework.

What motivates using this DAG framework for plotting is similar what motivates the plot helper: ideally, the plot function should focus on the visualization of some data; everything else before (data selection, transformation, etc.) and after (adjusting plot aesthetics, saving the plot, etc.) should be automated. The DAG allows for arbitrary operations, making it a highly versatile and powerful framework. It uses a configuration-based syntax that is optimized for specification via YAML. Additionally, it allows to cache results to a file; this is very useful when the analysis of data takes much longer than the plotting itself.

To learn more, visit the dantro documentation of the DAG transformation framework.

Hint

If you are missing an operation, you can register it yourself using register_operation(). Add something like the following to your model-specific plot module:

"""model_plots/MyModel/__init__.py"""

# Your regular imports here

# --- Register custom operations ...
from utopya.plotting import register_operation

# ... from some imported module
import numpy as np
register_operation(name='np.mean', func=np.mean)

# ... from a lambda
register_operation(name='MyModel', func=lambda d: d**2)

# ... from some custom callable
def my_operation(data, *, some_parameter):
    """Some operation on the given data"""
    # Do something with data and the parameter
    return new_data

register_operation(name='MyModel.my_operation', func=my_operation)

Of course, custom operations can also be defined somewhere else within your plot modules, e.g. an operations.py file, and imported into __init__.py using from .operations import my_operation.

Note that you are not allowed to override any existing operation. To avoid naming conflicts, it is advisable to use a unique name for custom operations, e.g. by prefixing the model name for some model-specific operation.

Important: Your model-specific custom operations should be defined in the model-specific plot module, i.e.: accessible after importing model_plots/<your_model_name>/__init__.py. Prior to plotting, the PlotManager pre-loads that module, such that the register_operation calls are actually invoked.

Implementing plot functions

Below, you will learn how to implement a plot function that can be used with the plot creator.

The is_plot_func() decorator

When defining a plot function, we recommend using this decorator. It takes care of providing essential information to the ExternalPlotCreator and makes it easy to configure those parameters relevant for the plot function. For example, to specify which creator should be used for the plot function, the creator_type can be given. To control usage of the data transformation framework, the use_dag flag can be used and the required_dag_tags argument can specify which data tags the plot function expects.

Other possible plot function signatures

Warning

The examples below are for the ExternalPlotCreator and might need to be adapted for the specialized plot creators.

Examples for those creators are given in the dantro documentation and here.

Without DAG framework

If you wish not to use the data transformation framework, simply omit the use_dag flag or set it to False in the decorator. When not using the transformation framework, the creator_type should be specified, thus binding the plot function to one type of creator.

from utopya import DataManager
from utopya.plotting import is_plot_func, PlotHelper, ExternalPlotCreator

@is_plot_func(creator_type=ExternalPlotCreator)
def my_plot(dm: DataManager, *, hlpr: PlotHelper, **additional_kwargs):
    """A plot function using the plot helper framework.

    Args:
        dm: The DataManager object that contains all loaded data.
        hlpr: The associated plot helper
        **additional_kwargs: Anything else from the plot config.
    """
    # Select some data ...
    data = dm['foo/bar']

    # Create a lineplot on the currently selected axis
    hlpr.ax.plot(data)

    # When exiting here, the plot helper saves the plot.

Note

The dm argument is only provided when not using the DAG framework.

Bare basics

If you really want to do everything by yourself, you can also disable the plot helper framework by passing use_helper=False to the decorator. The hlpr argument is then not passed to the plot function.

There is an even more basic version of doing this, leaving out the is_plot_func() decorator:

from utopya import DataManager

def my_bare_basics_plot(dm: DataManager, *, out_path: str,
                        **additional_kwargs):
    """Bare-basics signature required by the ExternalPlotCreator.

    Args:
        dm: The DataManager object that contains all loaded data.
        out_path: The generated path at which this plot should be saved
        **additional_kwargs: Anything else from the plot config.
    """
    # Your code here ...

    # Save to the specified output path
    plt.savefig(out_path)

Note

When using the bare basics version, you need to set the creator argument in the plot configuration in order for the plot manager to find the desired creator.