Handling Data

Handling Data#

In this section we will take a look at the Utopia frontend’s capabilities to handle simulation data.

Hint

These capabilities are implemented in dantro. See the dantro documentation for more information.

The DataManager#

Objects of this class are the home of all your simulation data. One such DataManager object is initialized together with the Multiverse and thereafter available as dm attribute. It is set up with a load configuration and, upon invocation of its load_from_cfg() method, will load the simulation data using that configuration. It is equipped to handle hierarchical data, storing it as a data tree.

Hint

To visually inspect the tree representation, you can use the tree property via print(dm.tree). This also works with every group-like member of the tree.

This functionality is all based on the dantro package, which provides a uniform interface to handle hierarchically structured data. However, while the interface is uniform, the parts of the data tree can be adapted to ideally handle the underlying data.

One example of a specialization is the GridDC class, which is a specialization of a data container that represents data from a grid. It is tightly coupled to the data output of Utopia on C++ side, where the most efficient way to write data is along the index of the entities, rather than the x and y coordinates. However, to handle that data, one expects data with the dimensions x, y, and time; the GridDC takes care of reshaping that data in this way.

Handling Large Amounts of Data#

To handle large amounts of simulation data (which is not uncommon), the DataManager provides so-called proxy loading for HDF5 data: instead of loading the data directly into memory, the structure and metadata of the HDF5 file is used only to generate the data tree. At the place where normally the data would be stored in the data containers, a proxy object is placed (in this case: Hdf5Proxy). Upon access to the data, the proxy gets automatically resolved, leading to the data being loaded into memory and replacing the proxy object in the data container.

Objects that were loaded as proxy are marked with (proxy) in the tree representation. To load HDF5 data as proxy, use the hdf5_proxy loader in the Default Load Configuration.

These proxy objects already make handling large amounts of data much easier, because the data is only loaded if needed.

Loading files in parallel#

Despite data being loaded into proxy objects, this process can take a considerable amount of time if there are many groups or datasets in the to-be-loaded HDF5 files. In such a case, the DataManager will be busy mostly with creating the corresponding Python objects, and less so with loading the actual data from the files. (In other words, this would be a task that is CPU-limited, not I/O limited.)

Subsequently, there is a benefit in using multiple CPUs to build the data tree in such scenarios. The dantro data loading interface supports parallel loading and Utopia allows to control this behavior directly via the CLI:

utopia eval MyModel --load-parallel

The above command will enable parallel loading and it will use all available CPUs for that; see the CLI --help for details.

If you want more control, you can also directly configure it via the meta-configuration. Have a look at the corresponding section in the Utopia Multiverse configuration for available options, e.g. for using parallel loading depending on the number of files or their total file size:

parallel:
  enabled: false

  # Number of processes to use; negative is deduced from os.cpu_count()
  processes: ~

  # Threshold values for parallel loading; if any is below these
  # numbers, loading will *not* be in parallel.
  min_files: 5
  min_total_size: 104857600  # 100 MiB

Hint

The parallel option is basically available for every entry in the data_manager.load_cfg. However, given the constant overhead of starting new loader processes, it makes most sense for the data entry, where the HDF5 files’ content is loaded.

How about huge amounts of data?#

There will be scenarios in which the data to be analyzed exceeds the limits of the physical memory of the machine. Here, proxy objects don’t help, as they only postpone the loading.

For that purpose, dantro, which heavily relies on xarray for the representation of numerical data, is able to make use of its dask integration. The dask package allows working on chunked data, e.g. HDF5 data, and only loads those parts that are necessary for a calculation, afterwards freeing up the memory again. Additionally, it does clever things by first building a tree of operations that are to be performed, then optimizing that tree, and only when the actual numerical result is needed, does the data need to be loaded. Furthermore, as the data is chunked, it can potentially profit from parallel computation. More on that can be found here.

To use dask when loading Utopia data, arguments need to be passed to the proxy that it should not be resolved as the actual data, but as a dask representation of it. This is done by setting the resolve_as_dask flag. Arguments can be passed to the proxy by adding the proxy_kwargs argument to the configuration of a data entry. Add the following part to the root level of your run configuration, which will update the defaults:

data_manager:
  load_cfg:
    data:
      proxy_kwargs:
        resolve_as_dask: true

parameter_space:
  # ... your usual arguments

Note

When plotting data via utopia eval, you can also specify a run configuration. Check the utopia eval --help to find out how.

Once this succeeded, you will see proxy (hdf5, dask) in the tree representation of your loaded data.

There are two other ways to set this entry (following Utopia’s configuration hierarchy principle):

  • In the CLI, you can additionally use the --set-cfg argument for utopia eval and utopia run to set the entry:

utopia eval MyModel --set-cfg data_manager.load_cfg.data.proxy_kwargs.resolve_as_dask=true

  • To permanently set this entry, you can write it to your user configuration:

utopia config user --get --set data_manager.load_cfg.data.proxy_kwargs.resolve_as_dask=true

This then applies to all models you work with. As dask does slow down some operations, it only makes sense to set this if you are mostly working with large data and tend to forget enabling dask!

Configuration and API Reference#

Default Load Configuration#

Below, the default DataManager configuration is included, which also specifies the default load configuration. Each entry of the load_cfg key refers to one so-called “data entry”. Files that match the glob_str are loaded using a certain loader and placed at a target_path within the data tree.

# The DataManager takes care of loading the data into a tree-like structure
# after the simulations are finished.
# It is based on the DataManager class from the dantro package. See there for
# full documentation.
data_manager:
  # Where to create the output directory for this DataManager, relative to
  # the run directory of the Multiverse.
  out_dir: eval/{timestamp:}
  # The {timestamp:} placeholder is replaced by the current timestamp such that
  # future DataManager instances that operate on the same data directory do
  # not create collisions.
  # Directories are created recursively, if they do not exist.

  # Define the structure of the data tree beforehand; this allows to specify
  # the types of groups before content is loaded into them.
  # NOTE The strings given to the Cls argument are mapped to a type using a
  #      class variable of the DataManager
  create_groups:
    - path: multiverse
      Cls: MultiverseGroup

  # Where the default tree cache file is located relative to the data
  # directory. This is used when calling DataManager.dump and .restore without
  # any arguments, as done e.g. in the Utopia CLI.
  default_tree_cache_path: data/.tree_cache.d3

  # Supply a default load configuration for the DataManager
  # This can then be invoked using the dm.load_from_cfg() method.
  load_cfg:
    # Load the frontend configuration files from the config/ directory
    # Each file refers to a level of the configuration that is supplied to
    # the Multiverse: base <- user <- model <- run <- update
    cfg:
      loader: yaml                          # The loader function to use
      glob_str: 'config/*.yml'              # Which files to load
      ignore:                               # Which files to ignore
        - config/parameter_space.yml
        - config/parameter_space_info.yml
        - config/full_parameter_space.yml
        - config/full_parameter_space_info.yml
        - config/git_info_project.yml
        - config/git_info_framework.yml
      required: true                        # Whether these files are required
      path_regex: config/(\w+)_cfg.yml      # Extract info from the file path
      target_path: cfg/{match:}             # ...and use in target path

    # Load the parameter space object into the MultiverseGroup attributes
    pspace:
      loader: yaml_to_object                # Load into ObjectContainer
      glob_str: config/parameter_space.yml
      required: true
      load_as_attr: true
      unpack_data: true                     # ... and store as ParamSpace obj.
      target_path: multiverse

    # Load the configuration files that are generated for _each_ simulation
    # These hold all information that is available to a single simulation and
    # are in an explicit, human-readable form.
    uni_cfg:
      loader: yaml
      glob_str: data/uni*/config.yml
      required: true
      path_regex: data/uni(\d+)/config.yml
      target_path: multiverse/{match:}/cfg
      parallel:
        enabled: true
        min_files: 1000
        min_total_size: 1048576  # 1 MiB

    # Example: Load the binary output data from each simulation.
    # data:
    #   loader: hdf5_proxy
    #   glob_str: data/uni*/data.h5
    #   required: true
    #   path_regex: data/uni(\d+)/data.h5
    #   target_path: multiverse/{match:}/data
    #   enable_mapping: true   # see DataManager for content -> type mapping

    #   # Options for loading data in parallel (speeds up CPU-limited loading)
    #   parallel:
    #     enabled: false

    #     # Number of processes to use; negative is deduced from os.cpu_count()
    #     processes: ~

    #     # Threshold values for parallel loading; if any is below these
    #     # numbers, loading will *not* be in parallel.
    #     min_files: 5
    #     min_total_size: 104857600  # 100 MiB

Note that this is extended by the Utopia Multiverse configuration.