DataIO

Writing data, reading configuration files, and reporting data to the Python frontend. More...

Collaboration diagram for DataIO:

Modules
	DataManager
	High-level interface for data writing.
	Graph Utilities
	Provides functions which wrap common tasks for writing out data from graphs.
	Chunking Utilities
	Provides algorithms for automatically optimize the chunksize in which data is written to the hard disk when writing compressed or extendable hdf5 files.
	HDF5
	Low-level interface for using HDF5 with modern C++ and STL.
	Monitor
	Interface for monitoring custom quantities for a simulation via standard out.
	Config Utilities
	Allows reading configuration entries from YAML files.
	Filesystem

Namespaces
namespace	Utopia::DataIO::GraphLoad
namespace	Utopia::DataIO::GraphUtilsHelper

Functions
template<class Exc >
YAML::Exception	Utopia::DataIO::improve_yaml_exception (const Exc &e, const Config &node, std::string prefix={})
	Improves yaml-cpp exceptions occurring for a given node.
std::string	Utopia::DataIO::to_string (const Config &node)
	Given a config node, returns a string representation of it.
template<typename CVecT , DimType dim = 0>
CVecT	Utopia::DataIO::get_as_arma_vec (const std::string &key, const DataIO::Config &node)
	Retrieve a config entry as Armadillo column vector using get_.
template<class T , class Keys = std::list<std::string>>
Config	Utopia::_internal::__recursive_setitem (Config d, Keys &&key_sequence, const T &val)
	Helper function for recursive_setitem.
Config	Utopia::_internal::recursive_getitem (const Config &d, const std::vector< std::string > &key_sequence)
	Recursively retrieve an element from the configuration tree.
Config	Utopia::_internal::recursive_getitem (const Config &d, const std::string &key_sequence, const std::string &delims=".")
	Overload for recursive_getitem, accepting a string-like key sequence.
template<class T >
void	Utopia::_internal::recursive_setitem (Config &d, std::list< std::string > key_sequence, const T val)
	Recursively sets an element in a configuration tree.
template<class T >
void	Utopia::_internal::recursive_setitem (Config &d, const std::string &key_sequence, const T val, const std::string &delims=".")
	Overload for recursive_setitem that splits a string into a key sequence.

Detailed Description

Writing data, reading configuration files, and reporting data to the Python frontend.

Function Documentation

__recursive_setitem()

template<class T , class Keys = std::list<std::string>>

Config Utopia::_internal::__recursive_setitem	(	Config	d,
		Keys &&	key_sequence,
		const T &	val
	)

Helper function for recursive_setitem.

Expects an (already deep-copied) node that is then recursively iterated through along the key sequence. For the last key in the sequence, the value is being set. Empty path segments in the key sequence are skipped.

Exceptions

std::invalid_argument

upon trailing empty path segments in the key sequence.

{
    // Get the next key
    std::string key;
    while (key.empty() and not key_sequence.empty()) {
        key = key_sequence.front();
        key_sequence.pop_front();
    }
    if (key.empty()) {
        throw std::invalid_argument(
            "During recursive_setitem, failed to retrieve a valid key for "
            "continuing recursion. Make sure the given key sequence ("
            + join(key_sequence, " -> ") + ") contains no empty elements!"
        );
    }
 
    // Check for end of recursion
    if (key_sequence.empty()) {
        if (d.IsScalar()) {
            // Discard the scalar and replace it with an empty mapping
            d = Config{};
        }
        d[key] = val;
        return d;
    }
 
    // Continue recursion, creating the intermediate node if it does not exist
    if (not d[key]) {
        d[key] = Config{};
    }
    d[key] = __recursive_setitem(d[key], std::forward<Keys>(key_sequence),
                                 val);
    return d;
}

get_as_arma_vec()

template<typename CVecT , DimType dim = 0>

CVecT Utopia::DataIO::get_as_arma_vec	(	const std::string &	key,
		const DataIO::Config &	node
	)

Retrieve a config entry as Armadillo column vector using get_.

Note

This method is necessary because arma::Col::fixed cannot be constructed from std::vector. In such cases, the target vector is constructed element-wise.

Template Parameters

CVecT	The Armadillo vector type to return
dim	The dimensionality of the vector (only needed for)

{
    // Extract the field vector element type; assuming Armadillo interface
    using element_t = typename CVecT::elem_type;
 
    // Check if it can be constructed from a vector
    if constexpr (std::is_constructible< CVecT, std::vector< element_t > >())
    {
        return get_as< std::vector< element_t > >(key, node);
    }
    else
    {
        static_assert(dim > 0,
                      "Need template argument dim given if target type is not "
                      "constructible from std::vector.");
 
        // Needs to be constructed element-wise
        CVecT      cvec;
        const auto vec = get_as< std::array< element_t, dim > >(key, node);
 
        for (DimType i = 0; i < dim; i++)
        {
            cvec[i] = vec[i];
        }
 
        return cvec;
    }
}

improve_yaml_exception()

template<class Exc >

YAML::Exception Utopia::DataIO::improve_yaml_exception	(	const Exc &	e,
		const Config &	node,
		std::string	prefix = {}
	)

Improves yaml-cpp exceptions occurring for a given node.

                                            {})
{
    // The string stream for the new, improved error message
    std::stringstream e_msg;
    e_msg << prefix << " ";
    e_msg << "Got " << boost::core::demangle(typeid(e).name()) << ". ";
 
    // Create a custom error message depending on whether the node is a
    // zombie or a mark is available
    if (not node)
    {
        // Was a zombie
        e_msg << "The given node was a Zombie! Check that the key you are "
                 "trying to read from actually exists. ";
    }
    else if (not node.Mark().is_null())
    {
        // A mark is available to use as a hint
        // NOTE Mark() provides the line and column in the config file, if
        //      available, i.e.: if not a zombie node
        e_msg << "Check that the corresponding line of the config file "
                 "matches the desired read operation or type conversion. ";
    }
 
    // Give some information on the node's content:
    e_msg << "The content of the node is:" << std::endl << YAML::Dump(node);
 
    // Return the custom exception object; can be thrown on other side
    return YAML::Exception(node.Mark(), e_msg.str());
}

recursive_getitem() [1/2]

Config Utopia::_internal::recursive_getitem	(	const Config &	d,
		const std::string &	key_sequence,
		const std::string &	delims = "."
	)

Overload for recursive_getitem, accepting a string-like key sequence.

Returns the configuration node at the path specified by the key sequence.

Parameters

d	The configuration tree to retrieve the element from
key_sequence	A string denoting the path within the tree to get the item from. The path is split into segments using Utopia::split
delims	Delimiters (plural!) that split the string into a key sequence.

Returns

Config The configuration tree element at the given path.

Warning

Any character in the delims sequence acts as separator. See Utopia::split for more information on behaviour of the delims argument.

{
    return recursive_getitem(d, split(key_sequence, delims));
}

recursive_getitem() [2/2]

Config Utopia::_internal::recursive_getitem	(	const Config &	d,
		const std::vector< std::string > &	key_sequence
	)

Recursively retrieve an element from the configuration tree.

Actually uses a loop internally to avoid recursion... ¯_(ツ)_/¯

Note

Only works with string-like keys, i.e. can't access sequences.

Warning

Due to yaml-cpp quirks, creates a deep copy of the given config to keep clear of any mutability side effects. Subsequently, this function should not be used in performance-critical code. Furthermore, to set an element in the configuration tree, the recursive_setitem functionality should be used – assigning a value to the returned Config object will not lead to a change in the tree provided via the d argument.

Parameters

d	The configuration tree to retrieve the element from
key_sequence	A key sequence denoting the path within the tree to get the item from.

Returns

Config The configuration tree element at the given path.

{
    using Utopia::KeyError;
    using Utopia::join;
 
    Config rv = YAML::Clone(d);
    for (const auto& key : key_sequence ) {
        try {
            rv = get_as<Config>(key, rv);
        }
        catch (KeyError& err) {
            throw KeyError(
                key, rv,
                "recursive_getitem failed for key or key sequence '"
                + join(key_sequence, " -> ") + "'!"
            );
        }
    }
    return rv;
}

recursive_setitem() [1/2]

template<class T >

void Utopia::_internal::recursive_setitem	(	Config &	d,
		const std::string &	key_sequence,
		const T	val,
		const std::string &	delims = "."
	)

Overload for recursive_setitem that splits a string into a key sequence.

Parameters

d	The configuration tree to set the entry in.
key_sequence	A string to split into a sequence of keys to walk along inside the tree.
val	The value to assign to the element of the tree specified by the key sequence.
delims	Delimiters to generate a key sequence from the given string key_sequence.

Warning

Any character in the delims sequence acts as separator. See Utopia::split for more information on behaviour of the delims argument.

{
    recursive_setitem(
        d, split<std::list<std::string>>(key_sequence, delims), val
    );
}

recursive_setitem() [2/2]

template<class T >

void Utopia::_internal::recursive_setitem	(	Config &	d,
		std::list< std::string >	key_sequence,
		const T	val
	)

Recursively sets an element in a configuration tree.

This also creates all intermediate segments that may be missing.

As the YAML::Node::operator[] does not return references, this internally has to create a (deep) copy of the given node d. After recursing through it and setting the value, it will finally assign to the d reference.

Note

Only works with string-like keys, i.e. can't access sequences.

Parameters

d	The configuration tree to set the entry in.
key_sequence	A container with a sequence of keys to walk along inside the tree.
val	The value to assign to the element of the tree specified by the key sequence.

{
    if (key_sequence.empty()) {
        throw std::invalid_argument(
            "Key sequence for recursive_setitem may not be empty!"
        );
    }
    d = __recursive_setitem(YAML::Clone(d), std::move(key_sequence), val);
}

to_string()

std::string Utopia::DataIO::to_string

(

const Config &

node

)

Given a config node, returns a string representation of it.

This is done by dumping the config node into an std::stringstream.

{
    std::stringstream s;
    s << YAML::Dump(node);
    return s.str();
}