Utopia::DataIO::_chunk_helpers Namespace Reference

Functions
template<typename Cont , typename Predicate >
std::vector< unsigned short >	find_all_idcs (Cont &vec, Predicate pred)
	Finds all indices of elements in a vector that matches the given predicate. More...
template<typename Cont = std::vector< hsize_t >>
std::string	to_str (const Cont &vec)
	Helper function to create a string representation of containers. More...
template<typename Cont , typename Logger >
void	opt_chunks_target (Cont &chunks, double bytes_target, const hsize_t typesize, const unsigned int CHUNKSIZE_MAX, const unsigned int CHUNKSIZE_MIN, const bool larger_high_dims, const Logger &log)
	Optimizes the chunks along all axes to find a good default. More...
template<typename Cont , typename Logger >
void	opt_chunks_with_max_extend (Cont &chunks, const Cont &max_extend, const hsize_t typesize, const unsigned int CHUNKSIZE_MAX, const bool opt_inf_dims, const bool larger_high_dims, const Logger &log)
	Optimize chunk sizes using max_extend information. More...

Function Documentation

find_all_idcs()

template<typename Cont , typename Predicate >

std::vector< unsigned short > Utopia::DataIO::_chunk_helpers::find_all_idcs	(	Cont &	vec,
		Predicate	pred
	)

Finds all indices of elements in a vector that matches the given predicate.

Parameters

vec	The object to find the indices in
pred	The predicate to determine the indices to be found

Template Parameters

Cont	The container type
Predicate	The predicate type

opt_chunks_target()

template<typename Cont , typename Logger >

void Utopia::DataIO::_chunk_helpers::opt_chunks_target	(	Cont &	chunks,
		double	bytes_target,
		const hsize_t	typesize,
		const unsigned int	CHUNKSIZE_MAX,
		const unsigned int	CHUNKSIZE_MIN,
		const bool	larger_high_dims,
		const Logger &	log
	)

Optimizes the chunks along all axes to find a good default.

This algorithm is only aware of the current size of the chunks and the target byte size of a chunk. Given that information, it either tries to reduce the extend of chunk dimensions, or enlarge it. To do that, it iterates over all chunk dimensions and either doubles the extend or halves it. Once within 50% of the target byte size, the algorithm stops. Also, it takes care to remain within the bounds of CHUNKSIZE_MAX and CHUNKSIZE_MIN. If a target byte size outside of these bounds is given, it will adjust it accordingly. For a typesize larger than CHUNKSIZE_MAX, this algorithm cannot perform any reasonable actions and will throw an exception; this case should be handled outside of this function!

Parameters

chunks	The current chunk values that are to be optimized
bytes_target	Which byte size to optimize the chunks to
typesize	The byte size of a single entry, needed to calculate the total bytesize of a whole chunk
CHUNKSIZE_MAX	The maximum allowed bytesize of a chunk
CHUNKSIZE_MIN	The minimum allowed bytesize of a chunk
larger_high_dims	If true, dimensions with high indices will be favoured for enlarging chunk extend in that dim
log	The logger object to use

opt_chunks_with_max_extend()

template<typename Cont , typename Logger >

void Utopia::DataIO::_chunk_helpers::opt_chunks_with_max_extend	(	Cont &	chunks,
		const Cont &	max_extend,
		const hsize_t	typesize,
		const unsigned int	CHUNKSIZE_MAX,
		const bool	opt_inf_dims,
		const bool	larger_high_dims,
		const Logger &	log
	)

Optimize chunk sizes using max_extend information.

This algorithm is aware of the maximum extend of a dataset and can use that information during optimization, aiming to increase the size of the chunks towards CHUNKSIZE_MAX as far as possible without going beyond max_extend. The paradigm here is that the number of chunks needed for read/write operations should be minimized while trying to keep a chunk's byte size below a certain value. The algorithm distinguishes between dimensions that have a finite extend and those that can grow to H5S_UNLIMITED, i.e. "infinite" extend. First, the aim is to try to cover the max_extend in the finite dimensions. It checks if an integer multiple is needed to reach the maximum extend. If, after that, the target CHUNKSIZE_MAX is not yet reached and the opt_inf_dims flag is set, the chunk sizes in the unlimited dimensions are extended as far as possible, assuming that they were chosen unlimited because they will be filled at some point and larger chunk sizes will reduce the number of chunks needed during read/write operations.

Parameters

chunks	The current chunk values that are to be optimized
max_extend	The maximum extend of the dataset
typesize	The byte size of a single entry, needed to calculate the total bytesize of a whole chunk
CHUNKSIZE_MAX	The maximum allowed bytesize of a chunk
opt_inf_dims	Whether to optimize the infinite dimensions or not
larger_high_dims	If true, dimensions with high indices will be favoured for enlarging chunk extend in that dim
log	The logger object to use

Template Parameters

Cont	The container type for the chunks
Logger	The logger type

to_str()

template<typename Cont = std::vector< hsize_t >>

std::string Utopia::DataIO::_chunk_helpers::to_str

(

const Cont &

vec

)

Helper function to create a string representation of containers.