rising.loading¶
DataLoader¶
DataLoader¶
default_transform_call¶
BatchTransformer¶
patch_worker_init_fn¶
patch_collate_fn¶
Dataset¶
- class rising.loading.dataset.AsyncDataset(data_path, load_fn, mode='append', num_workers=0, verbose=False, **load_kwargs)[source][source]¶
Bases:
Dataset
A dataset to preload all the data and cache it for the entire lifetime of this class.
- Parameters
data_path (
Union
[Path
,str
,list
]) – the path(s) containing the actual data samplesload_fn (
Callable
) – function to load the actual datamode (
str
) – whether to append the sample to a list or to extend the list by it. Supported modes are:append
andextend
. Default:append
num_workers (
Optional
[int
]) – the number of workers to use for preloading.0
means, all the data will be loaded in the main process, whileNone
means, the number of processes will default to the number of logical cores.verbose (
bool
) – whether to show the loading progress.**load_kwargs – additional keyword arguments. Passed directly to
load_fn
Warning
if using multiprocessing to load data, there are some restrictions to which
load_fn()
are supported, please refer to thedill
orpickle
documentation- static _add_item(data, item, mode)[source][source]¶
Adds items to the given data list. The actual way of adding these items depends on
mode
- class rising.loading.dataset.Dataset(*args, **kwargs)[source][source]¶
Bases:
Dataset
Extension of
torch.utils.data.Dataset
by aget_subset
method which returns a sub-dataset.
Dataset¶
- class rising.loading.dataset.Dataset(*args, **kwargs)[source][source]¶
Bases:
Dataset
Extension of
torch.utils.data.Dataset
by aget_subset
method which returns a sub-dataset.
AsyncDataset¶
- class rising.loading.dataset.AsyncDataset(data_path, load_fn, mode='append', num_workers=0, verbose=False, **load_kwargs)[source][source]¶
Bases:
Dataset
A dataset to preload all the data and cache it for the entire lifetime of this class.
- Parameters
data_path (
Union
[Path
,str
,list
]) – the path(s) containing the actual data samplesload_fn (
Callable
) – function to load the actual datamode (
str
) – whether to append the sample to a list or to extend the list by it. Supported modes are:append
andextend
. Default:append
num_workers (
Optional
[int
]) – the number of workers to use for preloading.0
means, all the data will be loaded in the main process, whileNone
means, the number of processes will default to the number of logical cores.verbose (
bool
) – whether to show the loading progress.**load_kwargs – additional keyword arguments. Passed directly to
load_fn
Warning
if using multiprocessing to load data, there are some restrictions to which
load_fn()
are supported, please refer to thedill
orpickle
documentation- static _add_item(data, item, mode)[source][source]¶
Adds items to the given data list. The actual way of adding these items depends on
mode
dill_helper¶
load_async¶
Collation¶
- rising.loading.collate.do_nothing_collate(batch)[source][source]¶
Returns the batch as is (with out any collation :type batch:
Any
:param batch: input batch (typically a sequence, mapping or mixture of those).- Returns
the batch as given to this function
- Return type
Any
- rising.loading.collate.numpy_collate(batch)[source][source]¶
function to collate the samples to a whole batch of numpy arrays. PyTorch Tensors, scalar values and sequences will be casted to arrays automatically.
- Parameters
batch (
Any
) – a batch of samples. In most cases either sequence, mapping or mixture of them- Returns
- collated batch with optionally converted type
(to
numpy.ndarray
)
- Return type
Any
- Raises
TypeError – When batch could not be collated automatically
numpy_collate¶
- rising.loading.collate.numpy_collate(batch)[source][source]¶
function to collate the samples to a whole batch of numpy arrays. PyTorch Tensors, scalar values and sequences will be casted to arrays automatically.
- Parameters
batch (
Any
) – a batch of samples. In most cases either sequence, mapping or mixture of them- Returns
- collated batch with optionally converted type
(to
numpy.ndarray
)
- Return type
Any
- Raises
TypeError – When batch could not be collated automatically