SerializedPlugin

class SerializedPlugin

Bases: DataPlugin

__init__()

Parameters:

eid_map_path (str, optional:) – Path to .CSV file specifying mapping of EIDs between UKBB applications
eig_map_from (str, optional:) – Name of column containing EID set to map from.
eid_map_to (str, optional:) – Name of column containing EID set to map to.
transforms (PluginTransform, optional:) – An instance of PluginTransform realizing a mapping component name -> TensorTransform,
col_filter (Filter, optional) – A filter that selects the columns to be used
row_filter (Filter, optional) – A filter that selects the rows to be used
List[str] (tags) – A sequence of tags for later identification purposes of the DataPlugin
optional – A sequence of tags for later identification purposes of the DataPlugin

col_filter: PluginColFilter: The PluginColFilter instance used to filter columns of the plugin

collate_fn(batch) → Any: Function for collating data from this dataplugin.

eid_map: SliceableDict: Maps native eids to non-native eids. This mechanism is used to consolidate eids across different data modalities and their respective DataPlugin instances.

eid_map_rev: SliceableDict: Reverse map, mapping non-native eids to native eids

eids_filtered: Set[Eid]: The eids filtered by the row plugin

features_to_pos: Dict[str, Dict[str, SliceableDict]]: For each component and dimension, maps feature ids to their positions

classmethod from_compressed_pickle(path: str | Path, **kwargs)

Restore previously initialized (called prepare_data and setup) DataPlugin.

Parameters:

path: str | pathlib.Path: Path to serialized pre-prepared DataPlugin.

classmethod from_config(config: DictConfig, custom: Mapping[str, Any] = {}) → DataPlugin

Creates a DataPlugin instance from a DictConfig.

Parameters:

config (DictConfig) – The config object from which to initialize the DataPlugin
custom (dict, optional) – A mapping allowing for custom filters or transforms not defined inside the udm to be used for initialization. Default: {}

Return type:

DataPlugin

Restore previously initialized DataPlugin from WandB artifact.

Parameters:

artifact_identifier: str: Name and version of WandB artifact.
wandb_run:: WandB run, if available.
wandb_entity:: Name of the WandB entity, if no run is given.
wandb_project:: Name of the WandB project, if no run is given.
data_root:: Optional path where plugin is stored.

get_eid_map(from_eid_nr: str | None = None, to_eid_nr: str | None = None) → Dict[int | str, int | str]

Creates eid_mapping_dict {from_eid_nr_idx : to_eid_nr_idx}

Parameters:

from_eid_nr (str) – The identifier of the native eidset to map from
to_eid_nr (str) – The identifier of the eidset to map to and which will be used to acces items after mapping

Note

If the mapping path was not set, the eids will be assumed to be identical with the native eids and an empty dict will be returned.

Important

The separator in the eid_mapping csv file is assumed to be TAB

Return type:: Dict[Eid, Eid]

get_eids(filtered: bool = True) → List[int | str]

Returns the (visible) eids of all samples in this DataPlugin as list.

Return type:: List[Eid]

Note

These are the eids mapped according to the mapping file provided.

Parameters:: filtered (bool) – If False, will also return invisible eids (eids that are filtered out by the DataPlugin row_filter)

abstract get_feature_types() → Dict[str, dtype]

Returns the type of the features for each component.

Essentially, given any valid eid e of a dataplugin, this is equivalent to:

{cname : value.dtype for cname, value in dataplugin[e].items()}

Return type:: Dict[str, torch.dtype]

Example

>>> sample = dataplugin[0]
>>> sample
{
    'continuous': torch.tensor([1.000, 2.000, ...]),
    'categorical': torch.tensor([True, False, ...])
}
>>> sample['continuous'].dtype
torch.float32
>>> sample['categorical'].dtype
torch.bool
>>> dataplugin.get_features()
{
    'continuous': torch.float32,
    'categorical': torch.bool
}

get_features(filtered: bool = True) → Mapping[str, Sequence[Iterable[str | int]]]

Returns the names of all features as dictionary component_name -> PluginFeatures after filtering (unless specified otherwise).

Parameters:: filtered (bool, optional) – If True, returns the eids of the plugin after filtering. If set to False, will return the eids of the plugin prior to filtering. Default: True
Return type:: PluginFeatures

Example

Consider a plugin for images with two components, where the samples look like this:

>>> sample = plugin[0]
>>> sample
{
    'image': torch.tensor(...),
    'mask': torch.tensor(...)
}
>>> sample['image'].shape, sample['image'].dtype
(256, 256, 3)
>>> sample['mask'].shape, sample['mask'].dtype
(256, 256)

A matching return value of this method could be as follows:

>>> plugin.get_features()
{
    'image': (
        (0, 1, ..., 255),
        (0, 1, ..., 255),
        ('r', 'g', 'b')
    ),
    'mask': (
        (0, 1, ..., 255),
        (0, 1, ..., 255)
    ),
}

get_metadata() → Dict[str, Any]

Returns a dict mapping key -> anything containing information about the data plugin.

At least the following properties are returned:

‘tags’ - The tags assigned to the plugin at initialization
‘eids’ - A list of all eids controlled by the DataPlugin, see get_eids()
‘features’ - A list of the names for all DimensionFeatures of the DataPlugin, see get_features()
‘feature_types’ - A list of the data types for each component of the DataPlugin, see get_feature_types()

Return type:: Dict[str, Any]

get_native_eids(filtered: bool = False) → List[int | str]

Returns the native eids of all (visible) samples in this DataPlugin as list.

Parameters:: filtered (bool) – When True, returns only visible eids that have not been removed post-filtering. If False, will also return invisible eids (eids that are filtered out by the DataPlugin row_filter). Default: False
Return type:: List[Eid]

prepare_data() → None

Multi-node preparation method for the data of this DataPlugin. Will be called in the GeneralDatamodule .prepare_data() call.

See https://pytorch-lightning.readthedocs.io/en/stable/extensions/datamodules.html#prepare-data.

row_filter: PluginRowFilter: The PluginRowFilter instance used to filter columns of the plugin

serialize(path: str | Path)

Serialize and compress DataPlugin.

Parameters:: path (str | pathlib.Path) – Output path.

setup() → None

Single-node preparation method for the data of this DataPlugin. Will be called in the GeneralDatamodule .setup() call.

See https://pytorch-lightning.readthedocs.io/en/stable/extensions/datamodules.html#setup.

tags: List[str]: A list of strings that can be used to tag the DataPlugin to tell it apart from other DataPlugin instances, among other purposes.

trafo: PluginTransform: The PluginTransform to be applied to the data when sampling