nirs4all.data.loaders.loader module

nirs4all.data.loaders.loader.load_XY(x_path: str, x_filter: Any, x_params: dict[str, Any], y_path: str | None, y_filter: Any, y_params: dict[str, Any], m_path: str | None = None, m_filter: Any = None, m_params: dict[str, Any] | None = None, *, row_info: dict[str, Any] | None = None) → tuple[ndarray, ndarray, Any, list[str], list[str], str, SignalType | None][source]

Load X, Y, and metadata from single paths. For multi-source, this will be called multiple times.

Parameters: - x_path (str): Single path to X data file. - x_filter: Filter to apply to X data (not implemented yet). - x_params (dict): Parameters for loading X data, including:

  • header_unit: Unit for headers (“cm-1”, “nm”, “none”, “text”, “index”)

  • signal_type: Signal type (“absorbance”, “reflectance”, “reflectance%”, etc.)

  • delimiter, decimal_separator, has_header, na_policy, etc.

  • y_path (str): Path to the Y data file (can be None).

  • y_filter: Filter to apply to Y data (or indices if y_path is None).

  • y_params (dict): Parameters for loading Y data.

  • m_path (str): Path to metadata file (can be None).

  • m_filter: Filter to apply to metadata (not implemented yet).

  • m_params (dict): Parameters for loading metadata.

  • row_info (dict): Optional output for original row count and retained indices.

Returns: - tuple: (x, y, m, x_headers, m_headers, x_header_unit, x_signal_type) where:

  • x, y, m are numpy arrays/DataFrames

  • x_headers, m_headers are lists of column names

  • x_header_unit is the unit string for X headers (“cm-1”, “nm”, “none”, “text”, “index”)

  • x_signal_type is the signal type (SignalType enum or None for auto-detect)

Raises: - ValueError: If data is invalid or if there are inconsistencies.

nirs4all.data.loaders.loader.materialize_relation_table(config: dict[str, Any], source_frames: dict[str, Any], *, partition: str = 'train', target_col: str | None = None) → NormalizedObservationTable | None[source]

Materialise a NormalizedObservationTable from per-source frames.

This is the loader-level seam for the experimental relation pipeline (roadmap N2): given the per-source column frames already read from disk (or any frame[column] container – pandas DataFrame, dict of lists …) and a declared repetition_spec / link_by, it joins the sources by key into a validated relation table. Source row order is irrelevant – only the link_by keys matter – so shuffled sources are supported and heterogeneous sources are never aligned by accident.

This function is intentionally additive: it does not change the legacy handle_data() return contract. Callers in the relation profile invoke it explicitly; legacy positional loads never reach it.

Parameters:

  • config – The dataset configuration dict (carries the relational fields).

  • source_frames – Mapping source_id -> frame where frame[column] yields a per-row sequence. Each source must expose the join-key column declared by the spec.

  • partition – Partition label applied to every produced row.

  • target_col – Optional column holding the per-row (sample-level) target, shared across sources via the join.

Returns:

A validated NormalizedObservationTable, or None when no executable repetition_spec is declared.

Raises:

RelationValidationError – If the declared join is not executable (missing key column, divergent / non-unique ids, missing source …).