nirs4all.data.relations module

Relational identity model for heterogeneous multi-source repetitions.

This module is the internal foundation (roadmap phases N1/N2) for supporting spectral repetitions that differ per source, e.g. MIR=2, RAMAN=3, NIRS=2 measurements per physical sample. It does not materialise aligned Features; that is deferred to phase N3. The goal here is to resolve sample/source/observation identities once and to validate them, so that split, reshape, scoring and export do not each recompute their own notion of “sample”.

The canonical identity columns mirror the design document:

  • physical_sample_id – the physical sample, the statistical unit for splitting, evaluation and selection. sample_id is only a user-facing YAML alias that maps onto this stable key.

  • internal_sample_id – a deterministic integer assigned per physical sample.

  • source_id – the source / modality / instrument (MIR, RAMAN …).

  • observation_id – one real spectral measurement of a source for a sample.

  • rep_id – the repetition index within a given source.

  • derived_unit_id – a derived unit (e.g. one cartesian combo). None for raw observations.

  • origin_sample_id – provenance for derived / augmented rows; equals physical_sample_id for base rows.

  • unit_level / unit_id – the typed level of a row and its key.

  • row_id – a global row counter.

  • partition – train / test / predict.

  • target_id – the sample-level target value (constant per physical sample).

  • sample_influence_weight – an auditable fit-influence column (default 1.0). The effective weight is derived later (phase N7); N1 only carries a computable column.

  • quality_flag – per-row quality marker.

Nothing in this module mutates the legacy internal index; it is a standalone, in-memory contract. See NormalizedObservationTable and SampleRelationPlan.

class nirs4all.data.relations.ComboRecord(derived_unit_id: str, physical_sample_id: str, origin_sample_id: str, component_observation_ids: tuple[str, ...], rep_ids_by_source: dict[str, int])[source]

Bases: object

Lineage descriptor for a derived cartesian combination.

Variables:

  • derived_unit_id (str) – Stable combo id "<sample>::<rep_a>x<rep_b>...".

  • physical_sample_id (str) – The originating physical sample.

  • origin_sample_id (str) – Provenance (equals physical_sample_id).

  • component_observation_ids (tuple[str, ...]) – Ordered component observation ids.

  • rep_ids_by_source (dict[str, int]) – Mapping source_id -> rep_id for the components.

component_observation_ids: tuple[str, ...]
derived_unit_id: str
origin_sample_id: str
physical_sample_id: str
rep_ids_by_source: dict[str, int]
class nirs4all.data.relations.MissingRepetitionPolicy(value)[source]

Bases: StrEnum

Policy when the observed repetition count differs from expected.

DROP = 'drop'
IMPUTE_DECLARED = 'impute_declared'
PAD = 'pad'
STRICT = 'strict'
TRUNCATE = 'truncate'
class nirs4all.data.relations.MissingSourcePolicy(value)[source]

Bases: StrEnum

Policy when a source is entirely absent for a sample.

DROP_BRANCH = 'drop_branch'
DROP_INCOMPLETE = 'drop_incomplete'
IMPUTE_DECLARED = 'impute_declared'
MASK = 'mask'
PAD = 'pad'
PARTIAL_MODEL = 'partial_model'
STRICT = 'strict'
class nirs4all.data.relations.NormalizedObservationTable(records: Sequence[ObservationRecord])[source]

Bases: object

In-memory canonical table of observations / combos / samples.

The table is the single resolution of sample/source/observation identity. It is intentionally backed by plain Python records (not the polars index) so it can be built and validated without touching legacy storage. Phase N3 will consume it to materialise aligned features.

cardinalities() → dict[tuple[str, str], int][source]

Observation counts keyed by (physical_sample_id, source_id).

enumerate_combos(physical_sample_id: str) → list[ComboRecord][source]

Enumerate the full cartesian combos for a physical sample.

Each combo is the deterministic cross-product of one observation per source (in source_ids order), carrying full lineage. This is the combo-lineage helper referenced by N1; cartesian materialisation and cost caps belong to later phases.

Parameters:

physical_sample_id – The sample to enumerate.

Returns:

A deterministic list of ComboRecord. Empty if any source is missing for the sample.

fingerprint() → str[source]

Deterministic SHA-256 over the canonical, order-independent table.

Records are canonicalised and sorted so that source order, shuffled repetitions and insertion order do not change the digest.

internal_id_map() → dict[str, int][source]

Mapping physical_sample_id -> internal_sample_id.

property physical_sample_ids: list[str]

Sorted unique physical sample ids.

records: list[ObservationRecord]
records_for_sample(physical_sample_id: str) → list[ObservationRecord][source]

All records belonging to a physical sample (in row order).

property source_ids: list[str]

Sorted unique source ids.

targets_by_sample() → dict[str, Any][source]

Sample-level target value per physical sample.

to_columns() → dict[str, list[Any]][source]

Columnar view keyed by RELATION_TABLE_COLUMNS.

validate(spec: RepetitionSpec | None = None) → None[source]

Validate the structural invariants of the table.

Checks:

  • uniqueness of (physical_sample_id, source_id, rep_id);

  • constant target per physical sample;

  • non-contradictory sample-level metadata;

  • declared cardinalities when spec provides expected and strict_cardinality is set.

Parameters:

spec – Optional RepetitionSpec providing expected cardinalities and the strictness flag.

Raises:

RelationValidationError – On any violated invariant.

with_equal_sample_influence() → NormalizedObservationTable[source]

Return a copy whose weights equalise per-sample influence.

Each physical sample receives total influence 1.0, split uniformly across its observation rows (1 / n_obs(sample)). This is an auditable derivation; the default table keeps 1.0 per row (phase N7 owns the effective policy).

class nirs4all.data.relations.ObservationRecord(physical_sample_id: str, internal_sample_id: int, source_id: str, rep_id: int, observation_id: str, row_id: int, partition: Partition = Partition.TRAIN, target_id: Any = None, origin_sample_id: str | None = None, unit_level: UnitLevel = UnitLevel.OBSERVATION, unit_id: str = '', derived_unit_id: str | None = None, sample_influence_weight: float = 1.0, quality_flag: QualityFlag = QualityFlag.OK, component_observation_ids: tuple[str, ...]=(), metadata: Mapping[str, ~typing.Any]=<factory>, source_row: int | None = None)[source]

Bases: object

One row of the NormalizedObservationTable.

Carries the full canonical identity schema. For raw observations, unit_level is UnitLevel.OBSERVATION and derived_unit_id / component_observation_ids are empty. Derived combos set those fields and use UnitLevel.COMBO.

component_observation_ids: tuple[str, ...] = ()
derived_unit_id: str | None = None
internal_sample_id: int
metadata: Mapping[str, Any]
observation_id: str
origin_sample_id: str | None = None
partition: Partition = 'train'
physical_sample_id: str
quality_flag: QualityFlag = 'ok'
rep_id: int
row_id: int
sample_influence_weight: float = 1.0
source_id: str
source_row: int | None = None

Original row index of this observation within its source’s input order. Internal mapping field used by phase-N3 staging to index back into the per-source feature matrices. It is intentionally not part of the canonical RELATION_TABLE_COLUMNS schema and does not affect the fingerprint (source/row order must not change identity).

target_id: Any = None
to_dict() → dict[str, Any][source]

Return the canonical column mapping for this record.

unit_id: str = ''
unit_level: UnitLevel = 'observation'
class nirs4all.data.relations.Partition(value)[source]

Bases: StrEnum

Partition assignment of a relational row.

PREDICT = 'predict'
TEST = 'test'
TRAIN = 'train'
class nirs4all.data.relations.QualityFlag(value)[source]

Bases: StrEnum

Per-row quality marker.

IMPUTED = 'imputed'
MISSING = 'missing'
OK = 'ok'
OUTLIER = 'outlier'
PADDED = 'padded'
nirs4all.data.relations.RELATION_TABLE_COLUMNS: tuple[str, ...] = ('row_id', 'physical_sample_id', 'internal_sample_id', 'source_id', 'observation_id', 'rep_id', 'unit_level', 'unit_id', 'derived_unit_id', 'origin_sample_id', 'partition', 'target_id', 'sample_influence_weight', 'quality_flag', 'component_observation_ids')

Ordered canonical schema of the normalised observation table.

nirs4all.data.relations.REPETITION_MECHANISMS: tuple[str, ...] = ('repetition', 'rep_to_sources', 'rep_to_pp', 'rep_fusion')

Repetition mechanisms that must not compose implicitly (exclusivity matrix).

class nirs4all.data.relations.RelationConfig(enabled: bool = False, spec: RepetitionSpec | None = None, link_by: str | None = None)[source]

Bases: object

Parsed relational configuration extracted from a dataset config dict.

This is the bridge between the (additive, inert) schema fields carried in the legacy dataset dict – experimental_relation_pipeline, repetition_spec, relations and per-source link_by – and the executable RepetitionSpec / join contract.

Variables:

  • enabled (bool) – Whether the experimental relation pipeline is explicitly opted in (experimental_relation_pipeline: true).

  • spec (nirs4all.data.relations.RepetitionSpec | None) – The parsed RepetitionSpec, if a repetition_spec / relations block was provided.

  • link_by (str | None) – The effective join key, resolved from the spec, the relations block, or a per-source link_by.

enabled: bool = False
property is_relational: bool

Whether any relational contract is declared (flag, spec or link_by).

spec: RepetitionSpec | None = None
exception nirs4all.data.relations.RelationValidationError(message: str, *, code: str | None = None)[source]

Bases: ValueError

Raised when a relational configuration or table is incoherent.

Subclasses ValueError so that boundary handlers that already catch ValueError (loaders, config parsing) treat it uniformly. Carries an optional machine-readable code mirroring the existing REP-Exxx style.

Variables:

code – Short actionable error code, e.g. "REL-E001".

class nirs4all.data.relations.RepOrder(value)[source]

Bases: StrEnum

Whether repetitions within a source are exchangeable or ordered.

Variables:

  • EXCHANGEABLE – Repetitions carry no stable order (the default for NIRS replicate measurements); rep_id is assigned by sorted key.

  • ORDERED – Repetition order is semantically meaningful and preserved.

EXCHANGEABLE = 'exchangeable'
ORDERED = 'ordered'
class nirs4all.data.relations.RepetitionSpec(sample_id: str, target_level: str = 'physical_sample', sources: dict[str, ~nirs4all.data.relations.SourceRepetitionSpec]=<factory>, missing_repetition_policy: MissingRepetitionPolicy = MissingRepetitionPolicy.STRICT, missing_source_policy: MissingSourcePolicy = MissingSourcePolicy.STRICT, rep_order: RepOrder = RepOrder.EXCHANGEABLE, strict_cardinality: bool = False, link_by: str | None = None)[source]

Bases: object

Source-aware repetition specification (roadmap N2).

Describes how repetitions are organised per source rather than along a single global row axis. This is the explicit alternative to the legacy uniform repetition= shortcut, and the only contract under which heterogeneous sources (MIR=2, RAMAN=3 …) may be joined.

Variables:

  • sample_id (str) – Name of the user-facing column identifying the physical sample. Mapped onto the canonical physical_sample_id.

  • target_level (str) – Granularity at which targets live. Only "physical_sample" is supported (targets are sample-level).

  • sources (dict[str, nirs4all.data.relations.SourceRepetitionSpec]) – Mapping source_id -> SourceRepetitionSpec.

  • missing_repetition_policy (nirs4all.data.relations.MissingRepetitionPolicy) – Policy for repetition-count mismatches.

  • missing_source_policy (nirs4all.data.relations.MissingSourcePolicy) – Policy for entirely missing sources.

  • rep_order (nirs4all.data.relations.RepOrder) – Whether repetitions are exchangeable or ordered.

  • strict_cardinality (bool) – If True, every declared expected count must match exactly; otherwise mismatches are tolerated (and recorded).

  • link_by (str | None) – Explicit join key across sources. Defaults to sample_id. When set, sources are joined by key, never positionally.

classmethod from_config(cfg: Mapping[str, Any]) → RepetitionSpec[source]

Build a RepetitionSpec from a config mapping.

Accepts the experimental YAML shape:

repetition_spec:
  sample_id: sample_id
  link_by: sample_id
  rep_order: exchangeable
  strict_cardinality: true
  missing_repetition_policy: strict
  missing_source_policy: strict
  sources:
    MIR:   {rep_col: rep, expected: 2}
    RAMAN: {expected: 3}
Parameters:

cfg – The repetition_spec mapping.

Returns:

A validated RepetitionSpec.

Raises:

RelationValidationError – If the mapping is malformed.

property join_key: str

The effective join key (link_by if set, else sample_id).

missing_repetition_policy: MissingRepetitionPolicy = 'strict'
missing_source_policy: MissingSourcePolicy = 'strict'
rep_order: RepOrder = 'exchangeable'
sample_id: str
source_spec(source_id: str) → SourceRepetitionSpec[source]

Return the declaration for source_id (empty default if absent).

sources: dict[str, SourceRepetitionSpec]
strict_cardinality: bool = False
target_level: str = 'physical_sample'
to_dict() → dict[str, Any][source]

Serialise to a plain dict for manifests / fingerprints.

class nirs4all.data.relations.SampleRelationPlan(table: NormalizedObservationTable, spec: RepetitionSpec | None = None, rep_order: RepOrder = RepOrder.EXCHANGEABLE)[source]

Bases: object

Resolved relational plan binding a spec to a normalised table.

Holds the single resolution of sample identity that downstream phases (split, scoring, refit, export) must consume instead of recomputing.

Variables:
declared_cardinalities() → dict[str, int | None][source]

Declared expected count per source (None if not declared).

fingerprint() → str[source]

Deterministic digest combining the spec and the table.

classmethod from_sources(spec: RepetitionSpec, sources: Sequence[SourceObservations], *, partition: Partition | str = Partition.TRAIN) → SampleRelationPlan[source]

Build and validate a plan from per-source observations.

property physical_to_internal: dict[str, int]

Stable physical_sample_id -> internal_sample_id mapping.

rep_order: RepOrder = 'exchangeable'
spec: RepetitionSpec | None = None
table: NormalizedObservationTable
class nirs4all.data.relations.SourceObservations(source_id: str, sample_ids: Sequence[Any] | None, rep_ids: Sequence[Any] | None = None, targets: Sequence[Any] | None = None, metadata: Mapping[str, Sequence[Any]] | None = None)[source]

Bases: object

Per-source observation columns fed to build_relation_table().

This is the decoupled, IO-free input to the join contract: callers extract the join-key values (one per observation row), optional explicit repetition indices, optional targets and optional sample-level metadata columns.

Variables:
classmethod from_frame(source_id: str, frame: Any, *, sample_col: str | None, rep_col: str | None = None, target_col: str | None = None, metadata_cols: Sequence[str] | None = None) → SourceObservations[source]

Build SourceObservations from a column container.

Works with anything exposing frame[column] returning a sequence (pandas DataFrame, dict of lists, polars frame …). This is the seam a future loader (phase N3) uses to assemble per-source observations from already-read tables.

Parameters:

  • source_id – The source / modality name.

  • frame – A column container (frame[col] -> values).

  • sample_col – Column holding the physical sample id, or None to declare that the source exposes no key (positional join refused).

  • rep_col – Optional column holding the repetition index.

  • target_col – Optional column holding the per-row target.

  • metadata_cols – Optional sample-level metadata columns to carry.

Returns:

A SourceObservations instance.

metadata: Mapping[str, Sequence[Any]] | None = None
rep_ids: Sequence[Any] | None = None
sample_ids: Sequence[Any] | None
source_id: str
targets: Sequence[Any] | None = None
class nirs4all.data.relations.SourceRepetitionSpec(rep_col: str | None = None, expected: int | None = None)[source]

Bases: object

Per-source repetition declaration.

Variables:

  • rep_col (str | None) – Optional metadata column holding the repetition index for this source. None means repetition indices are assigned positionally within each (sample, source) group.

  • expected (int | None) – Expected number of repetitions for this source, or None to infer from the data. Validated when strict_cardinality is set.

expected: int | None = None
rep_col: str | None = None
to_dict() → dict[str, Any][source]

Serialise to a plain dict for manifests.

class nirs4all.data.relations.UnitLevel(value)[source]

Bases: StrEnum

Typed level of a relational unit.

Variables:

  • PHYSICAL_SAMPLE – One physical sample (the statistical unit).

  • SOURCE_SAMPLE – One (physical_sample, source) pair.

  • OBSERVATION – One (physical_sample, source, rep) measurement.

  • COMBO – A derived cartesian unit drawn from several observations.

  • ROW – A generic materialised row (representation output).

COMBO = 'combo'
OBSERVATION = 'observation'
PHYSICAL_SAMPLE = 'physical_sample'
ROW = 'row'
SOURCE_SAMPLE = 'source_sample'

Validate that declared link_by keys make positional alignment safe.

This is the positional safety contract, distinct from the real relational join (build_relation_table()). When several sources are about to be concatenated by row position but a link_by key is declared, the keys must establish an unambiguous 1:1 correspondence. Concretely:

  • every source must expose the key (no positional-only source);

  • keys must be unique within each source (when require_unique); otherwise the positional correspondence is ambiguous (heterogeneous repetitions);

  • every source must cover exactly the same set of keys; otherwise the sources are divergent and cannot be aligned by position.

Non-unique keys (repetitions) and divergent / missing keys are not silently tolerated here – they require the real relational join, not positional concatenation.

Parameters:

  • source_keys – Mapping source_id -> per-row key values. A None value marks a source that exposes no key column at all.

  • link_by – The declared join key (used only to tailor messages).

  • require_unique – Whether keys must be unique within each source.

Raises:

RelationValidationError – If positional alignment is unsafe.

Ensure a declared link_by is an executable join in relation mode.

A link_by that is parsed but silently ignored is a correctness trap. In the relational profile a declared key must resolve to a real column.

Parameters:

  • link_by – The declared join key (or None).

  • relation_mode – Whether the experimental relational pipeline is active.

  • available_columns – Columns available to satisfy the join, if known.

Raises:

RelationValidationError – If link_by cannot be executed as a join.

nirs4all.data.relations.audit_source_lengths(lengths: Sequence[int], *, relation_mode: bool = False, link_by: str | None = None) → None[source]

Audit multi-source row counts for positional alignment coherence.

In the legacy positional loader, sources are concatenated by row position, which is only valid when every source has the same number of rows. Unequal counts mean the sources are heterogeneous and cannot be aligned positionally; this previously produced a silently incoherent dataset.

Parameters:

  • lengths – Row count per source.

  • relation_mode – Whether an explicit relational pipeline is active. When True the heterogeneous case is handled by the relation table, so this audit defers (it does not bless positional alignment).

  • link_by – The declared join key, if any (used to tailor the message).

Raises:

RelationValidationError – If sources have unequal lengths and no relation plan can resolve them.

nirs4all.data.relations.build_relation_table(spec: RepetitionSpec, sources: Sequence[SourceObservations], *, partition: Partition | str = Partition.TRAIN) → NormalizedObservationTable[source]

Build a NormalizedObservationTable by joining sources on a key.

This is the real link_by join contract (roadmap N2): sources are joined by their key values, never by row position. Source order and per-row order are irrelevant – only the keys matter – so shuffled sources are supported.

Parameters:

  • spec – The source-aware repetition specification.

  • sources – One SourceObservations per source.

  • partition – Partition label applied to every produced row.

Returns:

A validated NormalizedObservationTable.

Raises:

RelationValidationError – If the join is not executable as a true join (missing key column), if keys are ambiguous, or if any structural invariant fails.

nirs4all.data.relations.check_repetition_exclusivity(steps: Sequence[Any], *, has_global_repetition: bool = False) → None[source]

Enforce the repetition exclusivity matrix.

Legacy uniform repetition=, rep_to_sources, rep_to_pp and the relational rep_fusion are mutually exclusive paths; combining them is rejected so they never compose implicitly.

Parameters:

  • steps – The pipeline step list.

  • has_global_repetition – Whether a global repetition= is configured.

Raises:

RelationValidationError – If more than one mechanism is active.

nirs4all.data.relations.detect_repetition_mechanisms(steps: Sequence[Any], *, has_global_repetition: bool = False) → set[str][source]

Detect which repetition mechanisms a pipeline activates.

Parameters:

  • steps – The pipeline step list (dicts / objects). Keyword steps such as {"rep_to_sources": ...} are recognised by their keys.

  • has_global_repetition – Whether the dataset declares a global, uniform repetition= column.

Returns:

The set of active mechanism names (subset of REPETITION_MECHANISMS).

nirs4all.data.relations.make_observation_id(source_id: str, physical_sample_id: str, rep_id: int) → str[source]

Return the canonical observation id "<source>:<sample>:<rep>".

nirs4all.data.relations.parse_relation_config(config: Mapping[str, Any]) → RelationConfig | None[source]

Parse the experimental relational fields out of a dataset config dict.

Reads the additive fields emitted by DatasetConfigSchema() (and any equivalent hand-written dict). Returns None when nothing relational is declared so legacy callers can cheaply skip the relational path.

Parameters:

config – The (legacy-shaped) dataset configuration dict.

Returns:

A RelationConfig if any relational field is present, else None.

Raises:

RelationValidationError – If a declared repetition_spec / relations block is malformed.