nirs4all.data.reduction module

Prediction levels, unit ids and the full ReductionPlan contract.

This module is the roadmap reduction socle (N0 minimal adapter, completed in N4): a single, typed entry point that the existing reduction mechanisms (Predictions.aggregate, Predictions.top(by_repetition=...) and TestAggregation) become adapters to, without changing any legacy output. It also introduces internal prediction-level / prediction-unit metadata and the typed fold_id accessor that later phases use to de-overload fold_id.

The N4 completion adds, on top of the N0 adapter:

• the full role / axis / level / method taxonomy (role=score|persist|fold_ensemble|meta_feature|final_output, axis=unit|fold|model|metric, method=mean|median|vote|weighted_mean|robust|custom);

• weight_source and task_compatibility metadata;

• deterministic reduction_id / ReductionPlan.fingerprint() / ReductionPlan.to_dict() / ReductionPlan.from_dict() manifest support;

• a serialisable fit/replay state contract (FitScope, ReducerState) for any reducer / transform that estimates parameters, with a hard leakage guard refusing state fitted on a validation / test partition.

Only leakage-free, group-stateless methods are executable in this phase (mean / median / vote / robust and weighted_mean when the weights are supplied as inputs). They run on all declared axes by treating group_ids as the caller-resolved bucket key for that axis (unit, fold, model or metric). Fitable reducers carry a contract and a state holder; executing one requires an explicit, non-leaked fitted state. Everything else (custom without a reducer) fails loud. None of this is wired into the persisted prediction format: the additive columns and public kwargs stay inert metadata.

class nirs4all.data.reduction.EvaluationScope(value)

Bases: StrEnum

Unit level at which metrics and model selection are evaluated.

COMBO = 'combo'

OBSERVATION = 'observation'

ROW = 'row'

SAMPLE = 'sample'

class nirs4all.data.reduction.FitScope(value)

Bases: StrEnum

Whether a reducer / transform estimates parameters, and from where.

Variables:

• STATELESS – No parameters are learned; the reduction is computed within each group at evaluation time and is leakage-free by construction.

• FOLD_TRAIN – Parameters are estimated on a CV fold’s train partition and replayed on that fold’s validation rows.

• FULL_TRAIN_REFIT – Parameters are estimated on the full training set during the final refit, producing a state separate from the per-fold states.

FOLD_TRAIN = 'fold_train'

FULL_TRAIN_REFIT = 'full_train_refit'

STATELESS = 'stateless'

exception nirs4all.data.reduction.LeakedReducerStateError(message: str, *, code: str | None = None)

Bases: ReductionError

Raised when a fitable reducer carries a state estimated out-of-fold.

Parameters of QC / outlier / trimmed-robust / imputer / calibration reducers must be fitted on a training partition and replayed on validation / test. A state fitted on a validation / test partition is a leakage path and is refused.

nirs4all.data.reduction.PSEUDO_FOLD_IDS: frozenset[str] = frozenset({'all', 'avg', 'ensemble', 'final', 'final_agg', 'w_avg'})

Legacy fold_id values that are NOT real CV folds.

Real CV fold ids are stringified non-negative integers ("0", "1" …). The values below are pseudo-predictions overloaded onto fold_id today:

• "final" / "final_agg" – final refit model (and its aggregated twin);

• "avg" – RMSECV averaged fold built from concatenated OOF predictions;

• "w_avg" – weighted-average fold;

• "ensemble" / "all" – ensemble / all-fold pseudo entries;

• any value ending in "_agg" – a repetition-aggregated twin.

class nirs4all.data.reduction.PredictionLevel(value)

Bases: StrEnum

Typed level a prediction row describes.

The main user-facing output is always SAMPLE; the other levels are intermediate / diagnostic / meta-feature levels.

Variables:

• ROW – A raw materialised row (e.g. one cartesian combo before reduction).

• OBSERVATION – One (sample, source, rep) measurement.

• SOURCE – One (sample, source) aggregate.

• COMBO – One derived cartesian unit.

• SAMPLE – One physical sample (the deliverable level).

COMBO = 'combo'

OBSERVATION = 'observation'

ROW = 'row'

SAMPLE = 'sample'

SOURCE = 'source'

class nirs4all.data.reduction.PredictionScope(value)

Bases: StrEnum

Role of a prediction entry, derived from the legacy fold_id.

Variables:

• OOF – An out-of-fold cross-validation prediction (or an aggregate of OOF predictions such as the avg / w_avg twins).

• REFIT – A prediction produced by a final refit model (fold_id="final").

• TEST – A held-out test-partition prediction.

OOF = 'oof'

REFIT = 'refit'

TEST = 'test'

class nirs4all.data.reduction.PredictionUnitId(level: PredictionLevel, physical_sample_id: str | None = None, source_id: str | None = None, rep_id: int | None = None, derived_unit_id: str | None = None)

Bases: object

Typed key for a prediction / reduction unit.

A light, hashable identity used to reason about what a prediction row represents, independent of the legacy fold_id overloading.

Variables:

• level (nirs4all.data.reduction.PredictionLevel) – The PredictionLevel of this unit.

• physical_sample_id (str | None) – The owning physical sample, if applicable.

• source_id (str | None) – The source, for observation / source levels.

• rep_id (int | None) – The repetition index, for observation level.

• derived_unit_id (str | None) – The combo id, for combo level.

as_key() → str

Return a canonical string key for this unit.

derived_unit_id: str | None = None

level: PredictionLevel

physical_sample_id: str | None = None

rep_id: int | None = None

source_id: str | None = None

nirs4all.data.reduction.REFIT_FOLD_IDS: frozenset[str] = frozenset({'final', 'final_agg'})

Legacy fold_id values that denote a final refit entry.

class nirs4all.data.reduction.ReducerState(fit_scope: FitScope = FitScope.STATELESS, fit_partition: str | None = None, fold_id: str | None = None, parameters: Mapping[str, ~typing.Any]=<factory>, state_id: str = '')

Bases: object

Serialisable fitted-state holder for a parameter-estimating reducer.

A pure data contract: it records where the parameters were fitted and the learned parameters themselves, so the state can be persisted and replayed. Validation (the leakage guard) is explicit via validate() rather than enforced at construction, so a leaked state can be represented and then rejected by the contract.

Variables:

• fit_scope (nirs4all.data.reduction.FitScope) – How the parameters were estimated (see FitScope).

• fit_partition (str | None) – The partition the parameters were fitted on (e.g. "train"). Validation / test partitions are leakage and rejected.

• fold_id (str | None) – The CV fold the fold-train state belongs to (None for a full-train refit state).

• parameters (collections.abc.Mapping[str, Any]) – The learned parameters (JSON-serialisable).

• state_id (str) – Stable identifier; derived from fingerprint() when empty.

fingerprint() → str

Deterministic SHA-256 over the canonical state descriptor.

fit_partition: str | None = None

fit_scope: FitScope = 'stateless'

fold_id: str | None = None

classmethod from_dict(data: Mapping[str, Any]) → ReducerState

Reconstruct a ReducerState from to_dict() output.

parameters: Mapping[str, Any]

state_id: str = ''

to_dict() → dict[str, Any]

Serialise the state for manifests / lineage.

validate() → None

Refuse leaked or structurally inconsistent fitted state.

Raises:

• LeakedReducerStateError – If the state was fitted on a validation / test partition.

• ReductionError – If the fold/scope combination is inconsistent (a full-train refit state pinned to a real CV fold, or a fold-train state pinned to a pseudo fold).

class nirs4all.data.reduction.ReductionAxis(value)

Bases: StrEnum

Which axis a reduction collapses.

FOLD = 'fold'

METRIC = 'metric'

MODEL = 'model'

UNIT = 'unit'

exception nirs4all.data.reduction.ReductionError(message: str, *, code: str | None = None)

Bases: ValueError

Raised when a reduction contract is misused or not executable.

Subclasses ValueError so existing boundary handlers treat it uniformly. Carries an optional machine-readable code in the REDUCE-Exxx style used across the relational modules.

Variables:

code – Short actionable error code, e.g. "REDUCE-E001".

class nirs4all.data.reduction.ReductionMethod(value)

Bases: StrEnum

Reduction method.

mean / median / vote mirror Predictions.aggregate() exactly. robust is a MAD-outlier-excluded mean (leakage-free, per group). weighted_mean requires the weights to be supplied as inputs. custom is declared-but-not-executable without an explicit reducer.

CUSTOM = 'custom'

MEAN = 'mean'

MEDIAN = 'median'

ROBUST = 'robust'

VOTE = 'vote'

WEIGHTED_MEAN = 'weighted_mean'

class nirs4all.data.reduction.ReductionPlan(role: ReductionRole = ReductionRole.SCORE, axis: ReductionAxis = ReductionAxis.UNIT, method: ReductionMethod = ReductionMethod.MEAN, input_level: PredictionLevel = PredictionLevel.OBSERVATION, output_level: PredictionLevel = PredictionLevel.SAMPLE, exclude_outliers: bool = False, outlier_threshold: float = 0.95, weight_source: str | None = None, task_compatibility: TaskCompatibility | None = None, fit_scope: FitScope = FitScope.STATELESS, fit_partition: str | None = None, fold_id: str | None = None, custom_name: str | None = None, reduction_id: str = '')

Bases: object

Canonical reduction contract that generalises the existing mechanisms.

Wraps the existing reduction semantics so scoring, ranking, refit and reporting can converge on one object. For the safe leakage-free methods, the unit axis remains byte-for-byte identical to the legacy Predictions.aggregate() path. The fold, model and metric axes reuse the same grouped reducer; callers provide the appropriate bucket key through group_ids.

Variables:

• role (nirs4all.data.reduction.ReductionRole) – Why the reduction is applied.

• axis (nirs4all.data.reduction.ReductionAxis) – Which axis it collapses.

• method (nirs4all.data.reduction.ReductionMethod) – mean / median / vote / weighted_mean / robust / custom.

• input_level (nirs4all.data.reduction.PredictionLevel) – The level the reduction consumes.

• output_level (nirs4all.data.reduction.PredictionLevel) – The level it produces.

• exclude_outliers (bool) – Whether to drop per-group outliers before reducing.

• outlier_threshold (float) – Confidence level for outlier detection.

• weight_source (str | None) – Column / source of weights (e.g. "val_score" for the fold-ensemble axis, "sample_influence_weight" for the unit axis).

• task_compatibility (nirs4all.data.reduction.TaskCompatibility | None) – Task family this reduction is meaningful for. Derived from method when left as None.

• fit_scope (nirs4all.data.reduction.FitScope) – Whether this reduction estimates parameters and from where.

• fit_partition (str | None) – The partition a fitable reduction is fitted on.

• fold_id (str | None) – The fold a fold-train reduction belongs to.

• custom_name (str | None) – Identifier of a custom strategy (e.g. "best_fold").

• reduction_id (str) – Stable identifier; derived from fingerprint() when left empty.

as_repetition_aggregation() → tuple[str, bool]

Return the (method, exclude_outliers) for the by_repetition path.

Maps the plan onto the legacy Predictions.top(by_repetition=...) aggregation knobs. Only the leakage-free, group-stateless methods are expressible there. robust becomes an outlier-excluded mean; weighted_mean / custom cannot be expressed on the legacy path and fail loud.

Raises:

ReductionError – If the method cannot drive by_repetition aggregation.

axis: ReductionAxis = 'unit'

custom_name: str | None = None

exclude_outliers: bool = False

fingerprint() → str

Deterministic SHA-256 over the canonical plan descriptor.

The volatile reduction_id is excluded so the digest is a stable function of the plan’s semantics.

fit_partition: str | None = None

fit_scope: FitScope = 'stateless'

fold_id: str | None = None

classmethod from_by_repetition(method: str | None = 'mean', *, exclude_outliers: bool = False) → ReductionPlan

Build a plan equivalent to top(by_repetition=..., method=...).

classmethod from_dict(data: Mapping[str, Any]) → ReductionPlan

Reconstruct a ReductionPlan from to_dict() output.

Tolerates the N0 dict shape (without the N4 fields) by relying on the dataclass defaults; the stored fingerprint is informational and not required for reconstruction.

classmethod from_legacy_aggregate(method: str = 'mean', *, exclude_outliers: bool = False, outlier_threshold: float = 0.95, role: ReductionRole | str = ReductionRole.SCORE) → ReductionPlan

Build a plan equivalent to a legacy Predictions.aggregate call.

classmethod from_test_aggregation(test_aggregation: Any) → ReductionPlan

Build a fold-axis plan from a legacy TestAggregation value.

Captures the fold-ensemble role/method/weight metadata faithfully. mean and weighted are executable grouped reductions when the caller supplies fold predictions keyed by the output unit. weighted keeps method=weighted_mean with weight_source="val_score". best is the non-averaging custom best-fold selection (also keyed by val_score) and remains fail-loud without an explicit reducer.

Parameters:

test_aggregation – A TestAggregation enum or its string value ("mean" / "weighted" / "best").

Returns:

A ReductionPlan on the fold axis.

input_level: PredictionLevel = 'observation'

is_task_compatible(task_type: Any) → bool

Whether this reduction is meaningful for task_type.

Unknown task types are not blocked (returns True); only an explicit regression/classification mismatch is reported.

method: ReductionMethod = 'mean'

outlier_threshold: float = 0.95

output_level: PredictionLevel = 'sample'

prediction_metadata(*, fold_id: Any | None = None, partition: str | None = None, evaluation_scope: str | None = None, is_refit: bool | None = None) → dict[str, Any]

Build additive (non-persisted) prediction metadata for this reduction.

Ties the reduction to the typed prediction_scope_from_legacy() accessor introduced in N0. These keys are internal helpers; they are not forced into the persisted workspace schema.

Parameters:

• fold_id – The legacy fold_id of the reduced entry. Defaults to the plan’s own fold_id.

• partition – The partition of the reduced entry, if known.

• evaluation_scope – The (reserved) evaluation scope label.

• is_refit – Explicit refit marker forwarded to the scope accessor.

Returns:

A dict with prediction_level, prediction_scope, evaluation_scope, reduction_role and reduction_id.

reduce(y_pred: ndarray, group_ids: ndarray, *, y_proba: ndarray | None = None, y_true: ndarray | None = None, weights: ndarray | None = None, task_type: Any | None = None, state: ReducerState | None = None) → dict[str, Any]

Reduce predictions by group.

For the safe leakage-free methods on the unit axis, the output is identical to the legacy Predictions.aggregate() call. The fold, model and metric axes use the same grouped reducer; in those cases group_ids is the already-resolved output bucket key (for example the physical sample id when collapsing fold predictions). Fitable plans (fit_scope != stateless) require an explicit, non-leaked ReducerState; custom fails loud.

Parameters:

• y_pred – Predicted values (n_samples,).

• group_ids – Group identifiers (n_samples,). The meaning depends on axis; for non-unit axes it is the caller-resolved output bucket key.

• y_proba – Optional class probabilities.

• y_true – Optional true values.

• weights – Optional per-row weights (required for weighted_mean).

• task_type – Optional task type; validated against task_compatibility when provided.

• state – Optional fitted state for a fitable plan.

Returns:

The aggregation dict (y_pred, group_ids, group_sizes and optional y_proba / y_true / outliers_excluded).

Raises:

• ReductionError – For an unsupported method or a missing fitted state.

• LeakedReducerStateError – If the supplied state leaks.

reduction_id: str = ''

role: ReductionRole = 'score'

task_compatibility: TaskCompatibility | None = None

to_dict() → dict[str, Any]

Serialise the plan for manifests / lineage.

validate_task(task_type: Any) → None

Raise if this reduction is incompatible with task_type.

weight_source: str | None = None

class nirs4all.data.reduction.ReductionRole(value)

Bases: StrEnum

Why a reduction is performed (one impl, several roles).

FINAL_OUTPUT = 'final_output'

FOLD_ENSEMBLE = 'fold_ensemble'

META_FEATURE = 'meta_feature'

PERSIST = 'persist'

SCORE = 'score'

class nirs4all.data.reduction.TaskCompatibility(value)

Bases: StrEnum

Task families a reduction method is meaningful for.

ANY = 'any'

CLASSIFICATION = 'classification'

REGRESSION = 'regression'

nirs4all.data.reduction.is_real_cv_fold(fold_id: Any) → bool

Return True if fold_id denotes a real cross-validation fold.

Real CV folds are stringified non-negative integers. Pseudo-folds ("final", "avg", "w_avg", "ensemble", "all" and any "*_agg" twin) return False.

Parameters:

fold_id – The legacy fold_id value (str, int or None).

Returns:

Whether the id is a genuine CV fold.

nirs4all.data.reduction.normalize_evaluation_scope(value: Any) → EvaluationScope | None

Return a typed evaluation scope from user-facing strings.

nirs4all.data.reduction.prediction_scope_from_legacy(fold_id: Any, *, partition: str | None = None, is_refit: bool | None = None) → PredictionScope

Derive a typed PredictionScope from legacy fields.

This is an additive accessor: it does not change fold_id values, it only interprets them. Refit entries (fold_id="final" / "final_agg" or an explicit is_refit) map to PredictionScope.REFIT; otherwise a test partition maps to PredictionScope.TEST; everything else (CV fold predictions and their OOF aggregates) maps to PredictionScope.OOF.

Parameters:

• fold_id – The legacy fold_id value.

• partition – The partition of the prediction ("train" / "val" / "test"), if known.

• is_refit – Explicit refit marker overriding the fold_id heuristic.

Returns:

The interpreted scope.