Cross-Validation

Cross-validation (CV) is the mechanism nirs4all uses to estimate how well a pipeline will perform on unseen data. Instead of training once and hoping for the best, the library trains the same pipeline multiple times on different subsets of the data and measures performance on the held-out portions.

Why Cross-Validate?

A model can memorise its training data. If you evaluate it on the same data it was trained on, the score will be misleadingly good. Cross-validation solves this by ensuring that every sample is predicted by a model that never saw it during training.

Fold 0:  [====TRAIN====] [=VAL=]         -> predictions for VAL samples
Fold 1:  [===TRAIN===]   [=VAL=] [TRAIN] -> predictions for VAL samples
Fold 2:  [=VAL=]         [====TRAIN====] -> predictions for VAL samples
                                            -------------------------
                                            All samples predicted once

After all folds are done, every training sample has been predicted exactly once – by a model that was not trained on it. These are called out-of-fold (OOF) predictions.

How It Works in nirs4all

Cross-validation is driven by a splitter step in the pipeline. The splitter defines how many folds to create and how to assign samples.

from sklearn.model_selection import ShuffleSplit

pipeline = [
    SNV(),
    ShuffleSplit(n_splits=5, test_size=0.2),
    {"model": PLSRegression(n_components=10)},
]

When this pipeline is executed:

  1. Pre-splitter steps (SNV) are applied to the entire training partition.

  2. The splitter writes 5 fold definitions into the dataset.

  3. For each fold (0 through 4):

    • Post-splitter transforms are fitted on the fold’s train subset and applied to train, val, and test.

    • The model is fitted on the fold’s train subset.

    • Predictions are made on val, test, and optionally train.

  4. OOF predictions from all 5 folds are pooled.

  5. Metrics are computed on the pooled OOF predictions.

Out-of-Fold (OOF) Predictions

The pooled OOF predictions form the basis for ranking pipeline variants. Each sample is predicted by the model that did not see it, so the resulting score is an honest estimate of generalisation performance.

Note

OOF is short for “out-of-fold”. It means the sample was in the validation portion of the fold, not in the training portion. OOF predictions are sometimes called “cross-validated predictions” in the literature.

OOF predictions are also the building block for stacking: when you use {"merge": "predictions"}, the OOF predictions from each branch become features for a meta-model. Because they are out-of-fold, there is no data leakage. See Branching and Merging for details.

The Test Partition

The test partition is entirely separate from cross-validation. It is never used to select models or tune hyperparameters. After each fold, the fold’s model predicts on the test set, and after refit (see below), the final model also predicts on the test set.

This gives you three leak-free test estimates:

Score

How it is computed

Ens_Test

Average of K fold-model predictions on test

W_Ens_Test

Quality-weighted average of fold predictions

RMSEP

Single refit model’s prediction on test (see below)

Scoring and Metrics

The default primary metric is MSE (mean squared error), but results are reported in several forms to match chemometrics conventions:

Metric

Full name

Meaning

RMSECV

Root Mean Square Error of Cross-Validation

RMSE of pooled OOF predictions

RMSEP

Root Mean Square Error of Prediction

RMSE of refit model on test set

RMSEC

Root Mean Square Error of Calibration

RMSE of model on its own training data

R2

Coefficient of determination

How much variance is explained

For classification tasks, the primary metric is balanced accuracy, with additional metrics like F1, precision, recall, and ROC-AUC.

Tip

RMSECV is computed from the pooled OOF vector, not as the mean of per-fold RMSEs. These two quantities are close but not identical because the square root is a non-linear function.

Refit

After cross-validation has ranked all pipeline variants, the best variant is refitted on the entire training partition (all folds combined). This produces a single model that has seen all available training data – the model you would deploy to production.

Phase 1 -- Cross-Validation
  For each variant:
    Train on K folds, collect OOF predictions
    Rank by RMSECV

Phase 2 -- Refit
  Take the top variant
  Retrain on ALL training samples (no folds)
  Evaluate on the test set -> RMSEP

The refit model typically performs better than any individual fold model because it has more training data. The comparison between RMSECV and RMSEP tells you whether the model benefits from the additional data.

Controlling refit

# Default: refit top 1
result = nirs4all.run(pipeline=pipeline, dataset=dataset)

# Disable refit (faster exploration)
result = nirs4all.run(pipeline=pipeline, dataset=dataset, refit=False)

# Refit top 3 variants
result = nirs4all.run(
    pipeline=pipeline, dataset=dataset,
    refit={"top_k": 3, "ranking": "rmsecv"},
)

See Understanding Scores, Evaluation, and Refit for multi-criteria refit, refit strategies for stacking pipelines, and detailed score interpretation.

result.best vs result.final

The RunResult object distinguishes between CV results and refit results:

Property

Scope

result.best_score

Primary test_score of the selected entry

result.cv_best_score

Validation score of the best CV entry

result.cv_best

Best CV entry (all fold details)

result.final

Refit entry (fold_id="final")

result.final_score

Refit model’s test score (RMSEP)

result.top(n)

Top N variants, ranked by CV or final score

result.best prefers the selected refit entry when available and otherwise returns the best CV entry. Its best_score reads that entry’s test_score; use result.cv_best_score for the CV validation score instead. The native pooled OOF report averages held-out predictions per physical sample before scoring unique samples. With overlapping validation folds, the explicit legacy lane’s concatenation weights repeated samples repeatedly, so its pooled CV score can differ despite identical per-fold predictions.

result.final identifies the refit entry. Read its scored partition and metadata to distinguish held-out evaluation from training resubstitution; its presence alone does not establish evaluation on fresh data.

Summary

Dataset
  |
  +-- train partition --+-- Fold 0 train --> fit model --> predict val (OOF)
  |                     +-- Fold 1 train --> fit model --> predict val (OOF)
  |                     +-- ...
  |                     +-- Pool OOF --> RMSECV --> rank variants
  |                     |
  |                     +-- Refit best on all train --> final model
  |
  +-- test partition  -----> evaluate final model --> RMSEP

Next Steps