Understanding Scores, Evaluation, and Refit

This page explains how nirs4all evaluates pipeline performance across cross-validation, ensemble scoring, and refit. It clarifies the meaning of every score you see in reports, how refit works, and how to control it.

The Two Phases of a Run

Every nirs4all.run() call proceeds in two phases:

Cross-Validation (CV) phase – Each pipeline variant is evaluated using K-fold cross-validation. Out-of-fold (OOF) predictions are collected and scored. This phase ranks variants and selects winner(s).
Refit phase – The winning variant(s) are retrained on the full training set (all folds combined) to produce a single production model. This model is evaluated on the held-out test set.

Phase 1: Cross-Validation
  For each variant:
    For each fold k (k = 0..K-1):
      Train on fold k's training samples
      Predict on val, test, train partitions
      Store per-fold scores

    After all folds:
      Pool OOF predictions -> RMSECV / CV balanced accuracy
      Average fold test predictions -> Ens_Test
      Weighted average -> W_Ens_Test

Phase 2: Refit
  Select top variant(s) by ranking criterion
  For each selected variant:
    Retrain model on ALL training samples
    Predict on test set -> RMSEP / Test balanced accuracy
    Store as fold_id="final"

Score Definitions (Regression)

RMSECV – Root Mean Square Error of Cross-Validation

The standard chemometrics CV score. Computed from pooled out-of-fold predictions: each fold model predicts only its held-out validation samples, and all predictions are concatenated into a single vector before computing RMSE.

PRESS = sum_i (y_i - y_hat_i_OOF)^2
RMSECV = sqrt(PRESS / N)

This is the default selection criterion for refit.

Important: RMSECV is NOT the mean of per-fold RMSEs. These are different aggregations (Jensen’s inequality applies to the square root). They are generally close but not identical.

MF_Val – Mean Fold Validation Score

Arithmetic mean of per-fold validation RMSEs:

MF_Val = (1/K) * sum_k RMSE_k^val

Available as an alternative refit selection criterion (ranking: "mean_val").

Ens_Test – Ensemble Test Score

Test RMSE from the simple average of K fold models’ predictions on the test set:

y_hat_ens = (1/K) * sum_k y_hat_k^test
Ens_Test = sqrt( (1/M) * sum_j (y_j - y_hat_ens_j)^2 )

This is NOT RMSEP. It shows how well the K fold models perform as an ensemble without retraining.

W_Ens_Test – Weighted Ensemble Test Score

Same as Ens_Test but using quality-weighted averaging of fold predictions (better-performing folds contribute more).

RMSEP – Root Mean Square Error of Prediction

RMSE on the held-out test set, evaluated after refit on the full training data:

RMSEP = sqrt( (1/M) * sum_j (y_j - y_hat_j)^2 )

This is the score of the final production model. Only available after refit.

RMSEC – Root Mean Square Error of Calibration

RMSE on the training set. Available on individual fold entries and on the final refit entry.

Summary: Three Leak-Free Test Predictions

Score	Source	When available
Ens_Test	Average of K fold models on test	After CV
W_Ens_Test	Weighted average of K fold models on test	After CV
RMSEP	Single refit model on test	After refit

Summary: Two Selection Scores (CV Phase)

Score	Definition	Use
RMSECV	Pooled OOF RMSE	Default refit selection
MF_Val	Mean of per-fold validation RMSEs	Alternative selection

Score Definitions (Classification)

For classification tasks, the primary metric is balanced accuracy (higher_is_better=True).

Score	Definition
CV balanced accuracy	Pooled OOF balanced accuracy: concatenate all OOF predictions and compute `balanced_accuracy_score(y_true, y_pred)`
Test balanced accuracy	Balanced accuracy on held-out test set after refit

Supported classification metrics: balanced_accuracy, accuracy, f1_score, precision, recall, roc_auc, cohen_kappa, matthews_corrcoef, jaccard.

Naming Conventions

Scores appear in reports using two naming modes:

NIRS Mode (default)

Standard chemometrics terminology.

Internal key	Display name	Definition
`cv_score`	RMSECV	Pooled OOF RMSE
`test_score`	RMSEP	Test RMSE after refit
`ens_test`	Ens_Test	Ensemble test from avg of K fold models
`w_ens_test`	W_Ens_Test	Weighted ensemble test
`mean_fold_cv`	MF_Val	Mean of per-fold validation RMSEs

ML Mode

Machine learning terminology.

Internal key	Display name
`cv_score`	CV_Score
`test_score`	Test_Score
`ens_test`	Ens_Test_Score
`w_ens_test`	W_Ens_Test_Score
`mean_fold_cv`	MF_CV

Set via report_naming:

result = nirs4all.run(
    pipeline=pipeline,
    dataset=dataset,
    report_naming="nirs",  # default; or "ml", "auto"
)

Refit: How It Works

What Refit Does

After cross-validation identifies the best pipeline variant(s), refit retrains the winning model on the entire training set (all folds combined). This produces a single model that leverages all available training data – the model you would deploy to production.

The refit model is then evaluated on the held-out test set to produce RMSEP (or test balanced accuracy for classification).

Why Refit Matters

During cross-validation, each fold model only sees a fraction of the training data (e.g., 80% in 5-fold CV). The refit model sees 100% of the training data, which typically improves generalization. The comparison between RMSECV (CV performance) and RMSEP (refit test performance) tells you whether the model benefits from more data.

Refit Strategies

nirs4all automatically selects the appropriate refit strategy based on pipeline topology:

Pipeline type	Strategy	Description
Simple pipeline	`simple_refit`	Retrain the single best variant on full training data
Stacking / merge predictions	`stacking_refit`	Retrain base models, regenerate OOF-style predictions, retrain meta-model
Competing branches	`competing_branches_refit`	Retrain winning branch model on full data
Separation branches	`separation_refit`	Retrain per-group models on full per-group data

Controlling Refit

The refit parameter in nirs4all.run() controls refit behavior:

# Default: refit top 1 by RMSECV
result = nirs4all.run(pipeline=pipeline, dataset=dataset)

# Disable refit
result = nirs4all.run(pipeline=pipeline, dataset=dataset, refit=False)

# Refit top 3 by RMSECV
result = nirs4all.run(
    pipeline=pipeline,
    dataset=dataset,
    refit={"top_k": 3, "ranking": "rmsecv"},
)

# Refit top 3 by mean fold validation score
result = nirs4all.run(
    pipeline=pipeline,
    dataset=dataset,
    refit={"top_k": 3, "ranking": "mean_val"},
)

# Multi-criteria refit: top 3 by RMSECV + top 1 by mean_val
result = nirs4all.run(
    pipeline=pipeline,
    dataset=dataset,
    refit=[
        {"top_k": 3, "ranking": "rmsecv"},
        {"top_k": 1, "ranking": "mean_val"},
    ],
)

Refit Parameters

Refit-specific training parameters can override CV training parameters:

pipeline = [
    {"model": MyModel(), "train_params": {"epochs": 50, "lr": 0.01},
     "refit_params": {"epochs": 100}},
]

refit_params merges on top of train_params: unspecified keys inherit from train_params, specified keys override.

warm_start is not currently supported for CV-to-refit weight transfer. The legacy run path accepts it but cold-fits the final model because its refit context has no CV artifact provider. The DAG-ML path explicitly refuses the request rather than reporting a warm start that did not occur.

Understanding the Report Tables

A typical run produces two summary tables:

Final Scores Table

Shows refit results for each selected model (fold_id="final"):

Column	Source
RMSEP	Refit model’s test score
RMSECV	Pooled OOF RMSE from CV phase
MF_Val	Mean of per-fold validation RMSEs
Ens_Test	Average of K fold models on test
W_Ens_Test	Weighted average of K fold models on test

Sorted by RMSEP (ascending for regression, descending for classification).

Top CV Chains Table

Shows the best cross-validation pipeline variants (fold_id="avg"):

Column	Source
RMSECV	Pooled OOF RMSE
MF_Val	Mean of per-fold validation scores
Ens_Test	Ensemble test score
W_Ens_Test	Weighted ensemble test score
f0, f1, …	Individual fold validation scores

Score Storage

fold_id Values

`fold_id`	Meaning
`"0"`, `"1"`, …	Individual CV fold results
`"avg"`	OOF-aggregated entry. `val_score` = RMSECV, `test_score` = Ens_Test
`"w_avg"`	Weighted ensemble. `test_score` = W_Ens_Test
`"final"`	Refit entry. `val_score=None`, `test_score` = RMSEP

Accessing Scores Programmatically

result = nirs4all.run(pipeline=pipeline, dataset=dataset)

# Best CV score (RMSECV)
result.best_score

# Best refit test score (RMSEP)
result.best_rmse  # alias for best test score

# Top N results (default: final/refit scope)
result.top(5)

# Top N results (CV-only scope)
result.top(5, score_scope="folds")

# CV-only best
result.cv_best

# Final refit result
result.final
result.final_score

The score_scope parameter controls which entries are considered:

Scope	Behavior
`"cv"`	Only cross-validation entries
`"final"` / `"refit"`	Only refit entries (`fold_id="final"`) (default)
`"mix"`	Finals first, then CV
`"flat"`	All entries treated equally

Practical Guidelines

When to Disable Refit

Quick exploration: When iterating on preprocessing choices and you only care about CV performance, refit=False saves time.
No test set: If your dataset has no held-out test partition, refit cannot produce RMSEP. The refit model is still trained but only RMSEC is available.

Interpreting Score Differences

RMSECV << RMSEP: The model may be overfitting to training data patterns that don’t generalize. Consider simpler models or more regularization.
RMSECV >> RMSEP: Unusual but possible with small datasets or when the test set happens to be “easier” than the CV validation folds.
Ens_Test < RMSEP: The ensemble of fold models outperforms the single refit model. This can happen when model variance is high – consider increasing regularization or using ensemble-based deployment.
Ens_Test ≈ RMSEP: Good sign. The refit model performs comparably to the ensemble, confirming stable model behavior.

Multi-Criteria Refit

When using multiple selection criteria, each criterion independently selects its top-K pipelines. Duplicates are skipped (if a pipeline is selected by RMSECV, it won’t be selected again by mean_val). The final scores table marks the best model per criterion with a star (*).

This is useful when different ranking criteria identify genuinely different model strengths (e.g., RMSECV favors models with consistent fold performance, while mean_val may favor models with lower average error).