Cross-Validation Examples

This section covers cross-validation strategies to properly evaluate model performance on NIRS data.

Overview 

Example	Topic	Difficulty	Duration
U01	CV Strategies	★★☆☆☆	~4 min
U02	Group Splitting	★★☆☆☆	~3 min
U03	Sample Filtering	★★☆☆☆	~3 min
U04	Repetition Aggregation	★★★☆☆	~4 min

U01: CV Strategies 

Select appropriate cross-validation for your data structure.

📄 View source code

What You’ll Learn 

Standard CV: KFold, ShuffleSplit, RepeatedKFold
Stratified CV for classification
Time-series CV for temporal data
Leave-One-Out for small datasets

CV Strategy Selection Guide 

Strategy	When to Use
KFold	Standard regression, moderate-large datasets
ShuffleSplit	Flexible test size, many random splits
RepeatedKFold	Small datasets, need robust estimates
StratifiedKFold	Classification, class imbalance
StratifiedShuffleSplit	Classification + flexible splits
TimeSeriesSplit	Temporal/sequential data
LeaveOneOut	Very small datasets (<50 samples)
SPXYFold	Spatially representative folds via SPXY
KennardStoneSplitter	Kennard-Stone based train/test split
KBinsStratifiedSplitter	Stratification for continuous targets

KFold - Standard K-Fold 

Divides data into K non-overlapping folds:

from sklearn.model_selection import KFold

pipeline = [
    MinMaxScaler(),
    SNV(),

    # 5-fold cross-validation
    KFold(n_splits=5, shuffle=True, random_state=42),

    PLSRegression(n_components=10)
]

Key parameters:

n_splits: Number of folds (typically 5-10)
shuffle: Randomize before splitting (recommended)
random_state: For reproducibility

ShuffleSplit - Random Splits 

More flexible than KFold:

from sklearn.model_selection import ShuffleSplit

# 10 random splits with 25% test
ShuffleSplit(n_splits=10, test_size=0.25, random_state=42)

Advantages:

Control test size exactly
Number of splits independent of test size
Good for large datasets

RepeatedKFold - Multiple Repetitions 

Repeats K-fold CV multiple times with different shuffles:

from sklearn.model_selection import RepeatedKFold

# 5-fold repeated 3 times = 15 total evaluations
RepeatedKFold(n_splits=5, n_repeats=3, random_state=42)

Best for:

Small datasets where variance is high
When you need robust uncertainty estimates

StratifiedKFold - Classification 

Preserves class proportions in each fold:

from sklearn.model_selection import StratifiedKFold

# Essential for imbalanced classification
StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

Critical for:

Imbalanced class distributions
Multi-class classification
Ensuring each fold has representative samples

TimeSeriesSplit - Temporal Data 

Expanding window approach for sequential data:

from sklearn.model_selection import TimeSeriesSplit

# Train on past, test on future
TimeSeriesSplit(n_splits=5)

Prevents:

Look-ahead bias
Data leakage from future to past

Use when:

Samples are ordered by time
Temporal dependencies exist

LeaveOneOut - Small Datasets 

Leave exactly one sample out per fold:

from sklearn.model_selection import LeaveOneOut

# N samples = N folds
LeaveOneOut()

Trade-offs:

✓ Maximum use of data for training
✓ N evaluations give complete picture
✗ Slow for large datasets
✗ High variance in estimates

U02: Group Splitting 

Handle grouped/clustered data correctly.

📄 View source code

What You’ll Learn 

GroupKFold for clustered samples
GroupShuffleSplit for random group splits
Handling biological replicates
Avoiding data leakage from groups

Why Group Splitting?

When samples are not independent:

Example: 5 measurements per fruit
├── Fruit 1: samples 1-5
├── Fruit 2: samples 6-10
├── Fruit 3: samples 11-15
└── ...

Wrong: Random split → Fruit 1's samples in both train AND test
Right: Group split → All of Fruit 1's samples in train OR test

GroupKFold 

from sklearn.model_selection import GroupKFold

# Requires group labels
groups = [0]*5 + [1]*5 + [2]*5 + [3]*5 + [4]*5  # 5 groups of 5

# Pass groups in dataset
result = nirs4all.run(
    pipeline=[
        MinMaxScaler(),
        SNV(),
        GroupKFold(n_splits=5),
        PLSRegression(n_components=10)
    ],
    dataset=(X, y, {"groups": groups})
)

GroupShuffleSplit 

Random group-aware splits:

from sklearn.model_selection import GroupShuffleSplit

GroupShuffleSplit(n_splits=10, test_size=0.2, random_state=42)

Common Grouping Scenarios 

Scenario	Group Definition
Biological replicates	Sample/individual ID
Time series by subject	Subject ID
Multi-site study	Site ID
Batch effects	Batch ID

U03: Sample Filtering 

Filter samples based on criteria during cross-validation.

📄 View source code

What You’ll Learn 

Filtering outliers
Conditioning on metadata
Train/test set requirements

Sample Filtering in Pipeline 

pipeline = [
    MinMaxScaler(),
    SNV(),

    # Filter: only include samples meeting criteria
    {"filter": {
        "y_range": (0, 100),       # Target value range
        "partition": ["train"]     # Which partitions to filter
    }},

    ShuffleSplit(n_splits=3),
    PLSRegression(n_components=10)
]

Filtering Options 

Filter	Description
`y_range`	Keep samples with y in (min, max)
`x_range`	Keep samples with X values in range
`metadata`	Filter based on metadata columns
`outliers`	Remove statistical outliers

U04: Aggregation 

Aggregate repeated measurements into one prediction per physical sample.

📄 View source code

What You’ll Learn 

Setting repetition in DatasetConfigs
Comparing raw vs sample-aggregated metrics
Overriding aggregation for individual plots
Ranking models with custom aggregation columns

Understanding Aggregation 

When several spectra belong to the same physical sample, evaluation can happen at two levels:

Raw: one prediction per spectrum
Aggregated: one prediction per sample after combining repetitions
Custom: one prediction per another metadata key such as lot_id

The example enables this with the dataset repetition column:

dataset_config = DatasetConfigs(
    {
        "train_x": "...",
        "train_y": "...",
        "train_m": "...",
        "test_x": "...",
        "test_y": "...",
        "test_m": "...",
    },
    repetition="sample_id",
)

Aggregation Options 

Option	Description
`repetition="sample_id"`	Use a metadata column as the default aggregation key
`predictions.top(..., by_repetition=False)`	Rank using raw per-spectrum metrics
`predictions.top(..., by_repetition="sample_id")`	Rank using sample-level metrics
`predictions.top(..., by_repetition="lot_id")`	Rank using another metadata grouping

Visualization with Aggregation 

analyzer = PredictionAnalyzer(predictions)

# Raw + sample_id aggregation
analyzer.plot_top_k(k=3, rank_metric="rmse")

# Raw + custom lot_id aggregation
analyzer.plot_heatmap(
    "partition",
    "model_name",
    rank_metric="rmse",
    display_metric="rmse",
    aggregate="lot_id",
)

# Raw only for this chart
analyzer.plot_histogram(display_metric="rmse", aggregate="")

When to Use It 

Repeated technical measurements with the same target
Multiple spectra per physical sample
Sample-level reporting or ranking

Avoid it when each spectrum is an independent sample or when the target changes between repetitions.

CV Best Practices 

1. Match CV to Data Structure 

# Independent samples
ShuffleSplit(n_splits=5, test_size=0.2)

# Classification
StratifiedKFold(n_splits=5)

# Grouped samples
GroupKFold(n_splits=5)

# Time series
TimeSeriesSplit(n_splits=5)

2. Number of Splits 

Dataset Size	Recommended Splits
< 50	LeaveOneOut or RepeatedKFold
50-200	5-fold with repeats
200-1000	5-10 fold
> 1000	ShuffleSplit (faster)

3. Always Shuffle for Regression 

# Good: Shuffle before splitting
KFold(n_splits=5, shuffle=True, random_state=42)

# Risky: Sequential splits may have patterns
KFold(n_splits=5, shuffle=False)  # Only for time series

4. Set random_state for Reproducibility 

# Reproducible
ShuffleSplit(n_splits=5, random_state=42)

# Different each run (not reproducible)
ShuffleSplit(n_splits=5)  # random_state=None

5. Validate Stratification 

For classification, check class distribution in each fold:

result = nirs4all.run(pipeline=pipeline, dataset=dataset)

# View fold distribution
nirs4all.run(
    pipeline=[..., "fold_chart", ...],
    dataset=dataset,
    plots_visible=True
)

Running These Examples 

cd examples

# Run all CV examples
./run.sh -n "U0*.py" -c user

# Run with plots to see fold distributions
python user/05_cross_validation/U01_cv_strategies.py --plots --show

Next Steps 

After mastering cross-validation:

Deployment: Save and deploy trained models
Explainability: Understand model decisions
Advanced: Nested CV, custom splitters