Sample Repetition & Aggregation

Overview

In NIRS applications, it’s common to have multiple spectral measurements (repetitions) for the same physical sample. For example:

4 scans per soil sample to reduce measurement noise
Multiple measurements at different positions on a grain sample
Repeated measurements for quality control

The repetition feature handles this by:

Fold grouping: Keeping all spectra from the same sample in the same cross-validation fold (prevents data leakage)
Prediction aggregation: Combining predictions from multiple spectra into one prediction per physical sample, then reporting both raw and aggregated metrics

Setting repetition automatically enables both behaviors. You only need a single parameter.

Quick Start

import nirs4all
from nirs4all.data import DatasetConfigs
from nirs4all.visualization.predictions import PredictionAnalyzer
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import ShuffleSplit
from sklearn.preprocessing import MinMaxScaler

# Define dataset with repetition column
dataset = DatasetConfigs(
    "path/to/spectra",
    repetition="sample_id"  # Metadata column identifying physical samples
)

# Define pipeline

# Run pipeline — folds are grouped by sample_id, aggregated metrics are reported
result = nirs4all.run(pipeline=[
    MinMaxScaler(),
    ShuffleSplit(n_splits=3, test_size=0.25),
    {"model": PLSRegression(n_components=10)}
], dataset=dataset, name="PLS", verbose=1)

# Create analyzer with same aggregate setting
analyzer = PredictionAnalyzer(result.predictions, default_aggregate=result.last_aggregate)

# All plots now use aggregation by default
fig = analyzer.plot_top_k(k=5)  # Automatically aggregated by sample_id

Setting Repetition

Via Constructor Parameter (Recommended)

# Metadata file should contain a column like 'sample_id', 'ID', 'batch', etc.
dataset = DatasetConfigs("path/to/data", repetition="sample_id")

Via Config Dictionary

config = {
    "train_x": "data/spectra.csv",
    "train_y": "data/targets.csv",
    "train_m": "data/metadata.csv",  # Contains 'sample_id' column
    "repetition": "sample_id"
}

dataset = DatasetConfigs(config)

Aggregate-Only (Without Fold Grouping)

In rare cases you may want aggregation with a different column than fold grouping, or aggregation by target values:

# Aggregate by target values (y_true) — no fold grouping
dataset = DatasetConfigs("path/to/data", aggregate=True)

# Different columns for fold grouping and aggregation
dataset = DatasetConfigs("path/to/data", repetition="sample_id", aggregate="batch_id")

Pipeline Output

When aggregation is enabled, the TabReport shows both raw and aggregated metrics:

|-----------|--------|----------|------|-------|
| Partition | Nsamp  | Nfeat    | R2   | RMSE  |
|-----------|--------|----------|------|-------|
| Cros Val  | 400    | 200      | 0.87 | 0.712 |
| Cros Val* | 100    | 200      | 0.92 | 0.598 |  <- Aggregated
| Test      | 100    | 200      | 0.85 | 0.756 |
| Test*     | 25     | 200      | 0.90 | 0.632 |  <- Aggregated
|-----------|--------|----------|------|-------|
* Aggregated by sample_id

The asterisk (*) rows show performance when predictions for repeated measurements are averaged before computing metrics.

Visualization with Aggregation

Automatic Aggregation via `default_aggregate`

When you set default_aggregate on the analyzer, all visualization methods use it automatically:

# Get aggregate setting from last run
analyzer = PredictionAnalyzer(result.predictions, default_aggregate=result.last_aggregate)

# All these plots use aggregation automatically
fig1 = analyzer.plot_top_k(k=5)
fig2 = analyzer.plot_histogram()
fig3 = analyzer.plot_heatmap('model_name', 'preprocessings')
fig4 = analyzer.plot_candlestick('model_name')

Overriding the Default

You can override the default for specific plots:

# Use default aggregation
fig1 = analyzer.plot_top_k(k=5)

# Override: disable aggregation for this plot
fig2 = analyzer.plot_top_k(k=5, aggregate='')

# Override: use different aggregation column
fig3 = analyzer.plot_top_k(k=5, aggregate='batch_id')

Manual Aggregation per Plot

Without setting a default, specify aggregation per method call:

analyzer = PredictionAnalyzer(predictions)

# Explicit aggregation
fig = analyzer.plot_top_k(k=5, aggregate='sample_id')
fig = analyzer.plot_heatmap('model', 'preprocessing', aggregate='sample_id')

Multi-Dataset Configuration

Different datasets can have different repetition columns:

config1 = {
    "train_x": "dataset1/spectra.csv",
    "train_y": "dataset1/targets.csv",
    "train_m": "dataset1/metadata.csv",
    "repetition": "sample_id"
}

config2 = {
    "train_x": "dataset2/spectra.csv",
    "train_y": "dataset2/targets.csv",
    "train_m": "dataset2/metadata.csv",
    "repetition": "batch_number"
}

dataset = DatasetConfigs([config1, config2])

Alternatively, use a list:

dataset = DatasetConfigs(
    [config1, config2],
    repetition=["sample_id", "batch_number"]
)

Priority Resolution

When repetition/aggregation is specified in multiple places, the priority order is:

Constructor parameter (highest priority)
Config dictionary (lower priority)

config = {
    "train_x": "...",
    "repetition": "sample_id"  # Config-level setting
}

# Constructor parameter overrides config dict
dataset = DatasetConfigs(config, repetition="batch_id")  # Uses "batch_id"

Automatic Linking

Setting repetition automatically enables aggregation with the same column
Setting aggregate (string) automatically enables fold grouping with the same column
You can override this by setting both repetition and aggregate to different values

Aggregation Algorithm

For regression tasks:

Predictions for samples in the same group are averaged
y_true values are also averaged (for consistent comparison)

For classification tasks:

Probabilities (if available) are averaged, then argmax is applied
Without probabilities, majority voting is used

Complete Example

"""
Example: Soil Analysis with Multiple Scans per Sample
Each soil sample has 4 spectral scans to reduce measurement noise.
"""

import nirs4all
from nirs4all.data import DatasetConfigs
from nirs4all.visualization.predictions import PredictionAnalyzer
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import ShuffleSplit
from sklearn.preprocessing import MinMaxScaler
import matplotlib.pyplot as plt

# Dataset config with aggregation
dataset = DatasetConfigs(
    {
        "train_x": "soil_data/spectra_train.csv",
        "train_y": "soil_data/targets_train.csv",
        "train_m": "soil_data/metadata_train.csv",  # Has 'sample_id' column
        "test_x": "soil_data/spectra_test.csv",
        "test_y": "soil_data/targets_test.csv",
        "test_m": "soil_data/metadata_test.csv",
    },
    aggregate="sample_id"
)

# Pipeline with hyperparameter search
pipeline = [
    MinMaxScaler(),
    ShuffleSplit(n_splits=5, test_size=0.2, random_state=42),
]

# Add models with different n_components
for n in [5, 10, 15, 20]:
    pipeline.append({"model": PLSRegression(n_components=n)})

# Run
result = nirs4all.run(pipeline=pipeline, dataset=dataset, name="SoilPLS", verbose=1)

# Analyze with aggregation
analyzer = PredictionAnalyzer(result.predictions, default_aggregate=result.last_aggregate)

# All visualizations use aggregated metrics
fig1 = analyzer.plot_top_k(k=3, rank_metric='rmse')
fig2 = analyzer.plot_heatmap('model_name', 'preprocessings')

plt.show()

Repetition Details

How Repetition Works

The repetition column identifies which spectra belong to the same physical sample. When set, it enables two behaviors:

Cross-validation grouping: All spectra from the same sample stay in the same fold, preventing data leakage
Prediction aggregation: Both raw (per-spectrum) and aggregated (per-sample) metrics are reported

from nirs4all.data import DatasetConfigs

# Set repetition at creation time (recommended)
dataset = DatasetConfigs("path/to/data", repetition="Sample_ID")

# Check current setting
print(f"Repetition column: {dataset.repetition}")

Repetition Statistics

Query statistics about how repetitions are distributed across samples:

# Get detailed statistics
stats = dataset.repetition_stats

print(f"Number of unique samples: {stats['n_groups']}")
print(f"Repetitions per sample: {stats['min']}-{stats['max']} (avg: {stats['mean']:.1f})")
print(f"Variable counts: {stats['is_variable']}")

The statistics dictionary includes:

n_groups: Number of unique samples
min: Minimum repetitions per sample
max: Maximum repetitions per sample
mean: Average repetitions per sample
std: Standard deviation of repetition counts
is_variable: True if samples have different numbers of repetitions

Repetition Groups

Access the raw grouping information:

# Get mapping of sample IDs to row indices
groups = dataset.repetition_groups
# Example: {'sample_001': [0, 1, 2, 3], 'sample_002': [4, 5, 6, 7], ...}

# Count samples
n_samples = len(groups)

# Find which samples have most repetitions
max_reps = max(len(v) for v in groups.values())
print(f"Maximum repetitions: {max_reps}")

Advanced Aggregation Configuration

In most cases, setting repetition is sufficient — aggregation is automatically enabled with the same column. The following options are for advanced use cases where you need finer control.

Independent Aggregation

Use set_aggregate() directly when the aggregation column differs from the repetition column:

# Aggregate by a different column than repetition
dataset.set_aggregate('batch_id')

# Aggregate by target values (y_true)
dataset.set_aggregate(True)

# Disable aggregation (keep fold grouping only)
dataset.set_aggregate(None)

# Check current setting
agg_setting = dataset.aggregate
print(f"Aggregation: {agg_setting}")

Aggregation Methods

Different aggregation methods are available depending on your task:

Method	Use Case	Description
`mean`	Regression (default)	Average predictions within each sample group
`median`	Regression (robust)	Median prediction within each group (outlier-resistant)
`robust_mean`	Regression (robust)	Mean after excluding outliers using T² statistic
`vote`	Classification	Majority voting or probability averaging

Set the aggregation method:

# Use median instead of mean
dataset.set_aggregate_method('median')

# Use robust mean (with automatic outlier exclusion)
dataset.set_aggregate_method('robust_mean')

# Check current method
method = dataset.aggregate_method
print(f"Aggregation method: {method}")

Outlier Exclusion

For robust aggregation, you can enable Hotelling’s T² based outlier exclusion:

# Enable outlier exclusion with 95% confidence threshold
dataset.set_aggregate_exclude_outliers(True, threshold=0.95)

# Use stricter threshold (99%)
dataset.set_aggregate_exclude_outliers(True, threshold=0.99)

# Disable outlier exclusion
dataset.set_aggregate_exclude_outliers(False)

# Check settings
is_enabled = dataset.aggregate_exclude_outliers
threshold = dataset.aggregate_outlier_threshold
print(f"Outlier exclusion: {is_enabled} (threshold: {threshold})")

When enabled, outlier repetitions are identified using the T² statistic and excluded before averaging. This is useful when you have occasional bad measurements within an otherwise good set of repetitions.

Complete Example

Here’s a complete workflow showing repetition handling with aggregation:

import nirs4all
from nirs4all.data import DatasetConfigs
from nirs4all.visualization.predictions import PredictionAnalyzer
from sklearn.cross_decomposition import PLSRegression
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import ShuffleSplit

# Load data with repetition — automatically enables fold grouping AND aggregation
dataset = DatasetConfigs(
    "soil_samples/",
    repetition="Sample_ID",          # Single entry point for both features
    aggregate_method="robust_mean",  # Optional: override aggregation method
    aggregate_exclude_outliers=True,  # Optional: T² outlier exclusion
)

# Run pipeline
result = nirs4all.run(
    pipeline=[
        MinMaxScaler(),
        ShuffleSplit(n_splits=5, test_size=0.2),
        {"model": PLSRegression(n_components=10)}
    ],
    dataset=dataset,
    name="SoilPLS",
    verbose=1,
)

# Results show both raw and aggregated metrics automatically
# Create analyzer with aggregation from the run
analyzer = PredictionAnalyzer(result.predictions, default_aggregate=result.last_aggregate)

# All visualizations use aggregated metrics
fig = analyzer.plot_top_k(k=3, rank_metric='rmse')

# Top models ranked by aggregated metrics
top_models = result.top(5, rank_metric='rmse', by_repetition='Sample_ID')
for i, model in enumerate(top_models, 1):
    print(f"{i}. RMSE={model.best_rmse:.4f}")

Properties and Methods Reference

Properties

dataset.repetition → Optional[str]

Get the column name identifying sample repetitions.
dataset.repetition_groups → Dict[Any, List[int]]

Get sample groups by repetition column (mapping of sample IDs to row indices).
dataset.repetition_stats → Dict[str, Any]

Get statistics about repetition counts (n_groups, min, max, mean, std, is_variable).
dataset.aggregate → Optional[str]

Get the aggregation setting (‘y’ for target-based, column name, or None).
dataset.aggregate_method → str

Get the aggregation method (‘mean’, ‘median’, ‘robust_mean’, or ‘vote’).
dataset.aggregate_exclude_outliers → bool

Get whether T² outlier exclusion is enabled.
dataset.aggregate_outlier_threshold → float

Get the outlier detection threshold (0-1).

Methods

dataset.set_repetition(column: Optional[str]) → None

Set the column name identifying sample repetitions. Pass None to disable. Note: when using DatasetConfigs(repetition=...), aggregation is automatically enabled.
dataset.set_aggregate(value: Union[str, bool, None]) → None

Enable sample-level aggregation. Pass True for y-based aggregation, column name for metadata-based, or None to disable. Normally set automatically from repetition.
dataset.set_aggregate_method(value: Optional[str]) → None

Set aggregation method (‘mean’, ‘median’, ‘robust_mean’, or ‘vote’).
dataset.set_aggregate_exclude_outliers(value: bool, threshold: float = 0.95) → None

Enable/disable T² based outlier exclusion before aggregation.

Sample Repetition & Aggregation

Overview

Quick Start

Setting Repetition

Via Constructor Parameter (Recommended)

Via Config Dictionary

Aggregate-Only (Without Fold Grouping)

Pipeline Output

Visualization with Aggregation

Automatic Aggregation via default_aggregate

Overriding the Default

Manual Aggregation per Plot

Multi-Dataset Configuration

Priority Resolution

Automatic Linking

Aggregation Algorithm

Complete Example

Repetition Details

How Repetition Works

Repetition Statistics

Repetition Groups

Advanced Aggregation Configuration

Independent Aggregation

Aggregation Methods

Outlier Exclusion

Complete Example

Properties and Methods Reference

Properties

Methods

See Also

Automatic Aggregation via `default_aggregate`