Hyperparameter Tuning

Automatically optimize model hyperparameters using Optuna integration.

Overview

nirs4all provides built-in hyperparameter optimization through Optuna, a state-of-the-art hyperparameter optimization framework. This allows you to automatically find the best model configuration without manual trial-and-error.

Key features:

Multiple search methods: Grid search, random search, Bayesian optimization (TPE), CMA-ES
Flexible parameter types: Integer ranges, float ranges, log-scale, categorical choices
Tuning approaches: Global search, per-preprocessing group, per-fold optimization
Pruning: Early termination of unpromising trials (median, successive halving, hyperband)
Multi-phase search: Exploration followed by exploitation
Custom metrics: Optimize for RMSE, R2, MAE, accuracy, etc.
Reproducibility: Seed support for deterministic results
Seamless integration: Works with any sklearn-compatible model

Basic Usage

Enable hyperparameter tuning by adding finetune_params to your model step:

from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import ShuffleSplit
import nirs4all

result = nirs4all.run(
    pipeline=[
        ShuffleSplit(n_splits=5, test_size=0.2, random_state=42),
        {
            "model": PLSRegression(),
            "finetune_params": {
                "n_trials": 20,
                "sampler": "tpe",
                "model_params": {
                    "n_components": ('int', 1, 20),
                }
            }
        }
    ],
    dataset="path/to/data",
    name="HyperparameterTuning"
)

print(f"Best score: {result.best_score:.4f}")

Configuration Options

finetune_params Structure

{
    "model": SomeModel(),
    "finetune_params": {
        "n_trials": 20,              # Number of optimization trials
        "sampler": "tpe",            # Search method
        "verbose": 1,                # Logging verbosity (0, 1, or 2)
        "approach": "single",        # Tuning approach
        "eval_mode": "best",         # For grouped: "best", "mean", "robust_best"
        "seed": 42,                  # Reproducible optimization
        "metric": "rmse",           # Metric to optimize (auto-infers direction)
        "pruner": "median",          # Prune unpromising trials
        "model_params": {            # Parameters to optimize
            "param1": ('int', 1, 10),
            "param2": ('float', 0.01, 1.0),
            "param3": [True, False],
        },
        "train_params": {            # Training params (also sampable)
            "epochs": ('int', 50, 300),
            "verbose": 0,            # Static value (not sampled)
        },
    }
}

Parameter Type Specifications

Tuple Format (Most Common)

# Integer ranges
"n_components": ('int', 1, 20)          # Uniform integer 1-20
"n_estimators": ('int_log', 10, 1000)   # Log-uniform integer

# Float ranges
"learning_rate": ('float', 0.001, 0.1)      # Uniform float
"alpha": ('float_log', 1e-4, 1e2)           # Log-uniform (for regularization)

# Categorical choices
"kernel": ['linear', 'rbf', 'poly']
"scale": [True, False]

Dict Format (Most Flexible)

# Integer with step
"n_estimators": {'type': 'int', 'min': 10, 'max': 200, 'step': 10}

# Integer with log scale
"n_components": {'type': 'int', 'min': 1, 'max': 1000, 'log': True}

# Float with log scale
"learning_rate": {'type': 'float', 'min': 1e-5, 'max': 1e-1, 'log': True}

# Categorical
"max_depth": {'type': 'categorical', 'choices': [3, 5, 7, 10]}

Search Methods (Samplers)

Method	Description	Best For
`grid`	Exhaustive search over all combinations	Small categorical spaces
`random`	Random sampling	Quick baseline, large spaces
`tpe`	Tree-Parzen Estimator (Bayesian)	Medium to large spaces (default)
`cmaes`	CMA Evolution Strategy	Continuous parameters
`auto`	Automatic selection based on parameter types	When unsure

Pruning

Pruners terminate unpromising trials early, saving computation time:

Pruner	Description
`none`	No pruning (default)
`median`	Prune if worse than median of completed trials
`successive_halving`	Prune poorest fraction at each step
`hyperband`	Adaptive resource allocation with early stopping

"finetune_params": {
    "n_trials": 50,
    "sampler": "tpe",
    "pruner": "median",         # Enable pruning
    "approach": "grouped",       # Pruning works with grouped approach
    "model_params": {
        "n_components": ('int', 1, 30),
    }
}

Custom Metrics

By default, finetuning minimizes MSE for regression or maximizes balanced accuracy for classification. Use metric to optimize for a different objective:

# Optimize for R2 (auto-infers direction=maximize)
"finetune_params": {
    "metric": "r2",
    "model_params": {"alpha": ('float_log', 1e-4, 1e2)},
}

# Optimize for RMSE (auto-infers direction=minimize)
"finetune_params": {
    "metric": "rmse",
    "model_params": {"n_components": ('int', 1, 20)},
}

Supported metrics and auto-inferred directions:

Metric	Direction	Task
`mse`, `rmse`, `mae`	minimize	Regression
`r2`	maximize	Regression
`accuracy`, `balanced_accuracy`, `f1`	maximize	Classification

Tuning Approaches

Single (Global Search)

Runs one optimization across all data:

"approach": "single"

Fastest: One search for the entire pipeline
Use when: You want one optimal configuration

Grouped (Per-Preprocessing)

Optimizes separately for each preprocessing variant:

"approach": "grouped",
"eval_mode": "best"  # or "mean" or "robust_best"

Balanced: Each preprocessing gets its own optimal hyperparameters
Use when: Using feature_augmentation to compare preprocessings

Individual (Per-Fold)

Optimizes separately for each cross-validation fold:

"approach": "individual"

Most thorough: Different hyperparameters per fold
Use when: Maximum customization, have computational budget

Advanced Features

Multi-Phase Optimization

Run different samplers sequentially on a shared study. Phase 1 explores broadly, Phase 2 exploits promising regions:

"finetune_params": {
    "seed": 42,
    "metric": "rmse",
    "phases": [
        {"n_trials": 50, "sampler": "random"},   # Broad exploration
        {"n_trials": 100, "sampler": "tpe"},      # Focused exploitation
    ],
    "model_params": {
        "alpha": ('float_log', 1e-4, 1e2),
        "l1_ratio": ('float', 0.0, 1.0),
    },
}

Force-Params (Seeding Known Configurations)

Enqueue a known good configuration as trial 0 so the optimizer always evaluates your baseline first:

"finetune_params": {
    "n_trials": 50,
    "sampler": "tpe",
    "force_params": {"n_components": 5},
    "model_params": {
        "n_components": ('int', 1, 20),
    },
}

Training Parameter Tuning

For neural networks and other models with training parameters, use train_params to tune training configuration alongside model hyperparameters:

"finetune_params": {
    "n_trials": 30,
    "sampler": "tpe",
    "model_params": {
        "filters_1": [8, 16, 32],
        "dropout_rate": ('float', 0.1, 0.5),
    },
    "train_params": {
        "epochs": ('int', 10, 100),      # Sampled by Optuna
        "batch_size": [16, 32, 64],       # Sampled (categorical)
        "verbose": 0,                     # Static value (not sampled)
    },
}

Reproducibility

Use seed for deterministic optimization results:

"finetune_params": {
    "n_trials": 50,
    "sampler": "tpe",
    "seed": 42,              # Same seed + data = same results
    "model_params": {...},
}

Stack Helper (Stacking Model Optimization)

For sklearn StackingRegressor and StackingClassifier, use the stack_params() helper to easily finetune the final estimator (metamodel):

from sklearn.ensemble import StackingRegressor
from sklearn.linear_model import Ridge
from sklearn.cross_decomposition import PLSRegression
from nirs4all.optimization.optuna import stack_params

# Define base estimators
base_estimators = [
    ("pls", PLSRegression(n_components=5)),
    ("ridge", Ridge(alpha=1.0)),
]

result = nirs4all.run(pipeline=[
    ShuffleSplit(n_splits=3, random_state=42),
    {
        "model": StackingRegressor(
            estimators=base_estimators,
            final_estimator=Ridge(),
            cv=3,
        ),
        "finetune_params": {
            "n_trials": 20,
            "sampler": "tpe",
            "model_params": stack_params(
                final_estimator_params={
                    "alpha": ('float_log', 1e-3, 1e2),      # Finetune metamodel alpha
                    "fit_intercept": [True, False],          # Finetune metamodel fit_intercept
                },
                passthrough=True,  # Stack-level parameter (optional)
            ),
        }
    }
], dataset="data/")

The stack_params() helper automatically namespaces final estimator parameters with the final_estimator__ prefix required by sklearn. This works seamlessly with the existing nested parameter system.

Combining with Preprocessing Search

Combine feature_augmentation with hyperparameter tuning to find the best preprocessing + hyperparameter combination:

from nirs4all.operators.transforms import StandardNormalVariate, Detrend, FirstDerivative

result = nirs4all.run(pipeline=[
    # Generate preprocessing variants
    {"feature_augmentation": [StandardNormalVariate, Detrend, FirstDerivative], "action": "extend"},

    ShuffleSplit(n_splits=3, random_state=42),

    {
        "model": PLSRegression(),
        "finetune_params": {
            "n_trials": 10,
            "sampler": "tpe",
            "approach": "grouped",  # Optimize per preprocessing
            "model_params": {
                "n_components": ('int', 1, 20),
            }
        }
    }
], dataset="data/")

Pipeline Generators vs Optuna Tuning

nirs4all offers two approaches to hyperparameter exploration:

1. Pipeline Generators (`_range_`, `_or_`)

Create multiple complete pipelines upfront:

pipeline = [
    {"model": PLSRegression(), "_range_": [1, 10], "param": "n_components"}
]

Pros: Full control, parallel execution, complete reproducibility
Cons: Combinatorial explosion with many parameters

2. Optuna Tuning (`finetune_params`)

Intelligent search within a single pipeline:

pipeline = [
    {
        "model": PLSRegression(),
        "finetune_params": {
            "n_trials": 50,
            "sampler": "tpe",
            "model_params": {"n_components": ('int', 1, 20)}
        }
    }
]

Pros: Efficient for large spaces, early stopping, Bayesian optimization
Cons: Sequential trials, less transparent

When to Use Each

Scenario	Recommended
Few parameters (< 10 combinations)	`_range_` / `_or_`
Many parameters (> 50 combinations)	`finetune_params`
Neural network architecture + training	`finetune_params` with `train_params`
Reproducible benchmark	`_range_` / `_or_`
Quick exploration	`finetune_params` with `random`

Viewing Optimization Results

Access optimization details from the predictions:

result = nirs4all.run(pipeline, dataset)

# Best configuration
print(f"Best score: {result.best_score:.4f}")

# Top configurations
for pred in result.top(5, display_metrics=['rmse', 'r2']):
    print(f"  {pred.get('model_name')}: RMSE={pred.get('rmse'):.4f}")

# Visualize results
from nirs4all.visualization.predictions import PredictionAnalyzer

analyzer = PredictionAnalyzer(result.predictions)
analyzer.plot_top_k(k=10, rank_metric='rmse')
analyzer.plot_heatmap(x_var="model_name", y_var="preprocessings", rank_metric='rmse')

Native engine (`n4m`) — portable, cross-language finetuning

By default finetuning runs on Optuna. Adding "engine": "n4m" to finetune_params runs the native optimizer shipped in nirs4all-methods (libn4m) instead — the same C-ABI optimizer that powers finetuning in R, MATLAB/Octave and WebAssembly. At a fixed seed, the trial sequence is identical across every binding.

{
    "model": PLSRegression(),
    "finetune_params": {
        "engine": "n4m",                # <-- the only change vs. the Optuna engine
        "n_trials": 30,
        "sampler": "tpe",               # random|sobol|lhs|ternary|ga|pso|cmaes|tpe|gp_ei
        "pruner": "median",             # none|median|successive_halving|hyperband|racing
        "approach": "grouped",          # single|grouped|individual
        "metric": "rmse",               # direction auto-inferred
        "seed": 42,
        "model_params": {
            "n_components": ("int", 1, 30),
        },
    },
}

Everything else is unchanged: the search-space DSL (tuple ranges, choice lists, dict specs, __-nested groups), the approaches (single / grouped / individual), metric / direction, eval_mode, and force_params warm-start all behave exactly as on the Optuna engine, and the same FinetuneResult (best params, trial history) is returned. Drop "engine" (or set it to "optuna") to go back.

Samplers. random, sobol (low-discrepancy, bit-exact to scipy.stats.qmc.Sobol), lhs, ternary (unimodal integers, e.g. PLS n_components), ga, pso, cmaes (continuous spaces), tpe (mixed/conditional spaces), gp_ei (Bayesian, sample-efficient). auto maps to tpe; the Optuna-only grid maps to random.

Pruners. median, successive_halving (ASHA), hyperband, and racing. Pruning reports each fold’s aggregated score and stops unpromising trials early. racing is the fold-safe choice when folds are exchangeable (successive halving assumes rank-preservation across rungs, which CV folds do not guarantee).

Conditional attributes (when). A parameter can be made active only when a sibling categorical takes a given value — the object__attribute conditional case. Add a when (or when_not) clause to a dict spec:

"model_params": {
    "kernel": ["linear", "rbf", "poly"],
    "gamma":  {"type": "float_log", "min": 1e-4, "max": 1e1, "when": {"kernel": ["rbf", "poly"]}},
    "degree": {"type": "int", "min": 2, "max": 4, "when": {"kernel": "poly"}},
}

gamma is then sampled (and passed to the model) only when kernel is rbf or poly; degree only for poly. Conditions nest (a conditional attribute can itself gate deeper attributes), which is the substrate for tuning sub-pipelines / operators in the search space. Use tpe (tree-structured) for conditional spaces. This clause is native-engine only.

Operators in the search space. With a scikit-learn Pipeline as the model, you can put the operators themselves in the search space — a joint search over structure and hyperparameters. A {"type": "categorical", "options": {name: operator}} spec selects which operator fills a pipeline step, and object__attribute (set_params) addressing with a when clause tunes each operator’s own hyperparameters conditionally:

"model": Pipeline([("scale", StandardScaler()), ("est", PLSRegression())]),
"finetune_params": {"engine": "n4m", "sampler": "tpe", "n_trials": 40,
    "model_params": {
        "scale": {"type": "categorical", "options": {"std": StandardScaler(), "mm": MinMaxScaler()}},
        "est":   {"type": "categorical", "options": {"pls": PLSRegression(), "ridge": Ridge()}},
        "est__n_components": {"type": "int",       "min": 1,    "max": 20, "when": {"est": "pls"}},
        "est__alpha":        {"type": "float_log", "min": 1e-4, "max": 1e2, "when": {"est": "ridge"}},
    }}

The optimizer chooses the scaler + estimator and tunes only the chosen estimator’s parameters (PLS’s n_components or Ridge’s alpha, never both). The options names give operators stable labels for when; est__param stays flat so scikit-learn set_params routes it to the chosen step. (Choosing preprocessing operators that run before the model step, or whole sub-pipelines, is the broader pipeline-compiler work tracked in nirs4all-methods docs/PIPELINE_FINETUNING.md.)

Requirements. An up-to-date nirs4all-methods install exposing the native optimizer — verify with python -c "from n4m.model_selection.optimizer import Optimizer". When the native optimizer is unavailable, finetune_params without "engine" still runs on Optuna.

Best Practices

Start small: Begin with n_trials=10 and sampler="random" to establish a baseline
Use TPE for refinement: Switch to sampler="tpe" with more trials for fine-tuning
Set reasonable bounds: Keep parameter ranges realistic to avoid wasted trials
Use log-scale for regularization: Parameters like alpha, learning_rate benefit from ('float_log', ...)
Monitor progress: Use verbose=2 to see trial-by-trial progress
Use cross-validation: Ensure robust evaluation with multiple folds
Seed for reproducibility: Always set seed when comparing configurations
Multi-phase for large spaces: Use random exploration followed by TPE exploitation
Custom metrics: Use metric to align optimization with your evaluation criteria