nirs4all.optimization.optuna module

Optuna Manager - External hyperparameter optimization logic

This module combines the best practices from the original optuna_manager for parameter handling and sampling with fold-based optimization strategies. It provides a clean interface for hyperparameter optimization across different strategies and frameworks.

class nirs4all.optimization.optuna.BinarySearchSampler(seed: int | None = None)[source]

Bases: BaseSampler

Gradient-based binary search sampler for unimodal integer parameters.

This sampler uses gradient information to climb toward the optimum, making it highly efficient for parameters with unimodal behavior (single peak), such as PLS n_components.

Typically reduces optimization from ~30-50 trials (TPE) to ~10-15 trials.

Strategy:

Initial phase: Test boundaries (low, high) and midpoint
Gradient detection: Test neighbors to determine direction of improvement
Search narrowing: Move search bounds in direction of gradient
Local refinement: Exhaustive search when range is small

Key difference from naive binary search: Instead of just refining around the best value found, this sampler follows the gradient direction. If the optimum is at n_components=5 but the initial midpoint (13) is better than boundaries, the algorithm will detect the gradient pointing left and explore [1, 13], eventually converging to 5.

Best for:

PLS/PCR n_components (most common use case)
KNN n_neighbors
Polynomial degree
Tree depth, number of leaves
Any integer parameter with clear unimodal behavior

Not suitable for:

Multi-modal parameters (multiple peaks)
Continuous float parameters (use TPE instead)
Categorical parameters (use Grid instead)

Parameters:: seed – Random seed for reproducibility.

Example

>>> finetune_params = {
...     "n_trials": 12,
...     "sampler": "binary",
...     "model_params": {
...         "n_components": ('int', 1, 30),
...     }
... }

Note

Works best with single integer parameter. For multiple parameters, will apply gradient-based search to integer parameters and random sampling to others.

infer_relative_search_space(study: Study, trial: FrozenTrial) → dict[str, Any][source]

Infer relative search space (not used by binary search).

Parameters:

study – Optuna study.
trial – Current trial.

Returns:

Empty dict (binary search doesn’t use relative search space).

sample_independent(study: Study, trial: FrozenTrial, param_name: str, param_distribution: BaseDistribution) → Any[source]

Sample using ternary search for unimodal integer parameters.

Ternary search strategy:

Test low, mid, high
Divide range into thirds with points m1, m2
Compare scores at m1 and m2 to eliminate one third
Repeat until convergence

For unimodal functions, this guarantees finding the optimum in O(log n) trials.

Parameters:

study – Optuna study.
trial – Current trial.
param_name – Parameter name.
param_distribution – Parameter distribution.

Returns:

Sampled parameter value.

sample_relative(study: Study, trial: FrozenTrial, search_space: dict[str, Any]) → dict[str, Any][source]

Sample relative parameters (not used by binary search).

Parameters:

study – Optuna study.
trial – Current trial.
search_space – Search space dict.

Returns:

Empty dict (binary search doesn’t use relative sampling).

nirs4all.optimization.optuna.stack_params(final_estimator_params: dict[str, Any] | None = None, **other_params: Any) → dict[str, Any][source]

Create Optuna-compatible parameter structure for sklearn Stacking models.

Helper to finetune the final_estimator (metamodel) of a StackingRegressor or StackingClassifier through Optuna. Automatically namespaces parameters with the final_estimator__ prefix required by sklearn.

The existing nested parameter flattening/unflattening system in OptunaManager will handle the __ separator transparently.

Parameters:

final_estimator_params – Parameter specs for the metamodel. Each value can be an Optuna parameter spec (range tuple, list, dict).
**other_params – Additional Stack parameters (e.g., cv, passthrough).

Returns:

Dict with properly namespaced parameters for Optuna sampling.

Example

>>> from nirs4all.optimization.optuna import stack_params
>>> finetune_params = {
...     "n_trials": 20,
...     "model_params": stack_params(
...         final_estimator_params={
...             "alpha": ("float", 1e-3, 1e0, "log"),
...             "fit_intercept": [True, False],
...         },
...         passthrough=True,  # Stack parameter
...     ),
... }
>>> # Optuna will sample final_estimator__alpha and final_estimator__fit_intercept