Generators

Generators let you define a search space inside a single pipeline definition. Instead of writing twenty pipelines by hand to compare different preprocessing methods or sweep hyperparameter values, you describe the space once and let nirs4all expand it into concrete variants.

Why Generators?

Real NIRS projects involve many choices:

  • Which preprocessing? (SNV, MSC, Detrend, derivatives, combinations)

  • How many PLS components? (5, 10, 15, 20, …)

  • Which model family? (PLS, Random Forest, SVM, …)

Without generators, you would write a separate pipeline for each combination. With generators, you express the search space declaratively, and the library creates every combination for you.

How Generators Work

A generator keyword is a special dictionary key that nirs4all recognises during pipeline normalisation. Before any training happens, the pipeline definition is expanded: each generator keyword is replaced with concrete values, producing a list of fully-specified pipeline variants.

Pipeline with generators       Expansion           Execution

[{_or_: [SNV, MSC]},    -->  [SNV, PLS(5)]    -->  train + score
 PLS({_range_: [5,10]})]     [SNV, PLS(10)]   -->  train + score
                              [MSC, PLS(5)]    -->  train + score
                              [MSC, PLS(10)]   -->  train + score

All variants are executed (optionally in parallel with n_jobs), and the results are ranked by cross-validation score. result.top(n) shows the best performers.

The _or_ Keyword

The most common generator. It creates one variant per alternative:

pipeline = [
    {"_or_": [SNV(), MSC(), Detrend()]},
    PLSRegression(n_components=10),
]
# Expands to 3 variants: SNV+PLS, MSC+PLS, Detrend+PLS

_or_ accepts classes, instances, or dicts. Each item in the list becomes a step in one of the expanded variants.

The _range_ Keyword

Creates a linear sweep of numeric values. Useful for parameters like n_components, window sizes, and polynomial orders:

pipeline = [
    SNV(),
    PLSRegression(n_components={"_range_": [5, 25, 5]}),
]
# Expands to 5 variants: n_components = 5, 10, 15, 20, 25

The three arguments are [start, stop, step]. The range is inclusive on both ends. If the step is omitted, it defaults to 1.

Other Generator Keywords

nirs4all provides several additional generator keywords for different search patterns. Each is covered briefly here; see Generator keywords: choices, combinations and their exact outputs for full syntax and examples.

_log_range_ – logarithmic sweep

Spaces values on a logarithmic scale. Ideal for regularisation parameters and learning rates that span several orders of magnitude:

{"alpha": {"_log_range_": [1e-4, 1e0, 5]}}
# Values: 0.0001, 0.001, 0.01, 0.1, 1.0

_grid_ – Cartesian product of parameters

Generates every combination of the specified parameter values for a single step. This is the classic grid search:

{"_grid_": {"n_components": [5, 10, 15], "scale": [True, False]},
 "model": PLSRegression}
# 6 variants: 3 values x 2 values

_cartesian_ – Cartesian product of pipeline stages

Like _grid_, but operates across pipeline stages rather than parameters of a single step:

{"_cartesian_": [
    {"_or_": [SNV(), MSC()]},
    {"_or_": [PLSRegression(10), RandomForestRegressor()]},
]}
# 4 variants: 2 preprocessors x 2 models

_zip_ – parallel paired iteration

Pairs values element-by-element. The lists must have the same length:

{"_zip_": {"n_components": [5, 10, 15], "scale": [True, False, True]}}
# 3 variants: (5, True), (10, False), (15, True)

_chain_ – sequential ordered choices

Like _or_, but guarantees evaluation in the order listed. Useful when order matters for reporting:

{"_chain_": [config_a, config_b, config_c]}
# 3 variants, evaluated in that exact order

_sample_ – random distribution sampling

Samples values from a statistical distribution. Useful for random search in large parameter spaces:

{"alpha": {"_sample_": {
    "distribution": "log_uniform",
    "from": 1e-4, "to": 1e-1, "num": 20,
}}}
# 20 randomly sampled values between 0.0001 and 0.1

Combining Generators

Generators compose naturally. Place multiple generators in the same pipeline, and nirs4all generates the Cartesian product of all of them:

pipeline = [
    {"_or_": [SNV(), MSC(), Detrend()]},     # 3 alternatives
    PLSRegression(n_components={"_range_": [5, 15, 5]}),  # 3 values
]
# 3 x 3 = 9 total variants

You can also control the combination logic with modifier keywords like pick (select a subset), count (limit total variants), and arrange (control combination strategy). See Generator keywords: choices, combinations and their exact outputs for the full modifier reference.

Parallel Execution

When you have many variants, running them in parallel can significantly reduce total time. Use the n_jobs parameter:

result = nirs4all.run(
    pipeline=pipeline,
    dataset=dataset,
    n_jobs=-1,   # use all CPU cores
)

Each variant runs independently, using joblib for process-based parallelism.

Viewing Results

After execution, the RunResult gives you access to ranked results:

# Best overall
print(result.best_score)

# Top 5 variants
for entry in result.top(5):
    print(entry["pipeline_name"], entry["rmse"])

result.top(n) returns the N best-performing variants, ranked by the primary metric (RMSECV for regression, balanced accuracy for classification).

Generators vs. Branching

Generators and branching (Branching and Merging) both let you explore multiple strategies, but they serve different purposes:

Aspect

Generators

Branching

Goal

Compare alternatives, pick the best

Combine results (stacking, merge)

Independence

Variants run independently

Branches share state and folds

Output

Ranked list of configurations

Merged predictions or features

Use generators when you are searching for the best configuration. Use branching when you want to combine multiple models or preprocessing strategies into one prediction.

Next Steps