nirs4all.api.run module

Module-level run() function for nirs4all.

This module provides the primary entry point for training ML pipelines on NIRS data. It wraps PipelineRunner.run() with a simpler, more ergonomic interface.

Example

>>> import nirs4all
>>> result = nirs4all.run(
...     pipeline=[MinMaxScaler(), PLSRegression(10)],
...     dataset="sample_data/regression",
...     verbose=1
... )
>>> print(f"Best RMSE: {result.best_rmse:.4f}")

Execute a training pipeline on a dataset.

This is the primary entry point for training ML pipelines on NIRS data. It provides a simpler interface than creating PipelineRunner and config objects directly.

Parameters:

pipeline –
Pipeline definition. Can be: - List of steps (most common): [MinMaxScaler(), PLSRegression(10)] - Dict with steps: {"steps": [...], "name": "my_pipeline"}

or the public wrapper alias {"pipeline": [...], "name": "my_pipeline"}
- Path to YAML/JSON config file: "configs/my_pipeline.yaml"
- PipelineConfigs object (backward compatibility)
- List of pipelines: [pipeline1, pipeline2, ...] - each pipeline is executed independently (cartesian product with datasets)
dataset –
Dataset definition. Can be: - Path to data folder: "sample_data/regression" - Numpy arrays: (X, y) or X alone - Dict with arrays: {"X": X, "y": y, "metadata": meta} - SpectroDataset instance - IO DataProvider: PLAN executes once before cross-validation.

A generate_view callback can then provide attested raw train/validation/refit views for supported splitters and concrete model or per-source branch shapes. This bounded DAG-ML profile requires refit=True and save_artifacts=False; it also supports N4M random-search tuning and an isolated Python worker. See the multimodal data guide for the exact supported shapes.
- List of SpectroDataset instances (multi-dataset)
- DatasetConfigs object (backward compatibility)
- List of datasets: [dataset1, dataset2, ...] - each dataset is used with each pipeline (cartesian product)
name – Optional pipeline name for identification and logging. If not provided, a name will be generated.
session – Optional Session object for resource reuse across multiple runs. When provided, shares workspace and configuration. Omitted runner shortcuts inherit the Session’s configuration; explicit values (including False or None) override it.
verbose – Verbosity level (0=quiet, 1=info, 2=debug, 3=trace). Default: 1
save_artifacts – Whether to save binary artifacts (models, transformers). Default: True
save_charts – Whether to save charts and visual outputs. When omitted, defaults to False for portable native execution and True for the general DAG-ML and explicit legacy execution paths.
plots_visible – Whether to display plots interactively. Default: False
random_state – Random seed for reproducibility. Default: None (no seeding)
refit –
Refit configuration. After cross-validation selects the winning pipeline variant(s), retrain on the full training set. - True: Refit top 1 by RMSECV (default). - False or None: Disable refit. - dict: Single criterion, e.g. {"top_k": 3, "ranking": "mean_val"}. - list[dict]: Multiple criteria for union selection, e.g.

[{"top_k": 3, "ranking": "rmsecv"}, {"top_k": 1, "ranking": "mean_val"}].

Ranking methods: "rmsecv" (OOF concatenated val score), "mean_val" (mean of individual fold val scores).
cache – Optional CacheConfig for step-level caching. - None: Use default CacheConfig (step cache OFF, CoW snapshots ON). - CacheConfig(step_cache_enabled=True): Enable step caching.
project – Optional project name to tag the run with. If the project does not exist yet it will be created automatically.
report_naming – Naming convention for metrics in reports and summaries. - "nirs" (default): Chemometrics terminology (RMSECV, RMSEP, etc.) - "ml": Machine learning terminology (CV_Score, Test_Score, etc.) - "auto": Auto-detect based on context (defaults to “nirs”) Affects column headers in final summary tables. Internal variable names use ML conventions regardless of this setting.
engine – Execution backend selector. Without an explicit argument or N4A_ENGINE, capabilities are checked before execution: portable Methods Archive V2 requests use "native"; general Python requests use "dag-ml". An explicit "native" is strict and never retries through another engine. result.execution_engine reports the actual backend. Explicit "legacy" runs the pure-Python orchestrator as a compatibility path. Explicit "dual" runs a bounded, native-first parity oracle for exact NumPy (X, y) regression, KFold(shuffle=False), and one PLSRegression model. It validates native OOF evidence before invoking the temporary legacy oracle, compares ledger-governed scores/predictions, removes the legacy workspace, and returns the native result. Every other dual shape fails closed before execution. Native engine overrides remain available through $N4A_ENGINE; legacy rollback always requires the explicit public engine="legacy" selector.
native_profile – Explicit closed native profile. "n4m.pls_role_pipeline.v1" supports PLS scale and n_components controls for train/REFIT, and the model step’s native finetune_params. Requires an explicit standalone engine="native" run. Omitting it preserves the historical portable PLS and general DAG profiles.
tuning – Typed native tuning specification for the currently supported DAG-ML subset. With engine="dag-ml", this supports explicit array datasets, a single estimator or linear sklearn-like transformer→estimator chain, Optuna/n4m-compatible space, required score_data for objective scoring, optional winner projection, workspace persistence/resume, and optional conformal calibration. Broader DAG branch/merge graphs, dataset loaders, arbitrary structural model selection and legacy execution remain fail-closed. Deterministic model-local finetune_params grids are still the native path for the older graph-generation subset.
calibration – Optional top-level conformal calibration payload for the currently supported native tuning subset. This is equivalent to tuning["calibration"] and requires run(tuning=..., engine="dag-ml"). Calibration evidence is derived from tuning["winner"]; arbitrary legacy runs and payloads that supply their own calibration_data remain fail-closed.
results_path – Native results output root (dag-ml engine only, P3 Slice 2b-i; OFF by default). When given, the dag-ml run ADDITIONALLY writes a native results directory <results_path>/<run_id>/ (manifest.json + the verbatim score_set.json + predictions.parquet); $N4A_NATIVE_RESULTS enables it env-only, defaulting to ./nirs4all_results/<run_id>/. None + unset env → nothing is written and the run is behaviorally identical to today. The legacy SQLite+Parquet+joblib workspace is untouched; an explicit results_path is threaded as a named run() parameter (not a runner_kwarg), so it bypasses the dag-ml runner_kwarg allowlist. It has no effect on engine="legacy".
allow_fallback – Frozen compatibility parameter. False is required for native engines; True is rejected before input inspection. Select engine="legacy" explicitly to use the rollback lane (where this parameter has no effect).
**runner_kwargs – Additional PipelineRunner parameters. See PipelineRunner.__init__ for full list. Common options: - workspace_path: Workspace root directory - continue_on_error: Whether to continue on step failures - show_spinner: Whether to show progress spinners - log_file: Whether to write logs to disk - log_format: Output format (“pretty”, “minimal”, “json”) - show_progress_bar: Whether to show progress bars - max_generation_count: Max pipeline combinations (for generators)

Returns:

predictions: Predictions object with all pipeline results
- per_dataset: Dictionary with per-dataset execution details
- best: Best prediction entry (convenience accessor)
- best_score: Best model’s primary test score
- best_rmse, best_r2, best_accuracy: Score shortcuts

Use result.top(n=5) to get top N predictions, or result.export("path.n4a") to export the best model. Results trained from generated views cannot yet export a replayable model. When run(tuning=..., calibration=...) or nested tuning["calibration"] is supplied, returns a TunedSingleEstimatorConformalResult containing both the tuned run result and the calibrated conformal calibrated result.

Return type:

RunResult containing

Raises:

ValueError – If pipeline or dataset format is invalid.
FileNotFoundError – If pipeline config or dataset path doesn’t exist.
DualRunUnsupported – If the explicit dual request, native capability, packaged ledger, or concrete OOF evidence is outside the bounded oracle contract.
DualRunMismatchError – If the native and explicit temporary legacy legs differ outside the ledger tolerances.

Examples

Simple usage with list of steps:

>>> import nirs4all
>>> from sklearn.preprocessing import MinMaxScaler
>>> from sklearn.cross_decomposition import PLSRegression
>>>
>>> result = nirs4all.run(
...     pipeline=[MinMaxScaler(), PLSRegression(10)],
...     dataset="sample_data/regression",
...     verbose=1
... )
>>> print(f"Best RMSE: {result.best_rmse:.4f}")

With cross-validation and multiple models:

>>> from sklearn.model_selection import ShuffleSplit
>>>
>>> result = nirs4all.run(
...     pipeline=[
...         MinMaxScaler(),
...         ShuffleSplit(n_splits=3),
...         {"model": PLSRegression(10)}
...     ],
...     dataset="sample_data/regression",
...     name="PLS_experiment",
...     verbose=2,
...     save_artifacts=True
... )

Multiple pipelines executed independently:

>>> pipeline_pls = [MinMaxScaler(), PLSRegression(10)]
>>> pipeline_rf = [StandardScaler(), RandomForestRegressor()]
>>>
>>> result = nirs4all.run(
...     pipeline=[pipeline_pls, pipeline_rf],  # Two independent pipelines
...     dataset="sample_data/regression",
...     verbose=1
... )
>>> print(f"Total configs: {result.num_predictions}")

Cartesian product of pipelines × datasets:

>>> pipelines = [pipeline1, pipeline2, pipeline3]
>>> datasets = [dataset_a, dataset_b]
>>>
>>> # Runs 6 combinations: p1×da, p1×db, p2×da, p2×db, p3×da, p3×db
>>> result = nirs4all.run(
...     pipeline=pipelines,
...     dataset=datasets,
...     verbose=1
... )

Using a session for multiple runs:

>>> with nirs4all.session(verbose=1) as s:
...     r1 = nirs4all.run(pipeline1, data, session=s)
...     r2 = nirs4all.run(pipeline2, data, session=s)
...     print(f"Pipeline 1: {r1.best_score:.4f}")
...     print(f"Pipeline 2: {r2.best_score:.4f}")

Export the best model:

>>> result = nirs4all.run(pipeline, dataset)
>>> result.export("exports/best_model.n4a")