nirs4all.pipeline.execution.orchestrator module

Pipeline orchestrator for coordinating multiple pipeline executions.

class nirs4all.pipeline.execution.orchestrator.PipelineOrchestrator(workspace_path: str | Path | None = None, verbose: int = 0, mode: str = 'train', save_artifacts: bool = True, save_charts: bool = True, enable_tab_reports: bool = True, continue_on_error: bool = False, show_spinner: bool = True, keep_datasets: bool = False, max_preprocessed_snapshots_per_dataset: int = 3, plots_visible: bool = False, random_state: int | None = None, n_jobs: int = 1, report_naming: Literal['nirs', 'ml', 'auto'] = 'nirs', should_stop: Callable[[], bool] | None = None)[source]

Bases: object

Orchestrates execution of multiple pipelines across multiple datasets.

High-level coordinator that manages: - Workspace initialization via WorkspaceStore - Global predictions aggregation - Best results reporting - Dataset/pipeline normalization

Variables:

  • workspace_path – Root workspace directory

  • store – WorkspaceStore for SQLite-backed persistence

  • verbose – Verbosity level

  • mode – Execution mode (train/predict/explain)

  • save_artifacts – Whether to save binary artifacts

  • save_charts – Whether to save charts and visual outputs

  • enable_tab_reports – Whether to generate tab reports

  • keep_datasets – Whether to keep dataset snapshots

  • plots_visible – Whether to display plots

execute(pipeline: PipelineConfigs | list[Any] | dict | str, dataset: DatasetConfigs | SpectroDataset | list[SpectroDataset] | ndarray | tuple[ndarray, ...] | dict | list[dict] | str | list[str], pipeline_name: str = '', dataset_name: str = 'dataset', max_generation_count: int = 10000, artifact_loader: Any = None, target_model: dict[str, Any] | None = None, explainer: Any = None, refit: bool | dict[str, Any] | list[dict[str, Any]] | None = True, store_run_id: str | None = None, manage_store_run: bool = True) → tuple[Predictions, dict[str, Any]][source]

Execute pipeline configurations on dataset configurations.

Parameters:

  • pipeline – Pipeline definition (PipelineConfigs, list[steps], Dict, or file path)

  • dataset – Dataset definition (DatasetConfigs, SpectroDataset, numpy arrays, Dict, or file path)

  • pipeline_name – Optional name for the pipeline

  • dataset_name – Optional name for array-based datasets

  • max_generation_count – Maximum number of pipeline combinations to generate

  • artifact_loader – ArtifactLoader for predict/explain modes

  • target_model – Target model for predict/explain modes

  • explainer – Explainer instance for explain mode

  • refit – Refit configuration. - True: Enable refit (retrain winning model on full training set, default). - False or None: Disable refit (legacy behavior). - dict: Refit options (reserved for future use).

  • store_run_id – Optional existing store run ID to join. When provided, skips begin_run() and uses this ID directly.

  • manage_store_run – Whether to manage the store run lifecycle (begin/complete/fail). Set to False when the caller manages the run lifecycle externally (e.g. multi-pipeline batch).

Returns:

Tuple of (run_predictions, dataset_predictions)

exception nirs4all.pipeline.execution.orchestrator.RunCancelledError[source]

Bases: RuntimeError

Raised when a run is cancelled via the should_stop hook.

The orchestrator checks the hook at cooperative boundaries (before each dataset, before each pipeline variant, before the refit phase) and aborts with this exception; the store run is marked failed with the cancellation message by the standard failure path.