Pipeline Workflow

The workflow layer turns intent into controlled execution. It takes your pipeline definition and datasets, expands variants from generators, executes steps with context-aware controller logic, and records outputs in a reproducible store.

Workflow Mission

Transform a high-level pipeline definition into:

  1. Expanded variants from generators (_or_, _range_)

  2. Controlled execution with context-aware step logic

  3. Traceable outputs with fold/partition/branch context

  4. Durable persistence in workspace store

Primary Execution Stack

The execution engine is layered for modularity:

Component

Role

PipelineRunner

Top-level facade for run/predict/explain/retrain

PipelineOrchestrator

Coordinates Cartesian product of datasets × pipeline configs

PipelineExecutor

Executes a single pipeline on a single dataset

StepRunner

Parses, routes, and executes individual steps

StepParser

Normalizes raw step syntax into ParsedStep

ControllerRouter

Selects the matching controller by priority

OperatorController

Behavior adapter that executes the step

End-to-End Training Flow

Here’s what happens when you call nirs4all.run():

        sequenceDiagram
    participant U as User
    participant R as PipelineRunner
    participant O as PipelineOrchestrator
    participant E as PipelineExecutor
    participant S as StepRunner
    participant C as Controller
    participant W as WorkspaceStore

    U->>R: run(pipeline, dataset)
    R->>O: execute(...)
    O->>W: begin_run(...)
    O->>E: execute steps per dataset/config
    E->>W: begin_pipeline(...)
    loop each step
        E->>S: execute(step, context)
        S->>S: parse step
        S->>S: route controller
        S->>C: execute(...)
        C-->>S: context updates + artifacts
        S-->>E: StepResult
    end
    E->>W: save chains, predictions, artifact refs
    E->>W: complete_pipeline(...)
    O->>W: complete_run(...)
    O-->>R: run predictions + per-dataset outputs
    R-->>U: RunResult

Pipeline Normalization Phase

Before execution, PipelineConfigs normalizes your pipeline definition. It accepts:

  • Lists of step objects

  • Dicts with pipeline keys

  • JSON/YAML file paths

Generator expansion happens here:

  • _or_ (variant alternatives) → multiple pipeline configs

  • _range_ (parameter sweeps) → grid of parameter values

Each expanded variant receives a generated identity.

Example

pipeline = [
    {"_or_": [SNV(), MSC(), Detrend()]},
    PLSRegression(n_components={"_range_": [5, 10, 15]}),
]

# Expands to 9 variants:
# SNV + PLS(5), SNV + PLS(10), SNV + PLS(15)
# MSC + PLS(5), MSC + PLS(10), MSC + PLS(15)
# Detrend + PLS(5), Detrend + PLS(10), Detrend + PLS(15)

Dataset Normalization Phase

Dataset inputs are normalized via DatasetConfigs, which handles:

  • Path-based loading ("sample_data/regression")

  • Config parsing (dicts with loader settings)

  • Direct SpectroDataset instances

Once normalized, the orchestrator operates on consistent dataset objects.

Orchestration Topology

The orchestrator executes a Cartesian product of each dataset with each expanded pipeline config.

This allows sweep-style experiments naturally:

  • 3 datasets × 20 pipeline variants = 60 executions

During this phase, the orchestrator:

  • Starts run-level persistence

  • Initializes shared runtime components

  • Manages post-run summaries

Context Initialization

The executor initializes an ExecutionContext containing:

Component

Purpose

DataSelector

Immutable data selection parameters (partition, processing, layout, etc.)

PipelineState

Mutable state (current step number, y_processing name, execution mode)

StepMetadata

Controller coordination data (keyword, operator type)

Custom dict

Controller-specific state (e.g., branch paths)

Aggregate column

For prediction aggregation policy

RuntimeContext Role

RuntimeContext is the infrastructure context. It carries runtime services rather than data state:

Service

Purpose

store

WorkspaceStore for SQLite-backed persistence

artifact_registry

Artifact management during training

artifact_loader / artifact_provider

For predict/explain modes

step_runner

For executing sub-steps (e.g., inside branches)

run_id, pipeline_id, pipeline_uid, pipeline_name

Identifiers

trace_recorder

Recording execution traces

cache_config / step_cache

Step-level caching

retrain_config

Retrain mode control

phase

Current execution phase (CV or REFIT)

This separates data semantics from infrastructure wiring.

Step Processing Pipeline

Each step goes through three mini-phases:

  1. Parse — convert syntax into ParsedStep

  2. Route — select the best matching controller by priority

  3. Execute — apply controller logic with context and dataset

Example: Transform Step

# Pipeline definition
pipeline = [MinMaxScaler()]

# Step processing:
# 1. Parse: MinMaxScaler() → ParsedStep(operator=MinMaxScaler(), keyword=None)
# 2. Route: Check all controllers → TransformerMixinController matches (priority 10)
# 3. Execute: Controller calls fit_transform on training data, stores artifact

Keyword Semantics

Pipeline keywords influence controller matching. Key keywords:

Keyword

Purpose

model

Define model step

y_processing

Target scaling

tag

Mark samples (non-removal)

exclude

Exclude samples from training

branch

Parallel/separation branches

merge

Combine branch outputs

_or_

Generator for variant alternatives

_range_

Parameter sweep generator

Additional internal keywords: feature_augmentation, sample_augmentation, auto_transfer_preprocessing, concat_transform.

Controller Routing Logic

Routing is simple and deterministic:

  1. Evaluate all registered controllers

  2. Keep those whose matches() returns True

  3. Sort by numeric priority (lower wins)

  4. Instantiate the first match

Priority Strategy

Lower priority values win earlier. Use low priority for specific behaviors and higher priority for generic ones.

Example: TensorFlowModelController (priority 4) wins over SklearnModelController (priority 6) for TensorFlow objects.

Execution Modes

NIRS4ALL supports four execution modes:

Train Mode

The richest lifecycle:

  • Split controllers define folds

  • Transform controllers fit and transform

  • Model controllers train and score

  • Predictions are stored with fold/partition context

  • Traces capture step-level artifact references

Predict Mode

Replays persisted artifacts:

  • Training-only steps are skipped

  • Controllers load stored artifacts

  • Transformations replay in pipeline order

  • Model artifacts generate final predictions

Explain Mode

Resembles predict mode but:

  • Collects explanation artifacts (SHAP values)

  • Generates SHAP charts

  • Returns feature importance

Refit Mode

Retrains the winning configuration:

  • Runs after CV evaluation

  • Trains on full training partition (excluding holdout/test)

  • Produces deployment-ready final model

Out-of-Fold (OOF) Training and Prediction Lifecycle

OOF is the core mechanism behind honest validation and stacking-safe features. Each prediction is produced by a model that did not train on that row.

        flowchart TD
    A[Split definitions in dataset] --> B[Fold 0 train]
    A --> C[Fold 1 train]
    A --> D[Fold K train]
    B --> E[Val preds fold 0]
    C --> F[Val preds fold 1]
    D --> G[Val preds fold K]
    E --> H[OOF assembly]
    F --> H
    G --> H
    H --> I[Ranking, diagnostics, stacking features]

The OOF contract:

  • Each row linked to sample identity

  • Fold identity persisted for auditability

  • Partition and branch context retained

  • Model/preprocessing signatures retained

This allows:

  • Fair variant comparison (no leakage)

  • Leak-safe meta-model features (for stacking)

  • Fold-specific instability diagnosis

Stacking Example

When building a meta-model (stacking), base model predictions for meta-model training must be OOF-generated to avoid target leakage.

Wrong: Use in-sample predictions from base models

  • Meta-model sees predictions from models trained on the same data

  • Optimistically biased

Right: Use OOF predictions from base models

  • Meta-model sees predictions from models that never saw those samples

  • Honest generalization

CV-to-Refit Lifecycle Pattern

A robust production pattern: train across folds first, then retrain the winner on full training data.

        sequenceDiagram
    participant O as Orchestrator
    participant CV as CV Execution
    participant S as Selector/Ranker
    participant R as Refit Executor
    participant W as Workspace

    O->>CV: run all candidates on folds
    CV->>W: store fold metrics + OOF + fold artifacts
    CV-->>S: candidate score table
    S-->>R: winner config id
    R->>R: train winner on full training partition
    R->>W: store refit artifact and deployment chain
    W-->>O: both CV history and final model available

Lifecycle steps:

  1. Phase A: Run candidate pipelines with fold-based evaluation. Persist fold metrics, artifacts, and OOF outputs.

  2. Phase B: Rank candidates using a predefined metric. Select the winner with deterministic tie-break rules.

  3. Phase C: Refit only the winning configuration on the full training partition (excluding holdout/test).

  4. Phase D: Persist refit artifacts as the deployment-default chain. Keep CV artifacts for audit.

Why this matters:

  • CV answers model-selection quality

  • Refit answers deployment-readiness

Conflating them creates either under-trained deployment artifacts or biased selection.

Retrain and Transfer Workflow

Retrain uses a source model/chain plus new data. Modes include:

  • Full retrain — retrain all steps from scratch

  • Transfer-style reuse — freeze preprocessing, retrain model

  • Fine-tune — load model weights and continue training

Per-step mode control determines whether to train or reuse artifacts.

Branching Workflow

Branching enables parallel subflows in two families:

Duplication Branches

All samples go to each branch (parallel preprocessing/modeling):

{"branch": [
    [SNV(), PLSRegression(10)],
    [MSC(), RandomForestRegressor()],
]}

Use case: Compare preprocessing chains, then merge for ranking.

Separation Branches

Samples are partitioned across branches (by metadata, tag, or source):

{"branch": {"by_metadata": "site"}}
{"branch": {"by_tag": "outlier_flag", "values": {"clean": False, "outliers": True}}}

Use case: Subgroup-specific models (e.g., one model per site).

Branch context is tracked in the execution context and becomes important for artifact and prediction resolution.

Merge Workflow

Merge operations reconcile branch outputs. Strategies:

Strategy

Use case

Branch type

predictions

Use OOF predictions as stacking features

Duplication branches

features

Use transformed features from branches

Duplication branches

concat

Reassemble samples in original order

Separation branches

Merge logic must handle asymmetry (shape differences, model availability differences). Merging is one of the highest-complexity areas.

Feature Augmentation Workflow

Feature augmentation orchestrates repeated preprocessing operations in extend, add, or replace modes.

  • Extend: Add new processing chains (grow 3D array)

  • Add: Add derived features to existing processing

  • Replace: Overwrite existing processing

This changes processing growth patterns and directly influences memory footprint.

Sample Augmentation Workflow

Sample augmentation generates synthetic variants in:

  • Standard count mode: Fixed number of augmented samples per base sample

  • Balancing-aware mode: Based on labels or metadata (e.g., oversample minority class)

Especially useful for imbalanced classification.

Trace and Chain Lifecycle

Execution trace captures step outcomes and artifacts. The chain builder converts trace data into replayable chains containing:

  • Ordered steps

  • Model step index

  • Fold artifact references

  • Shared preprocessing references

  • Branch/source metadata

Chains are first-class deployment and replay units.

Prediction Persistence Workflow

Prediction rows are persisted with rich metadata:

  • dataset_name, pipeline_id, pipeline_uid, pipeline_name

  • fold_id, partition

  • branch_id, branch_name

  • model_class, preprocessing_signature

  • Scalar metrics (rmse, r2, accuracy, etc.)

Array payloads (y_true, y_pred, y_proba) are stored in a dedicated array table.

This split keeps scalar query paths efficient while preserving detailed replay capability.

Workflow State Transitions

        stateDiagram-v2
    [*] --> Normalized
    Normalized --> RunningVariants
    RunningVariants --> RunningSteps
    RunningSteps --> Persisting
    Persisting --> Completed
    RunningSteps --> Failed
    Persisting --> Failed
    Failed --> [*]
    Completed --> [*]

Workflow Contracts

Key contracts that controllers and the execution engine must uphold:

  1. Step controllers must return context updates consistently

  2. Prediction mode must never assume training-time fitting

  3. Artifact naming and indexing must remain deterministic for replay

  4. Branch and fold semantics must be preserved through persistence

  5. Run metadata must reflect actual execution conditions

Next Steps

Now that you understand how pipelines execute, explore:

See also

Related Examples: