nirs4all.pipeline.dagml.node_runner module

Node runner — execute one dag-ml NodeTask with real nirs4all operators + data.

This is the heart of the host-controller side of the migration: it replaces the shipped conformance adapter’s hash-synthesized features with the real SpectroDataset rows (via MaterializationResolver) and the real operator (via route_graph_node), mirroring the dag-ml NodeTask/NodeResult JSON contract exactly (verified against dag-ml/examples/adapters/sklearn_process_controller.py).

Frame contract (per phase): data_views is keyed by data:<input> with a partition — FIT_CV gets fold_train + a sibling …:validation (fold_validation) and predicts the validation samples (partition: validation); REFIT gets full_train, fits, persists a model artifact, predicts the train samples (partition: final); PREDICT reloads the artifact and predicts the predict view. The NodeTask carries node_plan (params, kind) but not the operator class, so the runner takes a node_lookup that maps node_id → the compiled graph node (which carries the operator) — both produced nirs4all side by build_dagml_plan.

Only model/tuner nodes produce predictions. Bound transform nodes fit their operators on the native data view and pass fitted chains through data-edge handles; unbound transforms use the legacy reconstructed chain at the model node. y_transform nodes remain floating target transforms applied by the model callback.

nirs4all.pipeline.dagml.node_runner.capture_torch_topology(training: Any, frames: list[dict[str, Any]], store: dict[Any, Any]) → dict[str, Any][source]

Retain the complete native-selected Torch closure before detach.

nirs4all.pipeline.dagml.node_runner.emit_named_refit_origin(task: dict[str, Any], estimator: Any, specification: dict[str, Any], fit_ids: list[str], *, graph_node: dict[str, Any], target_names: list[str]) → dict[str, Any][source]

Bind the actual completed named FIT to its immutable native REFIT origin.

nirs4all.pipeline.dagml.node_runner.run_meta_model_node(task: dict[str, Any], resolver: MaterializationResolver, node_lookup: Callable[[str], dict[str, Any]], model_store: MutableMapping[Any, Any], *, graph_metadata: dict[str, Any] | None = None) → dict[str, Any][source]

Execute a STACKING meta-model node from its base branches’ OOF + off-fold predictions (#10).

The meta-node is a model-kind node whose x is not data but the base branches’ predictions, delivered by dag-ml as prediction_inputs (Option A: each spec carries values aligned 1:1 with sample_ids). dag-ml delivers two delivery kinds, keyed by suffix (_ordered_oof_specs()):

  • "{producer}.{port}" — the base producer’s Validation OOF (the requires_oof edge refuses any train block, so it is leakage-safe). In FIT_CV this is inner-fold OOF over the outer training universe, and is what the meta-learner trains on.

  • "{producer}.{port}:outer" — the base producer’s outer-fold validation block. It is used only to evaluate the fitted meta-learner, never as a fit row.

  • "{producer}.{port}:refit" / ":predict" — the base producer’s off-fold block (REFIT held-out Test / PREDICT new-data Final, fold_id=None), delivered ONLY in REFIT/PREDICT and used ONLY to PREDICT the holdout, never to train.

The meta-feature matrix has one column block per base producer in deterministic producer order (sorted base key), aligned by sample_id, mirroring dag-ml’s join_oof_features.

  • FIT_CV (fold K): build X_meta from inner-fold OOF, fit the MetaModel on (X_meta, y_train), then predict the separate :outer matrix and emit the meta-model’s OOF (partition=validation, fold_id=foldK) + targets. The cross-fold OOF average of the meta producer is cv_best_score.

  • REFIT: fit + persist the meta-model on the FULL Validation OOF (all folds), then build the TEST meta-feature matrix from the base :refit off-fold inputs (the base models’ held-out Test predictions), predict the holdout, and emit a scored (test, fold_id=None) block + targets so dag-ml scores best_rmse. LEAKAGE INVARIANT: the meta-model is fit on Validation OOF ONLY (the :refit Test predictions never enter the fit); the Test meta-features come from the base models’ Test predictions, never their OOF/train.

nirs4all.pipeline.dagml.node_runner.run_model_node(task: dict[str, Any], resolver: MaterializationResolver, node_lookup: Callable[[str], dict[str, Any]], model_store: MutableMapping[Any, Any], edges: list[dict[str, Any]] | None = None, y_transform_node: dict[str, Any] | None = None, sample_metadata: dict[str, dict[str, Any]] | None = None, generated_views: GeneratedTaskViews | None = None, *, graph_metadata: dict[str, Any] | None = None) → dict[str, Any][source]

Execute a model-kind NodeTask with the real operator + real data; return a NodeResult.

When X transforms have native data bindings, the model consumes their fitted chain through a data-edge handle. Otherwise it reconstructs the upstream X-transform chain (e.g. SNV→PLS) into a sklearn Pipeline fitted on the model’s fold-train rows. A y_transform_node (a nirs4all y_processing step — a floating graph node with no edge to the model) is fit on fold-train y, applied before fitting, and inverse-transformed over the predictions, exactly reproducing nirs4all target scaling.

sample_metadata ({wire_id: {col: value}}) lets a separation-branch model honor its branch_view selector: train/predict ids are restricted to the partition, and the held-out TEST prediction (the refit best_rmse path) is likewise restricted to the branch’s partition.