nirs4all.pipeline.dagml_bridge module

nirs4all pipeline → dag-ml DSL frontend (migration spike, compile-only).

First slice of the #1 migration gap (dag-ml/docs/migration-nirs4all/): dag-ml’s compiler ingests serialized nirs4all-style compat DSL JSON, but nirs4all has no serializer from a live pipeline (operator instances) to that DSL. This module provides it for the linear transform → y_processing → splitter → model shape — the parity gate-zero slice (baseline_vertical_slice).

Scope (deliberately narrow for the spike):

• Supported: bare transformer/splitter instances → {"class", "params"} (dag-ml infers transform-vs-splitter from the class; splitters become campaign controller calls, not graph nodes); {"y_processing": op} → y_transform; {"model": op} → model.

• Not yet: branch / merge / tag / exclude / generators / augmentation / multi-source — these raise NotImplementedError naming the offending step, to be filled in against dag-ml/docs/design/DSL_NIRS4ALL_PARITY.md.

dag-ml is a CORE dependency since the ADR-17 cutover (the default engine); the import is still guarded so this module imports cleanly even if a broken wheel lacks the native backend. This is compile-only (DSL lowering); execution via host controllers is a later migration phase.

nirs4all.pipeline.dagml_bridge.compile_with_dagml(pipeline: list[Any], dsl_id: str = 'nirs4all-pipeline') → Any

Lower a nirs4all pipeline and compile it to a dag-ml CompiledPipelineArtifact.

Raises:

• ImportError – if the dag-ml core dependency is somehow missing (pip install nirs4all).

• NotImplementedError – if the pipeline uses an unsupported construct.

nirs4all.pipeline.dagml_bridge.constrained_operator_survivor_sequences(generator_node: dict[str, Any]) → list[list[Any]]

The PRUNED survivor SEQUENCES of a constrained operator generator (ADR-17 1a + 1b-cartesian).

Expands the constrained _or_-pick / _cartesian_ node through nirs4all’s own expand_spec — the SAME constraint source of truth both engines use (and the engine-agnostic survivor lock in test_generators_conformance_extra) — and normalizes each variant to a LIST of operator instances (a single-op variant becomes a one-element list). dag-ml’s native operator-SELECT later compiles ONE survivor branch per sequence and, applying pick + _mutex_/_requires_/_exclude_ during its own sequence-build (prune_sequences_by_constraints), produces the byte-identical pruned set, so the per-variant variant_label fingerprints agree by content (see operator_choice_variant_label()). The survivor ORDER follows expand_spec (legacy OrStrategy/CartesianStrategy input order), which matches dag-ml’s stable enumerate-retain order — so the {variant_label → config_name} map (built in expand order) and dag-ml’s reports align by content, not position.

nirs4all.pipeline.dagml_bridge.is_grid_param_generator_spec(grid_map: Any) → bool

True ONLY for a _grid_ over MODEL params the native dag-ml Grid generator can lower at parity.

A step-level {"_grid_": {param: [values], …}, "model": M} lowers to one native dag-ml Grid param generator (_grid_param_generator()). dag-ml’s Grid is a flat Cartesian product of each param’s PLAIN value list — its PipelineDslGeneratorValue is an untagged literal, so it does NOT recursively expand a nested generator value. nirs4all’s GridStrategy DOES expand a nested-generator value (n_components: {"_range_": …}) BEFORE the product, so only a grid whose every value is a list (or bare scalar) of PLAIN, finite, JSON-native scalars — with NO nested dict (a nested generator / object value) and NO nested-generator list element — is representable natively and produces the identical variant set. ALL of:

• a non-empty dict of str → values (the bare _grid_ param-map, NO count/_seed_ modifier — those reach here only when the caller already split the keyword off, and a modifier changes the variant set vs. expand_spec, so it is excluded by the caller); AND

• every value is a list of plain scalars (or a single plain scalar), each scalar finite and JSON-native (bool/int/float/str/None) — NO dict, NO list-of-dicts (a nested generator the native Grid cannot expand), NO numpy / non-finite / non-JSON value; AND

• the param insertion order is already alphabetical (list(grid_map) == sorted(grid_map)): dag-ml’s Grid nests params in BTreeMap (sorted) order while nirs4all nests in INSERTION order, so a non-alphabetical grid would produce a different variant ENUMERATION order than expand_spec — keep it on the Python path so the per-variant config map stays aligned.

Any other shape (a nested-generator value, a modifier-bearing or non-alphabetical grid, a non-list non-scalar value, a non-JSON / non-finite scalar) → False (stays on the correct Python expand_spec path, still dag-ml-native via that route). When in doubt, this returns False.

nirs4all.pipeline.dagml_bridge.is_param_generator_spec(spec: Any) → bool

True ONLY for the exact _range_ / _log_range_ list forms lowered natively at proven parity.

Conservative by design: a single key (_range_ or _log_range_) whose value is a list of exactly three numbers — {"_range_": [a, b, s]} or {"_log_range_": [a, b, n]}. dag-ml’s native log_range now round-trips through build_execution_plan (the float-label fingerprint drift is fixed by canonical_generator_number at value generation, dag-ml 2a77a7f), and its base-10 geometric expansion matches nirs4all’s _log_range_ exactly, so the list form is native. Everything else still falls back to the correct Python expand_spec path:

• _grid_ — value-level lowering is not proven equivalent to step-level grid expansion;

• the dict {"from"/"to"/...} form, a wrong-length list, or any modifier key (count/_seed_) — would change the variant set versus expand_spec.

Other keys in the dict (e.g. a model sibling) are handled by the caller, not here.

nirs4all.pipeline.dagml_bridge.lower_constrained_operator_pipeline(steps: list[Any], dsl_id: str = 'nirs4all-pipeline') → dict[str, Any]

Lower a CONSTRAINED operator-generator pipeline to ONE NATIVE Generator DSL step (ADR-17 item 5 slice B).

steps is the splitter-free pipeline [<constrained generator>, …downstream…] (the predicate _is_constrained_operator_generator() admits exactly one constrained _or_-pick / _cartesian_ generator plus one concrete downstream model, optionally a y_processing). The generator lowers to ONE canonical {"kind": "generator", …} step that carries its operator OPTIONS as branches/stages, its pick/arrange/then_*/count modifiers, its lowered operator-content constraints, and the downstream model (+ any y_processing) as the tail — and dag-ml PRUNES (the CATCH-22 fix). dag-ml applies pick + the _mutex_/_requires_/_exclude_ prune to the SAME survivor set expand_spec produced (proven byte-identical in ADR-17 1a), appends the tail so every survivor terminates in the model, and stamps each per-variant report with the multi-op variant_label fingerprint of [<survivor ops>, <tail>] — the EXACT sub-sequence operator_sequence_variant_label() recomputes host-side.

This REPLACES the previous survivor PRE-EXPANSION (constrained_operator_survivor_sequences → expand_spec → one _or_ branch per survivor): with the native tail field on PipelineDslGeneratorStep and compile_operator_variant_models reaching the prune for a model-terminated generator, the host no longer pre-prunes — it hands the constraints + pick to dag-ml. The {variant_label → config_name} map (built in dagml.result from expand_spec) still aligns because dag-ml prunes the SAME survivor set, in the SAME legacy order. A non-bare / None / non-routable choice is rejected upstream by the predicate; an option still lowers each transform via the strict bare-operator path so a stray non-JSON param fails loud (→ Python expand), and a constraint ref that names no option (a malformed constraint) likewise demotes.

nirs4all.pipeline.dagml_bridge.operator_choice_variant_label(choice: Any, downstream_steps: list[Any]) → str

The AUTHORITATIVE variant_label (hex sha256) for ONE bare-operator _or_ choice + its tail.

Maps a per-variant dag-ml report back to its operator-choice config (#23 Phase 7). The label is a cross-language CONTENT fingerprint dag-ml stamps on every operator-SELECT report; the host MUST call the dag-ml PyO3 helper (dag_ml.canonical_operator_variant_label) — NOT a pure-Python json.dumps + sha256 — because Python’s float formatting diverges from Rust’s (1e-05 / 1e-7 / 1-ULP shortest decimals, all common NIRS params), which would SILENTLY mis-key the map. Sharing the one Rust codepath makes the host label byte-identical to the report label by construction.

The fingerprint is computed over the FULL LOWERED choice sub-sequence — dag-ml’s lower_operator_variant_model activates the chosen _or_ branch PLUS every downstream step the branch flows into (the concrete model, any y_processing), so choice.steps is the choice’s transform step FOLLOWED by the downstream steps. The host therefore builds steps_json = [<choice transform>] + [<lowered downstream steps>] in the PipelineDslStep tagged shape ({"kind", "id", "operator", "params"}) — lowering each downstream step from its RAW form via _label_step_from_raw() (NOT through _step_to_dsl()’s default=repr params, which would stringify a non-JSON downstream param BEFORE strict checking). The id is ignored by the canonicalization (it reads only kind / operator class / params), but the OPERATOR SHAPE matters: a bare-operator transform choice keeps the object operator {"class": FQN}, while a model keeps the BARE-STRING operator FQN — exactly what dag-ml’s lowering produces. downstream_steps is the lowered pipeline tail after the generator (no splitter, no None — already split off by the caller).

The ENTIRE label payload is STRICTLY sanitized (chosen branch + downstream model + y_processing): every step’s operator + params is recursively coerced (numpy .item() / .tolist()) and any NaN / Inf / non-str-key / non-JSON value is REJECTED via _strict_json_safe() (NOT default=repr, which would emit "np.int64(5)" strings that break byte-identity, and NOT allow_nan=True, which would pass a non-finite the PyO3 helper then rejects mid-run). ANY failure to construct or fingerprint the label — a non-JSON / non-finite param OR the PyO3 helper raising — is converted to DagMlUnsupported so the inner Python-expand fallback fires instead of crashing.

A MULTI-STEP list choice ([SNV, Detrend], ADR-17 item 5 slice D) is its own flat operator SEQUENCE, so the label is over [<each transform>, <downstream>] — _operator_choice_operators() unpacks the choice into its operator list, matching dag-ml’s multi-transform branch survivor.

nirs4all.pipeline.dagml_bridge.operator_sequence_variant_label(operators: list[Any], downstream_steps: list[Any]) → str

The AUTHORITATIVE variant_label (hex sha256) for a MULTI-OPERATOR survivor sequence + its tail.

The constrained-generator generalization of operator_choice_variant_label(): a constrained _or_-pick / _cartesian_ survivor is a SEQUENCE of operators ([SNV, Detrend]), not a single choice. dag-ml lowers each survivor branch to [<transform op0>, …, <transform opN>] + downstream and fingerprints THAT whole sub-sequence (operator_variant_label(choice.steps)), so the host builds the SAME steps_json = [<each transform>, <lowered downstream>] and fingerprints it via the SAME Rust canonicalization (dag_ml.canonical_operator_variant_label) — byte-identical to the report label by construction. Each operator’s params are STRICTLY sanitized (_canonical_label_params(), NaN/Inf/non-JSON/non-str-key REJECTED → DagMlUnsupported → the inner Python-expand fallback); the downstream tail is lowered from its RAW form (_label_step_from_raw()). A single-element operators reproduces the flat-single label exactly.

nirs4all.pipeline.dagml_bridge.step_has_native_grid(step: Any) → bool

True iff a model step carries a step-level _grid_ the native dag-ml Grid can lower at parity.

A clean native grid is {"_grid_": <native param-map>, "model": M[, plain params]}: a step-level _grid_ whose param-map passes is_grid_param_generator_spec() AND NO variant-set-changing grid modifier sibling (count / _seed_) — those would subsample/reseed the grid, which the native Grid does not reproduce, so a modifier-bearing grid stays on the Python expand path. The single recognition point shared by the bridge lowering (_step_to_dsl()), the generation-kind classifier, and the plain-param application, so the three never disagree on which grids go native.