nirs4all.pipeline.dagml.native_results module

Native results persistence for the dag-ml backend (P3 Slice 2b-i, B-C HYBRID).

ADDITIVE, OFF-by-default, native-only on-disk results for run(engine="dag-ml"). The dag-ml path is in-memory and touches NO legacy workspace (no SQLite store, no ArrayStore); this writer PRESERVES that: it persists ONLY the things the dag-ml run already produced in memory (the native ScoreSet, the projected prediction rows, captured refit artifacts, and a run header) and NEVER imports/instantiates WorkspaceStore / ArrayStore.

Layout (one directory per run, default ./nirs4all_results/<run_id>/):

  • score_set.json — the dag-ml ScoreSet stored VERBATIM (the raw outcome["scores"] dict the projection consumed; for an operator sweep, the synthesized multi-variant ScoreSet _project_operator_sweep() builds and feeds the projection). The canonical native object is written AS-IS — never re-authored Python-side; the manifest records its content hash.

  • predictions.parquet — a columnar PROJECTION of the in-memory RunResult.predictions rows. A convenience view, NOT the source of truth. Arrays (y_true / y_pred / y_proba) are carried ONLY where the row has them (the direct-block rows); score-only rows store empty arrays + an arrays_present flag. SHAPE is carried (y_true_shape / y_pred_shape / y_proba_shape) so a multi-target row round-trips (the legacy portable parquet flattens without shape — incomplete for multi-target).

  • artifacts/ — the fitted REFIT model binaries (P3 Slice 2c-i). Each captured {estimator, y_transform} is joblib-serialized to artifacts/<node>/<variant>.joblib and recorded as a manifest artifacts[] ArtifactRef entry. ONLY the leakage-safe REFIT estimators are persisted (FIT_CV/OOF models never are). Both the in-process mechanism and the subprocess adapter capture refit models.

  • manifest.json — the run header (run_id, engine, versions, datasets, configs/variants, models, metric, task_type) + CAPABILITY FLAGS (has_model_artifacts true when any model artifact was captured + persisted; has_aggregate_predictions false for this slice) + the ScoreSet content hash + producer-node summaries + the ArtifactRef artifacts[] list + optional replay manifests for native composite exports + the manifest’s own schema_version.

nirs4all.pipeline.dagml.native_results.native_results_enabled(results_path: str | Path | None) → bool[source]

Whether the native results writer should fire for this run (OFF by default).

Enabled when EITHER an explicit results_path is given (the clean run() parameter, threaded past the empty _HONORED_RUNNER_KWARGS allowlist as a named arg) OR the N4A_NATIVE_RESULTS env var is set to a truthy value (the per-process override). With NEITHER, the dag-ml run stays 100% in-memory and writes nothing — behaviorally identical to today.

nirs4all.pipeline.dagml.native_results.read_native_results(run_dir: str | Path) → dict[str, Any][source]

Read a native results directory back into a Predictions-consumable form.

Returns {"manifest", "score_set", "predictions", "artifacts"} where predictions is a populated in-memory Predictions whose rows round-trip the writer’s projection (the per-sample arrays reshaped from their *_shape columns, so a multi-target row recovers its 2D shape), and artifacts is the list of rehydrated {artifact_id, estimator, y_transform, ...} model payloads (P3 Slice 2c-i; empty when the run has no model artifacts). VALIDATES the ScoreSet against the manifest’s recorded hash, and — for EACH model artifact — VERIFIES the on-disk bytes against the recorded content_fingerprint BEFORE joblib.load() (joblib.load executes code: verify-then-load on trusted input). A corrupt/edited score_set.json OR a tampered model artifact raises ValueError before any load.

nirs4all.pipeline.dagml.native_results.separation_replay_manifest(graph: dict[str, Any], metadata_key: str, artifacts: list[dict[str, Any]]) → dict[str, Any] | None[source]

Record the selected native artifact for every metadata-fanned model.

Operator SELECT leaves inactive choice nodes in the compiled union graph but refits only the winning choice of each metadata branch. Match graph nodes from the captured artifacts, never require artifacts from inactive choices.

nirs4all.pipeline.dagml.native_results.write_native_results(result: RunResult, score_set: dict[str, Any] | None, results_path: str | Path | None) → Path[source]

Write the native results directory for a dag-ml run; return the run directory.

Writes manifest.json + score_set.json (VERBATIM) + predictions.parquet + the artifacts/ model tree (the captured fitted REFIT estimators, P3 Slice 2c-i) under <root>/<run_id>/. Called ONLY when native_results_enabled() (OFF by default). NEVER touches the legacy workspace store. The fitted models are read from result._dagml_refit_artifacts (captured from the in-process store or the subprocess adapter); an empty list writes no artifacts/ payload and records has_model_artifacts:false.