nirs4all.pipeline.dagml.envelope module

Data-plan envelope + fold-set builders for the nirs4all → dag-ml(-data) bridge.

dag-ml-data owns the contract types and the fingerprint algorithm; this module only assembles the JSON inputs from a SpectroDataset + its IdentityMap and hands them to the dag_ml_data wheel, which computes every fingerprint internally and derives coordinator_relations from the SampleRelationTable. We never compute a fingerprint or hand-build a coordinator relation — that keeps the materialize-time fingerprint gate passing by construction.

Declares identity only, never X/y values:

• the DatasetSchema (one source, NIR spectra signal_1d → tabular_numeric),

• the DataPlan (materialize → adapt spectra.flatten → join to port:X),

• the SampleRelationTable (one row per observation; input field names origin_id / repetition_id / augmented / excluded).

Folds are a separate first-class contract (FoldSet), validated against the CV-universe relations by validate_fold_set_against_sample_relations — never carried in the envelope. That validator enforces a clean OOF partition (each sample validated exactly once): KFold satisfies it; ShuffleSplit does not (a known OOF-semantics gap flagged for the execution/mechanism phase).

dag_ml_data is a CORE dependency since the ADR-17 cutover; imports are still guarded so a broken wheel missing the native backend surfaces a clear error rather than a raw ImportError. Scope: single-source / no-repetition baseline.

nirs4all.pipeline.dagml.envelope.build_envelope(dataset: SpectroDataset, identity: IdentityMap, *, source_id: str = 'src0', sample_ints: list[int] | None = None, excluded_sample_ints: set[int] | None = None, metadata_by_sample: dict[str, dict[int, Any]] | None = None, tags_by_sample: dict[int, list[str]] | None = None, augmentation_by_sample: dict[int, str] | None = None, group_by_sample: dict[int, str] | None = None, final_sample_ints: list[int] | None = None) → dict[str, Any]

Build the validated CoordinatorDataPlanEnvelope.

Pass sample_ints to scope the envelope to a sample universe — e.g. the CV training pool, so the schema + coordinator_relations match the embedded FoldSet and pass validate_data_envelope_relations (every relation must live inside the fold set). Omit it for a whole-dataset envelope.

Pass excluded_sample_ints (a subset of sample_ints) to mark those rows excluded: true — the opt-in keep_in_oof=True exclude mode where Phase 1’s native bit drops them from fold TRAIN but keeps them in validation/OOF. Default (None) marks every row not-excluded.

Pass metadata_by_sample ({column: {sample_int: value}}) to emit per-sample metadata onto each relation — the native fan_out_data_aware_branches reads a by_metadata separation branch’s partition values from these relation metadata values.

Pass tags_by_sample ({sample_int: [tag, ...]}) to emit per-sample tag labels onto the native relations. Untagged rows omit the field.

Pass augmentation_by_sample ({sample_int: transform_id}) to attach the structured augmentation metadata to augmented rows (their origin_id is always emitted from the identity grain).

Pass group_by_sample ({sample_int: group_value}) to emit group_id on each relation for a repetition dataset — repetitions of one physical sample share a group value, and dag-ml-data refuses a fold that splits a group across train/validation. For a repetition dataset every relation is its own sample_id (each stored row is scored individually at the repetition grain), so the schema’s sample axis is one entry per row.

Pass final_sample_ints to request the V2 terminal prediction authority. It must contain every CV sample_ints entry and may add held-out prediction identities. A core that has not negotiated the V2 capability is refused instead of receiving a weakened envelope.

The wheel computes all fingerprints and derives coordinator_relations; a successful return means the envelope is contract-valid (the materialize-time fingerprint gate will accept it).

nirs4all.pipeline.dagml.envelope.num_targets(dataset: SpectroDataset) → int

The number of target columns (y width). 1 for the single-target baseline.

nirs4all.pipeline.dagml.envelope.sample_relations(identity: IdentityMap, *, source_id: str | None = 'src0', sample_ints: list[int] | None = None, excluded_sample_ints: set[int] | None = None, metadata_by_sample: dict[str, dict[int, Any]] | None = None, tags_by_sample: dict[int, list[str]] | None = None, augmentation_by_sample: dict[int, str] | None = None, group_by_sample: dict[int, str] | None = None) → dict[str, Any]

The SampleRelationTable rows, optionally scoped to a sample-int subset.

Pass sample_ints (e.g. the CV training pool) to scope the relations for fold validation; omit it for the full-dataset relations the envelope declares.

Pass excluded_sample_ints to emit excluded: true for those rows — the opt-in leakage-pure exclude mode (keep_in_oof=True): Phase 1’s native excluded bit then drops them from each fold’s TRAIN view while keeping them in validation/OOF and predict. Omit it (the default) and every row is excluded: false — the legacy mode removes excluded samples from the CV universe entirely (they are simply absent from the pool).

Pass metadata_by_sample ({column: {sample_int: value}}) to emit per-sample metadata onto each relation (Slice 1 metadata field). The native fan_out_data_aware_branches discovers a separation branch’s partition values from these relation metadata values, so a by_metadata branch criterion column must be present here. Omit it and every row carries empty metadata.

Pass tags_by_sample ({sample_int: [tag, ...]}) to emit per-sample tag labels onto each relation. Rows without tags omit the tags field so untagged fingerprints stay on the native contract’s skip-if-empty path.

Pass group_by_sample ({sample_int: group_value}) to emit group_id for a repetition dataset — several stored rows that share one physical sample carry the same group value (their repetition column). dag-ml-data’s validate_fold_set_against_sample_relations then refuses any fold that splits a group across train/validation (native group-leakage validation), so all repetitions of a sample must stay on the same fold side. Omit it and every row keeps group_id = None.

Augmented rows (identity minted on a dataset that already holds the synthetic rows) emit origin_id = the origin row’s observation_id (to_wire(origin_int), distinct from the child’s own observation_id); their sample_id stays the origin’s grouping key. Pass augmentation_by_sample ({sample_int: transform_id}) to attach the structured augmentation metadata ({transform_id, params_fingerprint}) for those rows. Base rows keep origin_id = None and no augmentation metadata (the dag-ml-data contract rejects an origin or augmentation on a non-augmented row).

nirs4all.pipeline.dagml.envelope.source_ids(dataset: SpectroDataset) → list[str]

The per-source ids the schema/plan/binding agree on.

Single-source keeps the legacy ["src0"] (BYTE-IDENTICAL); a multi-source dataset (features_sources() > 1) emits ["src0", "src1", …] — one per FeatureSource.

nirs4all.pipeline.dagml.envelope.source_order(dataset: SpectroDataset) → list[str]

Legacy by_source branch keys in exact source-index order.

Mirrors BranchController._get_source_names without importing the controller: use dataset.source_name(i) when a dataset exposes it, else fall back to source_i. These are the user-facing keys legacy by_source dict bodies resolve against, distinct from the native data-plan ids (src0/src1).

nirs4all.pipeline.dagml.envelope.supports_terminal_prediction_relation_authority() → bool

Return whether the loaded execution core accepts envelope V2 terminal relations.

nirs4all.pipeline.dagml.envelope.target_names(dataset: SpectroDataset) → list[str]

Declared multimodal target names, or positional names for spectral data.