Raw multimodal pipelines
The Python DAG host accepts aligned spectra, images, time series and mixed
metadata through nirs4all_io.MultimodalDataset. Images and series keep their
raw dimensions until their pipeline encoder runs. The synthetic example is a
software demonstration; it does not require a real corpus.
This capability currently requires the matching development builds of
nirs4all, nirs4all-io and dag-ml. It is not provided by the older published
packages bearing the same versions. Build wheels from these sources and install
them together; do not infer support from version numbers alone.
Run the demonstration
From the nirs4all repository, with its development environment installed:
python examples/user/02_data_handling/U07_multimodal.py --output /tmp/multimodal
The script writes dataset.json, pipeline.json, tuning.json, a paired search
checkpoint, multimodal.n4a and report.json. It trains from the round-tripped
JSON declarations. The report includes native CV/test scores, selected parameters,
eight trial records, twelve new predictions, elapsed time, Python allocation peak
(not total resident memory) and archive size.
To exercise cancellation and resume, use a new output directory:
python examples/user/02_data_handling/U07_multimodal.py --output /tmp/mm-resume --stop-after 2
python examples/user/02_data_handling/U07_multimodal.py --output /tmp/mm-resume --resume
python examples/user/02_data_handling/U07_multimodal.py --replay /tmp/mm-resume/multimodal.n4a --output /tmp/mm-predict
--search grid instead evaluates all eight combinations through the existing
native generator. Its run is separate from the resumable random-search profile.
For classification and multiple regression targets, run the second example:
python examples/user/02_data_handling/U08_multimodal_targets.py --output /tmp/mm-targets --case all
--case classification uses string labels, --case regression uses two complete
targets named concentration and moisture, and --case masked uses partially
observed versions of those targets. These names describe synthetic variables.
All cases use three grouped folds. Classification and masked regression each
run two durable trials; --resume reuses their completed search history.
Each case directory contains dataset.json, pipeline.json,
prediction_dataset.json, multimodal.n4a and report.json, plus the tuning
declaration and checkpoint when applicable. Reports preserve native scores,
per-target metrics, observed target counts and twelve new predictions. The script
reloads each archive with fitting forbidden and checks that predictions agree
exactly. examples/run.sh includes U07 through U14; its plot switches have no
effect on these examples, which write artifacts to fresh temporary directories.
Describe inputs
from nirs4all_io import MultimodalDataset, TensorSource
dataset = MultimodalDataset(
{
"nir": TensorSource(spectra, spectral_ids, representation_id="signal_1d",
axis_units={"wavelength": "nm"},
axis_coordinates={"wavelength": wavelengths}),
"image": TensorSource(images, image_ids, representation_id="rgb_image"),
"series": TensorSource(series, series_ids, representation_id="series_mv",
axis_units={"time": "s"}, axis_coordinates={"time": times}),
"metadata": TensorSource(metadata, metadata_ids, representation_id="tabular_mixed",
feature_names=["measurement", "category"]),
},
sample_ids=observation_ids, y=targets, groups=subject_ids,
partitions=partitions, # train/test; use predict for new unlabelled inputs
)
Every observation has a unique string ID. Repetitions have different observation
IDs; assign them to the same split group to keep them together. IO reorders each source to the canonical observation
order; the default source_alignment="strict" rejects missing, extra or duplicate
IDs. A group cannot span train and test.
Use an explicit grouped splitter such as GroupKFold(3).
Source arrays are copied and exposed read-only. Units are supplied by the caller;
no units, resampling, alignment by nearest time, or imputation are inferred.
dataset.to_dict() and MultimodalDataset.from_dict(payload) preserve the raw
arrays, dtypes, identities and axis metadata through JSON or YAML.
Declare independent experimental units
When several observations measure the same independent unit, declare both the unit and the repetition explicitly. Observation IDs remain unique. Split groups may be larger than units: for example, several specimens can share one batch. No statistical unit is inferred from a split group.
dataset = MultimodalDataset(
sources, sample_ids=observation_ids, y=targets,
task_type="regression", target_names=["concentration"],
groups=batch_ids, partitions=partitions,
independent_unit_ids=specimen_ids,
repetition_ids=scan_ids,
)
Both identity lists follow the observation order and are copied into immutable
tuples. A repeated unit requires repetition IDs; each (unit, repetition) pair
must be unique. Different units can use the same repetition name, such as
scan.0. Slicing and JSON round trips preserve these declarations. Omitting
both lists retains the existing observation-level behavior.
With engine="dag-ml", declaring units selects equal total influence per unit
and scores at the unit level. DAG-ML computes a row weight of 1 / count from
the observations of that unit in the actual fitting scope. For partial targets,
it uses the observed rows of each target separately. The Python operators pass
these native weights to every learned encoder and prediction head; they do not
recount, resample or average training observations. Supported estimators must
explicitly accept sample_weight in fit, including each learned step in a
model-owned sklearn pipeline.
For by_source pipelines, place weighted preprocessing and the prediction head
inside the same sklearn Pipeline model owner. The current _or_ transform
grammar creates separate fitting nodes, so this combination with independent
units is refused before FIT; it does not provide weighted model alternatives.
Native top-k selection of operator variants without experimental-unit weighting
remains a separate supported profile.
Regression scores use the mean prediction for each unit. Complete mono-target
classification votes over predicted labels; probability columns remain available
as prediction features. The target must be consistent among observations of a
unit. Native validation rejects unit leakage between fitting, validation and
test scopes, including inner folds. OOF feature buffers and returned prediction
rows retain observation IDs. The reported statistical key is
{"kind": "relation_metadata", "key": "independent_unit_id"}; it is distinct
from the split-group identity.
The current profile supports weighted sklearn models, complete-source early
fusion, and partial-source/per-target late fusion. Declare task_type explicitly
and use unit IDs made of ASCII letters, digits and _-.: (at most 128 bytes).
Named Torch, the Methods backend, standalone fitted transforms outside the model,
separate target transformations, branch-local tuning and early fusion with missing
sources are refused for this policy. Classification requires complete labels and
non-overlapping validation folds. These restrictions do not change runs that
omit the experimental-unit declaration.
See the public execution matrix for the current code and qualification status of each pipeline combination.
Encode variable-length series
Use RaggedSeriesSource when the time dimension varies between observations.
Its packed values array has shape (total_points, channels), while offsets
contains one boundary per sample plus the final boundary. The number of channels
is fixed. There is no padding, truncation, interpolation or implicit conversion
to a dense NumPy array.
from nirs4all_io import RaggedSeriesSource
from nirs4all.operators.models import MultimodalRegressor
from nirs4all.operators.transforms import SequenceSummary
from sklearn.linear_model import Ridge
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
series = RaggedSeriesSource(
packed_values, offsets, sample_ids,
time_coordinates=packed_times, # optional; strictly increasing per sample
channel_names=["temperature", "intensity"], time_unit="s",
)
encoder = make_pipeline(
SequenceSummary(channel_names=["temperature", "intensity"]),
StandardScaler(),
)
model = MultimodalRegressor(
transformers={"nir": StandardScaler(), "series": encoder},
model=Ridge(alpha=1.0),
)
series.values is a RaggedSeriesBatch. SequenceSummary computes mean,
population standard deviation, minimum and maximum per channel, then appends
the sequence length. Configure statistics=("mean", "std") and
include_length=False to choose a smaller representation. Output columns follow
channel order, then statistic order; get_feature_names_out() names them.
Fit retains the channel contract, without training values or lengths.
Declare a required series length with min_observations=2, or an applicability
range with channel_bounds=[(-20, 60), (0, 1000)], one inclusive numeric pair per
channel. Bounds use the source’s declared measurement units; the encoder does
not convert units. Every observed point is checked during fit and prediction,
before computing the summaries. These explicit constraints survive export and
replay. No range is inferred from training values. The defaults retain the
existing behavior: one observation suffices and no additional range is imposed.
All observations receive equal weight. Time coordinates are preserved by IO but are not used by this summary encoder, even for irregular sampling. Empty series and nonfinite observed values are refused by the encoder. Per-cell masks and automatic interpolation of missing time points are not supported.
U12 generates complete NIR and variable-length series through a finite
DataProvider, uses three grouped folds, exports the fitted model, and predicts
eight new observations whose sequences are longer than every training sequence:
python examples/user/02_data_handling/U12_multimodal_ragged_series.py --output /tmp/mm-ragged
python examples/user/02_data_handling/U12_multimodal_ragged_series.py --fusion intermediate --output /tmp/mm-ragged-mbpls
Early fusion uses Ridge; intermediate fusion uses MBPLS. The script limits BLAS to one thread, writes the provider recipe, pipeline, typed training/prediction datasets, archive and report, and checks JSON-roundtrip replay with fitting and training-provider execution forbidden. New lengths and source mapping order may change. Channel names/count, dtype, declared time unit and the time-coordinate schema must retain the fitted contract.
Handle absent modalities
source_alignment="left" explicitly allows a source to contain a subset of the
canonical sample IDs. IO aligns it to that order and records absent rows as
False in TensorSource.presence_mask. Extra and duplicate IDs still fail.
Alternatively, supply a boolean presence_mask with a full source buffer.
dataset.source_presence(rows=None) returns the named, read-only masks.
These describe entire source observations, not individual missing pixels.
Models reject absent sources by default. Opt in with
missing_source_policy="zero_with_indicator" on MultimodalRegressor or
MultimodalClassifier. Each encoder fits and transforms only present rows.
The model receives zero embeddings for absent rows and one presence column per
source, including when every row is present. Source weights multiply both the
embedding and its indicator. This is an explicit modelling choice, not a
reconstruction of missing measurements.
Both early and intermediate fusion accept this policy. With partial targets,
each target’s encoder uses the intersection of observed target and present
source rows. A source with no such training rows is rejected. At inference,
an entire modality may be absent: retain its name, dtype and trailing dimensions
in a zero-row TensorSource and use left alignment. Its fitted encoder is not
called. Presence patterns may change without changing the archive’s input schema.
Late regression fusion has a separate branch policy described below. Upstream
preprocessing outside these source-aware paths requires complete sources.
The third synthetic example combines incomplete sources with two partial targets, tunes the grouped pipeline, and predicts new observations with no images:
python examples/user/02_data_handling/U09_multimodal_missing_sources.py --output /tmp/mm-missing --stop-after 1
python examples/user/02_data_handling/U09_multimodal_missing_sources.py --output /tmp/mm-missing --resume
python examples/user/02_data_handling/U09_multimodal_missing_sources.py --fusion intermediate --output /tmp/mm-missing-mbpls
When calling the sklearn operator directly, pass
source_masks=dataset.source_presence() to fit, predict or predict_proba.
Public nirs4all.run() and archive prediction pass the aligned masks automatically.
Declare targets
y may have shape (n_samples,) or (n_samples, n_targets). Use target_names
to declare their order, for example target_names=["concentration", "moisture"],
and task_type="regression" or task_type="classification" to declare the task.
Names survive native scoring and archive replay. Unlabelled prediction datasets
use y=None; they can retain target_names and task_type without target values.
For classification, use MultimodalClassifier(transformers=..., model=...) with
a sklearn-compatible classifier, such as LogisticRegression. Public
nirs4all.predict() returns the original string or numeric class labels, including
non-contiguous numeric labels. The fitted operator exposes classes_ and offers
predict_proba() when its model supports probabilities; probability columns
follow classes_. The public archive prediction API currently returns labels,
not a probability matrix. Classification currently has a single target.
Complete multiple regression targets use MultimodalRegressor with its default
target_policy="complete" and a model supporting multiple outputs, such as Ridge.
Predictions preserve the target dimension. Complete multiple outputs are also
qualified with intermediate MBPLS fusion and native late fusion. Late fusion is
qualified for single-target classification as well.
For partially observed regression targets, declare both the mask and the policy:
dataset = MultimodalDataset(
sources, sample_ids=sample_ids, y=targets,
target_names=["concentration", "moisture"],
target_mask=observed, # bool, exactly the same shape as targets; True means observed
task_type="regression", # mandatory when any target cell is unobserved
groups=groups, partitions=partitions,
)
model = MultimodalRegressor(
transformers=transformers, model=Ridge(alpha=1.0), target_policy="per_target",
)
Each target gets an independent clone of every encoder and the final model.
Within each training fold, that clone only fits rows observed for that target;
supervised encoders never receive hidden target values. Fitted operators expose
target_models_ and target_counts_. Predictions still cover every requested row
and target. There is no imputation of missing target values.
Masks must be boolean and match y exactly. Observed cells must be finite;
masked cells may contain NaN, infinity or hidden finite values. The JSON codec
preserves those values and the mask without invalid JSON numbers. Every target
needs observed training rows, and scored target partitions need observed values.
Native metrics use only observed cells and retain named per-target metrics;
the scalar regression metric is their unweighted mean. A report’s row_count
counts prediction rows, not the number of observed cells for each target.
The default target_policy="complete" rejects partial masks. Partial regression
targets use the early or intermediate MultimodalRegressor path, or the
development late-fusion branch profile described below with
target_policy="per_target". Masked classification remains refused. Group-level
score aggregation with partial targets is outside these profiles.
Encode, fuse and tune
MultimodalRegressor(transformers={source_name: transformer, ...}, model=...)
clones and fits each source transformer on the current training fold. For
example, use StandardScaler for spectra, TensorPCA for raw fixed-shape image
or series tensors, and ColumnTransformer for mixed metadata.
fusion="early"concatenates the resulting numeric encodings.fusion="intermediate"sends a list of encoded blocks to a multiblock model, such asMBPLS. Usestandardize=Falseif source weights should retain their effect through MBPLS.Late fusion uses named
by_sourcebranches,merge="predictions"and a downstream model. DAG supplies nested grouped OOF predictions to its meta-model.
result = nirs4all.run(
[GroupKFold(3), {"model": model}], dataset, engine="dag-ml",
tuning={
"engine": "n4m", "sampler": "random", "seed": 17, "n_trials": 8,
"space": {
"transformers__image__n_components": [2, 4],
"source_weights__image": [0.5, 1.0],
"model__alpha": [0.1, 1.0],
},
"storage": "file:///absolute/study-directory", "study_name": "example",
},
save_charts=False,
)
result.export("multimodal.n4a")
prediction = nirs4all.predict("multimodal.n4a", new_raw_dataset)
result.close()
The durable profile covers one MultimodalRegressor or MultimodalClassifier,
or a late-fusion pipeline as described below. Topology is fixed. Sequential
N4M search accepts random, sobol, lhs, ternary, ga, pso, cmaes,
tpe, and gp_ei samplers. An omitted sampler retains the historical tpe
default. Regression
supports complete or partially observed targets and native regression metrics;
classification supports accuracy and balanced accuracy, defaulting to maximizing
balanced accuracy. Sequential search accepts median, successive_halving,
asha, hyperband, and racing pruning; pruned trials are recorded in both
checkpoints and the public tuning result. It rejects queued force_params.
Regression minimizes RMSE, MSE and MAE, and maximizes R² by default. An explicit
direction overrides that default. Classification requires non-overlapping
validation folds; repeated and overlapping holdouts are rejected before fitting
because averaging class indices is not a valid classification reduction.
DAG owns trial execution, scores and selection. The winner then
uses the existing native CV/refit path. Public tuning results use canonical
dotted parameter paths. Trial budget is total, including failed trials, and can
be increased on resume.
The tuning objective averages metrics across folds. RunResult.cv_best_score
uses the pooled out-of-fold predictions. These values can differ with unequal
fold sizes or nonlinear metrics such as RMSE and balanced accuracy; reports
retain both values and their native score evidence.
These CV values participate in hyperparameter selection; use the independent
test partition to evaluate the selected pipeline.
Joint Torch intermediate fusion (development)
Use the existing MultimodalRegressor with a supplied torch.nn.Module whose
forward(**inputs) accepts each source by name. Its encoders and prediction
head belong to the same module and train jointly with one Adam optimizer and
MSE loss. The source transformers must be None or "passthrough"; put learned
encoders inside your module.
model = MultimodalRegressor(
transformers={"nir": "passthrough", "clinical": "passthrough"},
model=joint_module, # Your nn.Module implementing forward(nir=..., clinical=...).
fusion="intermediate",
)
result = nirs4all.run(
[GroupKFold(3), {"model": model,
"train_params": {"epochs": 20, "batch_size": 16, "lr": 0.001}}],
cohort, engine="dag-ml", refit=True, random_state=42, save_charts=False,
)
result.export("joint-model.n4a")
result.close()
prediction = nirs4all.predict("joint-model.n4a", new_cohort)
The development profile accepts two to four complete fixed-shape float32 or
float64 sources and one complete numeric regression target. Supported IO
representations are signal_1d, tabular_numeric, gray_image, rgb_image,
mc_image, multispectral_image and series_mv. Each tensor retains its raw
non-sample axes until the user’s forward(**inputs) runs: RGB images have shape
(samples, height, width, 3) and series (samples, time, variable). The host
does not flatten, permute channels, pad or resample these inputs. Put any such
explicit conversion inside your module. CPU training converts inputs to float32; floating
module parameters and buffers must already be float32. Source names are ASCII
Python identifiers other than y. Source weights, missing-source policies,
partial targets, external preprocessing, model-local tuning and generated
provider views are outside this profile. Use one model with an optional index
or group splitter; omitting the splitter performs native full-training REFIT
and supplies no CV evidence.
This profile uses the default in-process DAG-ML runtime. Explicit CLI execution is refused before fitting because it does not transport the required source receipts. Each receipt records the real IO buffer, ordered sample IDs and input schema; the controller verifies them before training or prediction.
Alternatively, an importable Torch factory marked framework="pytorch" receives
input_shapes={name: source.values.shape[1:]} and returns your module, including
all raw image or series axes. Supplied module templates
retain their initial weights and are copied for each fold. Native tasks select
the training IDs, own CV/refit, and derive each CPU Torch callback seed from
the public random_state while restoring the caller’s RNG. Epochs and patience
are bounded by 100, batch size by 1024,
and learning rate by [1e-6, 0.1]; the adapter also bounds model size and work.
Export retains the actual completed REFIT origin, fitted tensor identities, named input contract, complete IO schema and selected owner. Replay checks these before prediction and does not fit again. New observation IDs and per-source row order are allowed through IO alignment. In this first named profile, retain the cohort’s source mapping order because the native source IDs are part of the captured contract. This is a trusted Python archive, requiring the user’s importable factory or module dependencies and matching installed development builds. It does not provide portable Torch weights for R or WASM. Ragged joint Torch inputs remain unsupported by this profile; use the explicit summary encoder for ragged series. The demonstration arrays are software fixtures.
Training controls during global search (1.3.3)
Since version 1.3.3, the direct multimodal model, each late-fusion branch
model and the meta-model can carry fixed train_params and refit_params:
model_step = {
"model": model,
"train_params": {"model__tol": 0.003},
"refit_params": {"model__alpha": 9.0},
}
The example uses a MultimodalRegressor containing Ridge; ordinary Ridge branch
and meta-model steps use tol and alpha directly. These are estimator
parameter overrides, not arbitrary fit() arguments. Controls are validated
before fitting or opening a study. Each trial and the winner’s CV use
train_params; terminal refit adds refit_params, which take precedence.
The resulting fitted predictor and its archive preserve the refit behavior.
The same contract applies to isolated parallel candidates.
Fixed training controls take precedence over candidate parameters. To search a
parameter, leave it out of train_params and declare it in tuning.space.
Changing either control mapping invalidates checkpoint resume before any fit.
CV-weight warm starts remain unsupported. The fixed controls above are available
in nirs4all 1.3.3 or newer.
Nested local search (development main)
Model-local finetune_params can be nested within global search for a direct
multimodal model or a by_source base model. The local engine must select the
Optuna or n4m host search profile. The global engine still selects n4m and
addresses the whole model or ensemble through tuning.space.
For example, let global search choose an image encoder dimension while local search chooses the direct model’s Ridge regularization:
result = nirs4all.run(
[GroupKFold(3), {
"model": multimodal_model,
"finetune_params": {
"engine": "n4m", "sampler": "random", "seed": 19,
"approach": "grouped", "eval_mode": "mean", "n_trials": 3,
"model_params": {"model__alpha": [0.1, 1.0, 10.0]},
},
"train_params": {"model__tol": 0.003},
"refit_params": {"model__alpha": 9.0},
}],
cohort,
tuning={"engine": "n4m", "sampler": "random", "seed": 19,
"n_trials": 3, "space": {"transformers__image__n_components": [1, 2]}},
engine="dag-ml", refit=True, save_charts=False,
)
Every global candidate and native outer training scope receives its own local search. Grouped local search clones the real splitter policy inside that training scope, including group constraints. Native inner tasks fit fresh encoder/preprocessing declarations on inner training rows only; an already fitted outer encoder is never reused to select local parameters. Direct models retain named raw source blocks, while base models consume only their declared source. Terminal refit performs a separate training-only local search; the step’s fixed train/refit controls retain their documented precedence.
A parameter must have one search owner. Overlapping global paths and local model, trial-training or forced parameters refuse before an optimizer opens; dotted and sklearn double-underscore paths name the same parameter. Changing the local configuration, upstream recipe, splitter, controls or training content invalidates global resume. Export retains the fitted selected predictor and its evidence; archive prediction performs no HPO or fit.
Meta-model local HPO remains refused because searching an already computed OOF
matrix would expose inner selection targets. This nested profile also refuses
generated views, augmentation, feature/prediction joins and all-observation
preprocessing policies, incomplete sources/targets and missing-source policies.
The closed engine="native" profile and CV-weight warm starts keep their
separate capability restrictions.
This nested composition is available on development main; it is not part of
the published 1.3.3 fixed-control qualification.
tuning.progress_callback(event) may return False between trials. Cancellation
raises MultimodalTuningStopped; a model failure is checkpointed and its original
error propagates. resume=True continues from terminal history. A crash during
an unfinished trial returns to the preceding checkpoint and may repeat that
unfinished trial. Changing training buffers, labels, schemas, folds or search
semantics refuses resume. Target names, task type and training masks are part of
the search identity. Source presence is also fingerprinted. Hidden target/source
values and held-out test data are excluded from search; modifying them does not
create a new search objective.
For the native multimodal N4M random-search path, set tuning.n_jobs=2 or
another positive worker count to evaluate independent candidates concurrently.
Specify sampler="random" for new parallel searches; explicit nonrandom
samplers and pruning require n_jobs=1.
Each candidate owns a Python process, its provider snapshot, fitted models and
Python/NumPy RNG. n_jobs=1 retains the existing sequential checkpoint
identity. Parallel checkpoints may contain proposals already opened for other
candidates; resume=True reuses those proposals and verifies saved generated
views before new evaluation. This setting does not enable parallelism in the
general PipelineObjective tuning adapter. For generated views it requires a
DAG-ML binding that advertises candidate-local generated views, whether the
outer job runs in-process or in an isolated Python worker. Older installed
bindings fail before the provider PLAN.
For generated views with N4A_DAGML_INPROCESS=0, HPO uses an isolated Python
worker. tuning.progress_callback runs in the calling process after each
checkpoint event; returning False stops at the saved trial boundary, and
returning True or None continues. resume=True continues from that
checkpoint. The parent’s should_stop may
terminate the worker during a trial; the last complete paired checkpoint
remains available, and the interrupted trial may repeat on resume.
The host resets the Python/NumPy RNG for each native task using a stable seed
derived from run(random_state=...), otherwise the tuning seed, otherwise zero.
This makes supported sklearn operators using the global RNG reproducible across
resume even when a progress callback consumes randomness. Explicit operator
seeds remain effective. Custom generators and GPU determinism are not guaranteed
by this policy.
Select independent source outputs
When a by_source pipeline exposes an independent prediction output per source,
native selection ranks candidate recipes using the first source in dataset
order, signed as selection_output_id="output:source_0". All source outputs,
validation scores and OOF arrays are retained. Other source scores do not
implicitly become a fused ranking metric; an explicit fusion model has its own
output and selection evidence. Exactly equal selection scores use the original
variant ID as the deterministic tie break. Ranked refits retain this same native
ordering and their own genuine captures.
Tune a late-fusion ensemble
Use the existing by_source branches followed by {"merge": "predictions"}
and a meta-model. Pass a global tuning mapping to nirs4all.run():
tuning = {
"engine": "n4m", "sampler": "random", "seed": 17, "n_trials": 4,
"space": {
"branches.image.0.n_components": [2, 4],
"branches.nir.1.alpha": [0.1, 1.0],
"branches.metadata.0.numeric.with_mean": [False, True],
"meta.alpha": [0.1, 1.0],
},
"storage": "file:///absolute/late-study", "study_name": "late-fusion",
}
branches.<source>.<step_index>.<parameter> addresses the named source and
zero-based position in its public step list. Positions include None entries;
those entries themselves cannot be tuned. meta.<parameter> addresses the
final estimator. Nested sklearn parameters accept dots or __, as with
branches.metadata.0.numeric__with_mean. The example assumes a PCA at image
position 0, Ridge at NIRS position 1 and a metadata ColumnTransformer with
a transformer named numeric at position 0.
Every candidate recomputes all branch encoders, base models and inner OOF predictions through the native scheduler. The meta-model fits those inner predictions and is scored on the outer validation folds. Training and test identities remain separate. Search paths and their destination nodes are part of the checkpoint identity; changing their routing refuses resume.
The returned RunResult represents the selected ensemble. Its direct
export() saves that ensemble even if an individual base model scores better.
The profile requires instantiated sklearn operators. Sources and targets must
be complete unless the branch policies below explicitly declare missingness.
Development main also accepts the fixed model training controls described
above; base-model local finetune_params may be nested in this global search
profile. Meta-model local HPO remains refused. Single-target classification and complete multi-target regression use
the same graph.
The synthetic U10 example writes the recipe, cohort, checkpoint, report and archive, then predicts twelve new observations with fit forbidden:
python examples/user/02_data_handling/U10_multimodal_late_tuning.py --output /tmp/mm-late --stop-after 2
python examples/user/02_data_handling/U10_multimodal_late_tuning.py --output /tmp/mm-late --resume
Late fusion with absent modalities and partial regression targets
Declare missing_source_policy="zero_with_indicator" on the named source branch:
pipeline = [
GroupKFold(3),
{"branch": {
"by_source": True,
"missing_source_policy": "zero_with_indicator",
"steps": {
"nir": [StandardScaler(), Ridge()],
"series": [SequenceSummary(), StandardScaler(), Ridge()],
},
}},
{"merge": "predictions"},
Ridge(),
]
Every branch encoder and base model fits only the present rows of its current native training scope, including the inner folds that supply meta-model inputs. The meta-model retains every OOF observation ID. Each branch contributes its predictions followed by one presence indicator. For an absent source, prediction columns are zero in the original target space and the indicator is zero; that source’s encoders and predictor are not called. The indicator remains a column when every source observation is present. Two single-target branches therefore produce four meta-model features. Complete multiple targets are also supported.
Zeros encode an absent branch for the meta-model; they do not reconstruct raw
measurements. Native meta-model scores cover the requested scored rows;
base-model predictions remain restricted to observations actually present for
that source and observed for that target.
An entirely unobserved source in any inner, outer or final training scope is
rejected before model callbacks, rather than borrowing fitted state from another
fold. The development profile also accepts complete single-target classification
and partially observed regression targets. For partial targets, add
target_policy="per_target" beside the branch’s missing_source_policy; each
target gets a fresh encoder/model fitted on the intersection of present source
rows and observed target rows in that actual outer, inner or REFIT scope.
Classification requires classifiers exposing predict_proba. Each branch
supplies all K columns in the shared training vocabulary order, followed by
its presence indicator. An absent observation supplies no class distribution;
the native joined features retain explicit validity and presence masks. Every
true training scope must contain the complete class vocabulary. The native
scheduler validates availability before model callbacks or global HPO.
This partial late profile is serial, with two to four named branches and a single meta-model. Global whole-stack HPO is supported; branch-local HPO with partial observations, external preprocessing before the source branch, augmentation, generated views, separate target transformations and already fitted preprocessing are refused. Classification targets must remain complete. See the partial late-fusion guide for the public pipeline declaration and the archive contract.
At prediction, presence patterns and series lengths may change. An entire series
modality can be absent: retain its declared channels, dtype, time unit and
coordinate schema in a zero-row RaggedSeriesSource, and use
source_alignment="left". The archive records and validates source order,
prediction widths, indicators and the fitted meta-model width. As with other
ragged inputs, SequenceSummary uses equal weights for time points and ignores
time coordinates; per-cell masks are unsupported.
U13 generates 24 training and eight test observations, with separate groups,
dense NIR and partially absent variable-length series. Its three native trials
tune branches.series.0.include_length, branches.nir.1.alpha and meta.alpha:
python examples/user/02_data_handling/U13_multimodal_late_missing_sources.py --output /tmp/mm-late-missing --stop-after 1
python examples/user/02_data_handling/U13_multimodal_late_missing_sources.py --output /tmp/mm-late-missing --resume
Use a fresh directory and omit both flags for a continuous run. The example
writes provider/pipeline/tuning recipes, typed datasets, durable checkpoints,
late-fusion.n4a and JSON/text reports. On resume it checks that the completed
prefix is retained and only remaining trial numbers are reported as new. It
then replays eight new observations with some series absent and eight with all
series absent, while fitting and provider materialization are forbidden. The
all-absent case additionally forbids calling the series encoder. The standalone
prediction datasets and report retain both sets of predictions.
Executable synthetic data providers
Pass an IO DataProvider directly as dataset. DAG-ML executes its source node
once in PLAN, before constructing cross-validation folds. The callback returns
an IO MultimodalDataset; IO validates raw shapes, identities and alignment.
The provider does not learn from observations. Learned augmentation belongs in
a fold-scoped training controller.
from nirs4all_io import DataProvider
def generate_cohort(*, seed, params, context):
# Reuse a scientific generator and return aligned raw TensorSource objects
# inside a MultimodalDataset, with explicit groups and partitions.
return make_synthetic_cohort(seed=seed, n_samples=params["n_samples"])
provider = DataProvider(
generate_cohort, provider_id="my.synthetic.cohort", provider_version="1",
params={"n_samples": 64}, seed=17,
)
result = nirs4all.run(pipeline, provider, save_charts=False)
The provider’s root seed is separate from the model’s random_state. Native
PLAN derives and records the effective task seed passed to the callback.
provider.materialize() is also available outside a run and uses the provider’s
root seed directly; pass an explicit seed to reproduce a native realization.
Callbacks must use their seed and parameters deterministically. Change
provider_version when changing the recipe’s implementation.
For partial production, supply base=fixed_cohort and return a mapping with
sample_ids plus sources and/or y. IO aligns it to the base’s fixed universe.
Adding sources is allowed; replacing existing sources or targets requires
replace_sources=("image",) or replace_targets=True. Groups and partitions
remain explicit and the base is not mutated.
After materialization, provider.get(sample_ids) and
provider.batches(batch_size, sample_ids=training_ids) return aligned cohort
views without calling the generator again. A batch iterator retains its cursor:
batches = provider.batches(16)
first = next(batches)
state = batches.state_dict() # JSON-serializable
restored = DataProvider(generate_cohort, provider_id="my.synthetic.cohort",
provider_version="1", params={"n_samples": 64}, seed=17)
restored.load_state_dict(state["provider"])
remaining = restored.batches(16)
remaining.load_state_dict(state)
Restoring the provider explicitly regenerates and verifies the recorded content before publishing it. Restoring the iterator checks the recipe, content, selection and batch configuration before accepting its cursor. This is a finite, materialized provider: batching does not make generation itself out-of-core. One run uses one fixed realization across folds and HPO trials; a changed recipe invalidates HPO resume. Dynamic row creation, learned generators, infinite streams and native per-epoch regeneration are outside this profile. A model controller may iterate the fixed cohort for its epochs.
nirs4all_io.provider_adapters.SklearnProviderAdapter(provider, source="nir")
exposes .arrays(training_ids) as (X, y) for an ordinary estimator’s fit.
Its .batches(...) supplies arrays for caller-managed partial_fit when the
estimator actually provides that method. N-D sources require an explicit
encoder; omitting source returns a mapping for source-aware estimators.
The optional TorchMapDataset and TorchIterableDataset classes in the same
module implement PyTorch’s real dataset interfaces. Pass sample_ids=training_ids
to constrain a loader to a fold. Iterable workers shard sample positions without
duplication. return_metadata=True retains IDs, groups, partitions and masks;
tuple mode refuses to discard missingness masks. Use
collate_fn=collate_provider_samples for mixed categorical/numeric sources.
Adapters capture the realized cohort, not the generator callback. Their dataset
values are independent of worker count; iterable delivery order may differ.
The provider cursor checkpoint does not capture DataLoader prefetch queues or a
model/optimizer checkpoint. PyTorch is imported only when explicitly requested.
The source recipe, effective seed, native lineage and content fingerprint are retained in run metadata and exported host models. Prediction uses the fitted model and explicitly supplied new inputs; it never executes the training provider. U11 demonstrates full and partial production, reuses the existing NIRS synthesis API, restores a batch cursor and exports/replays a model:
python examples/user/02_data_handling/U11_multimodal_data_provider.py --output /tmp/mm-provider
# With the optional CPU/GPU PyTorch dependency installed:
python examples/user/02_data_handling/U11_multimodal_data_provider.py --output /tmp/mm-provider-torch --torch-workers 2
Generated feature views per fold
For a bounded native CV run, generate_view can replace declared raw sources
after PLAN. The base cohort fixes IDs, targets, groups, partitions and source
schemas before splitting. The callback receives only the scheduler-selected
IDs and its deterministic view seed; return the same IDs in the same order.
For example, given a fully aligned cohort and a source-aware concrete
model (such as MultimodalRegressor):
import numpy as np
import nirs4all
from nirs4all_io import DataProvider, TensorSource
from sklearn.model_selection import GroupKFold
def prepare_nir(**_):
return {"sample_ids": list(cohort.sample_ids),
"sources": {"nir": cohort.sources["nir"]}}
def nir_for_view(*, sample_ids, seed, **_):
source = cohort.take(sample_ids).sources["nir"]
values = np.asarray(source.values)
# A deterministic fixture perturbation; replace with your own provider.
values = values + np.random.default_rng(seed).normal(0, 0.001, values.shape)
return {"sample_ids": sample_ids, "sources": {"nir": TensorSource(
values, sample_ids, representation_id=source.representation_id,
axis_units=source.axis_units, axis_coordinates=source.axis_coordinates,
)}}
provider = DataProvider(prepare_nir, generate_view=nir_for_view,
provider_id="example.fold-nir", base=cohort,
replace_sources=["nir"])
result = nirs4all.run([GroupKFold(3), {"model": model}], provider,
engine="dag-ml", refit=True, save_artifacts=False,
save_charts=False, random_state=19)
try:
print(result.best_rmse)
finally:
result.close()
The qualified profile also accepts exact KFold, StratifiedKFold and
StratifiedGroupKFold splitters, with a single concrete estimator. A concrete
sklearn Pipeline may learn preprocessing inside that estimator on generated
training views. With one complete raw source, one concrete sklearn-style
transformer may instead sit between the splitter and model as a separate DAG
node. The node fits on the generated fold-train or full-train view; its fitted
state is carried into that fold’s model and, for a qualified multimodal refit,
into the prediction archive. A shared transformer over multiple heterogeneous
raw sources is refused before fitting; use source-specific encoders inside the
multimodal model in that case. The existing by_source branch with a distinct
preprocessing list for each named source, followed by {"merge": {"sources": "concat"}} and a concrete model, also fits each source chain on its generated
fold-train view. This branch uses one native model task to attest all input
blocks. Its refitted source chains can be exported in a prediction-only .n4a
archive when the downstream model is a sklearn-style regressor, then replayed on a new explicit multimodal cohort, as can the source
encoders inside MultimodalRegressor or MultimodalClassifier. Ragged series
can be replaced as long as their
declared schema is retained. For MultimodalRegressor or MultimodalClassifier,
the model-only profile
also supports N4M tuning (engine="n4m") with the sequential samplers and
pruners described above, or parallel random search,
with a paired native/optimizer checkpoint. The checkpoint contains a manifest
of each completed trial’s generated views. On resume, the provider regenerates
and compares every saved view before another trial starts; changing the recipe
or the generated content refuses the resume. Use a stable storage URI and
study_name, then call run again with resume=True and the same provider
recipe. A completed search refits the selected model on generated training
views.
For one fitted MultimodalRegressor or MultimodalClassifier, result.export()
now writes a prediction-only .n4a containing the selected REFIT model, its
input schema and the verified generated-view manifest. Supply a new explicit
multimodal cohort to nirs4all.predict(archive, new_cohort); prediction does
not call the training provider or fit a model. Its sources must match the
training schema, including names, shapes, axes, units and feature names. The
archive does not regenerate training data and cannot be used for retraining.
Other generated-view estimator shapes still refuse export. Set
N4A_DAGML_INPROCESS=0 to run the qualified CV/refit profile in an isolated
Python worker: it receives the completed PLAN provider, creates its own view
store, and returns the native view manifest and fitted refit artifacts. The
provider callbacks must be serializable by cloudpickle, and that interpreter
must have the DAG-ML Python binding installed. This path does not use
dag-ml-cli. The parent’s should_stop callback can cancel the worker; no
completed run or refit archive is published in that case. Generated-view
HPO also runs in a Python worker when N4A_DAGML_INPROCESS=0, including
tuning.n_jobs>1 for the qualified N4M random-search profile. The worker
owns its trial views, optimizer and selected refit; parallel candidates each
run in a separate child process. It returns a usable RunResult for archive
export, and a paired checkpoint can be resumed in a later worker. The parent
stops active candidate processes when should_stop cancels the worker.
tuning.progress_callback is relayed to the calling process, including its
stop decision; a callback exception propagates to the caller and stops the
worker. The nested parallel worker path has been qualified from installed
candidate wheels on Linux, Windows x64 and macOS ARM. This does not qualify
Studio installers on those platforms. Separate transform nodes on multiple
raw sources remain outside this profile.
Multimodal synthetic data in this example are test fixtures, not a new product
generator.
For a complete runnable example of fold views, HPO and prediction-only replay:
python examples/user/02_data_handling/U14_multimodal_fold_provider.py --output /tmp/mm-fold-provider
python examples/user/02_data_handling/U14_multimodal_fold_provider.py --output /tmp/mm-fold-provider-hpo --hpo --n-jobs 2 --worker
Archive and scope
The archive contains fitted encoders, fusion and final model, raw input contracts, selected recipe and exact host dependency versions. Replay accepts new observation IDs and source mapping order but rejects source, dimension, dtype, feature name, axis, coordinate or unit changes. It performs the DAG PREDICT phase without fit and does not require the training workspace.
These are Python host archives containing trusted joblib objects, not portable Core/ONNX/WASM model archives. Replay requires the declared package versions and Python major/minor version. Dense inputs have fixed trailing dimensions, and temporal sources may use the typed ragged-series contract with variable-length observations. Both forms support the source and target masks described above. Cross-language ragged replay is not qualified. Classification encodings and inferred task types depend only on training targets; a held-out class absent from that vocabulary is refused.
See the installation qualification
for the independent wheel installation and replay with fit and source-directory
access forbidden, and scripts/verify_multimodal_archive.py to repeat that check.
The targets and missing-source qualification
records the U08/U09 checks and their current validation status.
The late-fusion tuning qualification
records U10, durable whole-stack search and its independent installation proof.