# Raw multimodal pipelines The Python DAG host accepts aligned spectra, images, time series and mixed metadata through `nirs4all_io.MultimodalDataset`. Images and series keep their raw dimensions until their pipeline encoder runs. The synthetic example is a software demonstration; it does not require a real corpus. This capability currently requires the matching development builds of `nirs4all`, `nirs4all-io` and `dag-ml`. It is not provided by the older published packages bearing the same versions. Build wheels from these sources and install them together; do not infer support from version numbers alone. ## Run the demonstration From the nirs4all repository, with its development environment installed: ```bash python examples/user/02_data_handling/U07_multimodal.py --output /tmp/multimodal ``` The script writes `dataset.json`, `pipeline.json`, `tuning.json`, a paired search checkpoint, `multimodal.n4a` and `report.json`. It trains from the round-tripped JSON declarations. The report includes native CV/test scores, selected parameters, eight trial records, twelve new predictions, elapsed time, Python allocation peak (not total resident memory) and archive size. To exercise cancellation and resume, use a new output directory: ```bash python examples/user/02_data_handling/U07_multimodal.py --output /tmp/mm-resume --stop-after 2 python examples/user/02_data_handling/U07_multimodal.py --output /tmp/mm-resume --resume python examples/user/02_data_handling/U07_multimodal.py --replay /tmp/mm-resume/multimodal.n4a --output /tmp/mm-predict ``` `--search grid` instead evaluates all eight combinations through the existing native generator. Its run is separate from the resumable random-search profile. For classification and multiple regression targets, run the second example: ```bash python examples/user/02_data_handling/U08_multimodal_targets.py --output /tmp/mm-targets --case all ``` `--case classification` uses string labels, `--case regression` uses two complete targets named `concentration` and `moisture`, and `--case masked` uses partially observed versions of those targets. These names describe synthetic variables. All cases use three grouped folds. Classification and masked regression each run two durable trials; `--resume` reuses their completed search history. Each case directory contains `dataset.json`, `pipeline.json`, `prediction_dataset.json`, `multimodal.n4a` and `report.json`, plus the tuning declaration and checkpoint when applicable. Reports preserve native scores, per-target metrics, observed target counts and twelve new predictions. The script reloads each archive with fitting forbidden and checks that predictions agree exactly. `examples/run.sh` includes U07 through U14; its plot switches have no effect on these examples, which write artifacts to fresh temporary directories. ## Describe inputs ```python from nirs4all_io import MultimodalDataset, TensorSource dataset = MultimodalDataset( { "nir": TensorSource(spectra, spectral_ids, representation_id="signal_1d", axis_units={"wavelength": "nm"}, axis_coordinates={"wavelength": wavelengths}), "image": TensorSource(images, image_ids, representation_id="rgb_image"), "series": TensorSource(series, series_ids, representation_id="series_mv", axis_units={"time": "s"}, axis_coordinates={"time": times}), "metadata": TensorSource(metadata, metadata_ids, representation_id="tabular_mixed", feature_names=["measurement", "category"]), }, sample_ids=observation_ids, y=targets, groups=subject_ids, partitions=partitions, # train/test; use predict for new unlabelled inputs ) ``` Every observation has a unique string ID. Repetitions have different observation IDs; assign them to the same split group to keep them together. IO reorders each source to the canonical observation order; the default `source_alignment="strict"` rejects missing, extra or duplicate IDs. A group cannot span train and test. Use an explicit grouped splitter such as `GroupKFold(3)`. Source arrays are copied and exposed read-only. Units are supplied by the caller; no units, resampling, alignment by nearest time, or imputation are inferred. `dataset.to_dict()` and `MultimodalDataset.from_dict(payload)` preserve the raw arrays, dtypes, identities and axis metadata through JSON or YAML. ## Declare independent experimental units When several observations measure the same independent unit, declare both the unit and the repetition explicitly. Observation IDs remain unique. Split groups may be larger than units: for example, several specimens can share one batch. No statistical unit is inferred from a split group. ```python dataset = MultimodalDataset( sources, sample_ids=observation_ids, y=targets, task_type="regression", target_names=["concentration"], groups=batch_ids, partitions=partitions, independent_unit_ids=specimen_ids, repetition_ids=scan_ids, ) ``` Both identity lists follow the observation order and are copied into immutable tuples. A repeated unit requires repetition IDs; each `(unit, repetition)` pair must be unique. Different units can use the same repetition name, such as `scan.0`. Slicing and JSON round trips preserve these declarations. Omitting both lists retains the existing observation-level behavior. With `engine="dag-ml"`, declaring units selects equal total influence per unit and scores at the unit level. DAG-ML computes a row weight of `1 / count` from the observations of that unit in the actual fitting scope. For partial targets, it uses the observed rows of each target separately. The Python operators pass these native weights to every learned encoder and prediction head; they do not recount, resample or average training observations. Supported estimators must explicitly accept `sample_weight` in `fit`, including each learned step in a model-owned sklearn pipeline. For `by_source` pipelines, place weighted preprocessing and the prediction head inside the same sklearn `Pipeline` model owner. The current `_or_` transform grammar creates separate fitting nodes, so this combination with independent units is refused before FIT; it does not provide weighted model alternatives. Native top-k selection of operator variants without experimental-unit weighting remains a separate supported profile. Regression scores use the mean prediction for each unit. Complete mono-target classification votes over predicted labels; probability columns remain available as prediction features. The target must be consistent among observations of a unit. Native validation rejects unit leakage between fitting, validation and test scopes, including inner folds. OOF feature buffers and returned prediction rows retain observation IDs. The reported statistical key is `{"kind": "relation_metadata", "key": "independent_unit_id"}`; it is distinct from the split-group identity. The current profile supports weighted sklearn models, complete-source early fusion, and partial-source/per-target late fusion. Declare `task_type` explicitly and use unit IDs made of ASCII letters, digits and `_-.:` (at most 128 bytes). Named Torch, the Methods backend, standalone fitted transforms outside the model, separate target transformations, branch-local tuning and early fusion with missing sources are refused for this policy. Classification requires complete labels and non-overlapping validation folds. These restrictions do not change runs that omit the experimental-unit declaration. See the [public execution matrix](../../reference/multimodal_execution_matrix.md) for the current code and qualification status of each pipeline combination. ## Encode variable-length series Use `RaggedSeriesSource` when the time dimension varies between observations. Its packed `values` array has shape `(total_points, channels)`, while `offsets` contains one boundary per sample plus the final boundary. The number of channels is fixed. There is no padding, truncation, interpolation or implicit conversion to a dense NumPy array. ```python from nirs4all_io import RaggedSeriesSource from nirs4all.operators.models import MultimodalRegressor from nirs4all.operators.transforms import SequenceSummary from sklearn.linear_model import Ridge from sklearn.pipeline import make_pipeline from sklearn.preprocessing import StandardScaler series = RaggedSeriesSource( packed_values, offsets, sample_ids, time_coordinates=packed_times, # optional; strictly increasing per sample channel_names=["temperature", "intensity"], time_unit="s", ) encoder = make_pipeline( SequenceSummary(channel_names=["temperature", "intensity"]), StandardScaler(), ) model = MultimodalRegressor( transformers={"nir": StandardScaler(), "series": encoder}, model=Ridge(alpha=1.0), ) ``` `series.values` is a `RaggedSeriesBatch`. `SequenceSummary` computes mean, population standard deviation, minimum and maximum per channel, then appends the sequence length. Configure `statistics=("mean", "std")` and `include_length=False` to choose a smaller representation. Output columns follow channel order, then statistic order; `get_feature_names_out()` names them. Fit retains the channel contract, without training values or lengths. Declare a required series length with `min_observations=2`, or an applicability range with `channel_bounds=[(-20, 60), (0, 1000)]`, one inclusive numeric pair per channel. Bounds use the source's declared measurement units; the encoder does not convert units. Every observed point is checked during fit and prediction, before computing the summaries. These explicit constraints survive export and replay. No range is inferred from training values. The defaults retain the existing behavior: one observation suffices and no additional range is imposed. All observations receive equal weight. Time coordinates are preserved by IO but are **not used** by this summary encoder, even for irregular sampling. Empty series and nonfinite observed values are refused by the encoder. Per-cell masks and automatic interpolation of missing time points are not supported. U12 generates complete NIR and variable-length series through a finite `DataProvider`, uses three grouped folds, exports the fitted model, and predicts eight new observations whose sequences are longer than every training sequence: ```bash python examples/user/02_data_handling/U12_multimodal_ragged_series.py --output /tmp/mm-ragged python examples/user/02_data_handling/U12_multimodal_ragged_series.py --fusion intermediate --output /tmp/mm-ragged-mbpls ``` Early fusion uses Ridge; intermediate fusion uses MBPLS. The script limits BLAS to one thread, writes the provider recipe, pipeline, typed training/prediction datasets, archive and report, and checks JSON-roundtrip replay with fitting and training-provider execution forbidden. New lengths and source mapping order may change. Channel names/count, dtype, declared time unit and the time-coordinate schema must retain the fitted contract. ## Handle absent modalities `source_alignment="left"` explicitly allows a source to contain a subset of the canonical sample IDs. IO aligns it to that order and records absent rows as `False` in `TensorSource.presence_mask`. Extra and duplicate IDs still fail. Alternatively, supply a boolean `presence_mask` with a full source buffer. `dataset.source_presence(rows=None)` returns the named, read-only masks. These describe entire source observations, not individual missing pixels. Models reject absent sources by default. Opt in with `missing_source_policy="zero_with_indicator"` on `MultimodalRegressor` or `MultimodalClassifier`. Each encoder fits and transforms only present rows. The model receives zero embeddings for absent rows and one presence column per source, including when every row is present. Source weights multiply both the embedding and its indicator. This is an explicit modelling choice, not a reconstruction of missing measurements. Both early and intermediate fusion accept this policy. With partial targets, each target's encoder uses the intersection of observed target and present source rows. A source with no such training rows is rejected. At inference, an entire modality may be absent: retain its name, dtype and trailing dimensions in a zero-row `TensorSource` and use left alignment. Its fitted encoder is not called. Presence patterns may change without changing the archive's input schema. Late regression fusion has a separate branch policy described below. Upstream preprocessing outside these source-aware paths requires complete sources. The third synthetic example combines incomplete sources with two partial targets, tunes the grouped pipeline, and predicts new observations with no images: ```bash python examples/user/02_data_handling/U09_multimodal_missing_sources.py --output /tmp/mm-missing --stop-after 1 python examples/user/02_data_handling/U09_multimodal_missing_sources.py --output /tmp/mm-missing --resume python examples/user/02_data_handling/U09_multimodal_missing_sources.py --fusion intermediate --output /tmp/mm-missing-mbpls ``` When calling the sklearn operator directly, pass `source_masks=dataset.source_presence()` to `fit`, `predict` or `predict_proba`. Public `nirs4all.run()` and archive prediction pass the aligned masks automatically. ## Declare targets `y` may have shape `(n_samples,)` or `(n_samples, n_targets)`. Use `target_names` to declare their order, for example `target_names=["concentration", "moisture"]`, and `task_type="regression"` or `task_type="classification"` to declare the task. Names survive native scoring and archive replay. Unlabelled prediction datasets use `y=None`; they can retain `target_names` and `task_type` without target values. For classification, use `MultimodalClassifier(transformers=..., model=...)` with a sklearn-compatible classifier, such as `LogisticRegression`. Public `nirs4all.predict()` returns the original string or numeric class labels, including non-contiguous numeric labels. The fitted operator exposes `classes_` and offers `predict_proba()` when its model supports probabilities; probability columns follow `classes_`. The public archive prediction API currently returns labels, not a probability matrix. Classification currently has a single target. Complete multiple regression targets use `MultimodalRegressor` with its default `target_policy="complete"` and a model supporting multiple outputs, such as Ridge. Predictions preserve the target dimension. Complete multiple outputs are also qualified with intermediate MBPLS fusion and native late fusion. Late fusion is qualified for single-target classification as well. For partially observed regression targets, declare both the mask and the policy: ```python dataset = MultimodalDataset( sources, sample_ids=sample_ids, y=targets, target_names=["concentration", "moisture"], target_mask=observed, # bool, exactly the same shape as targets; True means observed task_type="regression", # mandatory when any target cell is unobserved groups=groups, partitions=partitions, ) model = MultimodalRegressor( transformers=transformers, model=Ridge(alpha=1.0), target_policy="per_target", ) ``` Each target gets an independent clone of every encoder and the final model. Within each training fold, that clone only fits rows observed for that target; supervised encoders never receive hidden target values. Fitted operators expose `target_models_` and `target_counts_`. Predictions still cover every requested row and target. There is no imputation of missing target values. Masks must be boolean and match `y` exactly. Observed cells must be finite; masked cells may contain NaN, infinity or hidden finite values. The JSON codec preserves those values and the mask without invalid JSON numbers. Every target needs observed training rows, and scored target partitions need observed values. Native metrics use only observed cells and retain named per-target metrics; the scalar regression metric is their unweighted mean. A report's `row_count` counts prediction rows, not the number of observed cells for each target. The default `target_policy="complete"` rejects partial masks. Partial regression targets use the early or intermediate `MultimodalRegressor` path, or the development late-fusion branch profile described below with `target_policy="per_target"`. Masked classification remains refused. Group-level score aggregation with partial targets is outside these profiles. ## Encode, fuse and tune `MultimodalRegressor(transformers={source_name: transformer, ...}, model=...)` clones and fits each source transformer on the current training fold. For example, use `StandardScaler` for spectra, `TensorPCA` for raw fixed-shape image or series tensors, and `ColumnTransformer` for mixed metadata. - `fusion="early"` concatenates the resulting numeric encodings. - `fusion="intermediate"` sends a list of encoded blocks to a multiblock model, such as `MBPLS`. Use `standardize=False` if source weights should retain their effect through MBPLS. - Late fusion uses named `by_source` branches, `merge="predictions"` and a downstream model. DAG supplies nested grouped OOF predictions to its meta-model. ```python result = nirs4all.run( [GroupKFold(3), {"model": model}], dataset, engine="dag-ml", tuning={ "engine": "n4m", "sampler": "random", "seed": 17, "n_trials": 8, "space": { "transformers__image__n_components": [2, 4], "source_weights__image": [0.5, 1.0], "model__alpha": [0.1, 1.0], }, "storage": "file:///absolute/study-directory", "study_name": "example", }, save_charts=False, ) result.export("multimodal.n4a") prediction = nirs4all.predict("multimodal.n4a", new_raw_dataset) result.close() ``` The durable profile covers one `MultimodalRegressor` or `MultimodalClassifier`, or a late-fusion pipeline as described below. Topology is fixed. Sequential N4M search accepts `random`, `sobol`, `lhs`, `ternary`, `ga`, `pso`, `cmaes`, `tpe`, and `gp_ei` samplers. An omitted sampler retains the historical `tpe` default. Regression supports complete or partially observed targets and native regression metrics; classification supports accuracy and balanced accuracy, defaulting to maximizing balanced accuracy. Sequential search accepts `median`, `successive_halving`, `asha`, `hyperband`, and `racing` pruning; pruned trials are recorded in both checkpoints and the public tuning result. It rejects queued `force_params`. Regression minimizes RMSE, MSE and MAE, and maximizes R² by default. An explicit `direction` overrides that default. Classification requires non-overlapping validation folds; repeated and overlapping holdouts are rejected before fitting because averaging class indices is not a valid classification reduction. DAG owns trial execution, scores and selection. The winner then uses the existing native CV/refit path. Public tuning results use canonical dotted parameter paths. Trial budget is total, including failed trials, and can be increased on resume. The tuning objective averages metrics across folds. `RunResult.cv_best_score` uses the pooled out-of-fold predictions. These values can differ with unequal fold sizes or nonlinear metrics such as RMSE and balanced accuracy; reports retain both values and their native score evidence. These CV values participate in hyperparameter selection; use the independent test partition to evaluate the selected pipeline. ### Joint Torch intermediate fusion (development) Use the existing `MultimodalRegressor` with a supplied `torch.nn.Module` whose `forward(**inputs)` accepts each source by name. Its encoders and prediction head belong to the same module and train jointly with one Adam optimizer and MSE loss. The source transformers must be `None` or `"passthrough"`; put learned encoders inside your module. ```python model = MultimodalRegressor( transformers={"nir": "passthrough", "clinical": "passthrough"}, model=joint_module, # Your nn.Module implementing forward(nir=..., clinical=...). fusion="intermediate", ) result = nirs4all.run( [GroupKFold(3), {"model": model, "train_params": {"epochs": 20, "batch_size": 16, "lr": 0.001}}], cohort, engine="dag-ml", refit=True, random_state=42, save_charts=False, ) result.export("joint-model.n4a") result.close() prediction = nirs4all.predict("joint-model.n4a", new_cohort) ``` The development profile accepts two to four complete fixed-shape float32 or float64 sources and one complete numeric regression target. Supported IO representations are `signal_1d`, `tabular_numeric`, `gray_image`, `rgb_image`, `mc_image`, `multispectral_image` and `series_mv`. Each tensor retains its raw non-sample axes until the user's `forward(**inputs)` runs: RGB images have shape `(samples, height, width, 3)` and series `(samples, time, variable)`. The host does not flatten, permute channels, pad or resample these inputs. Put any such explicit conversion inside your module. CPU training converts inputs to float32; floating module parameters and buffers must already be float32. Source names are ASCII Python identifiers other than `y`. Source weights, missing-source policies, partial targets, external preprocessing, model-local tuning and generated provider views are outside this profile. Use one model with an optional index or group splitter; omitting the splitter performs native full-training REFIT and supplies no CV evidence. This profile uses the default in-process DAG-ML runtime. Explicit CLI execution is refused before fitting because it does not transport the required source receipts. Each receipt records the real IO buffer, ordered sample IDs and input schema; the controller verifies them before training or prediction. Alternatively, an importable Torch factory marked `framework="pytorch"` receives `input_shapes={name: source.values.shape[1:]}` and returns your module, including all raw image or series axes. Supplied module templates retain their initial weights and are copied for each fold. Native tasks select the training IDs, own CV/refit, and derive each CPU Torch callback seed from the public `random_state` while restoring the caller's RNG. Epochs and patience are bounded by 100, batch size by 1024, and learning rate by `[1e-6, 0.1]`; the adapter also bounds model size and work. Export retains the actual completed REFIT origin, fitted tensor identities, named input contract, complete IO schema and selected owner. Replay checks these before prediction and does not fit again. New observation IDs and per-source row order are allowed through IO alignment. In this first named profile, retain the cohort's source mapping order because the native source IDs are part of the captured contract. This is a trusted Python archive, requiring the user's importable factory or module dependencies and matching installed development builds. It does not provide portable Torch weights for R or WASM. Ragged joint Torch inputs remain unsupported by this profile; use the explicit summary encoder for ragged series. The demonstration arrays are software fixtures. ### Training controls during global search (1.3.3) Since version 1.3.3, the direct multimodal model, each late-fusion branch model and the meta-model can carry fixed `train_params` and `refit_params`: ```python model_step = { "model": model, "train_params": {"model__tol": 0.003}, "refit_params": {"model__alpha": 9.0}, } ``` The example uses a `MultimodalRegressor` containing Ridge; ordinary Ridge branch and meta-model steps use `tol` and `alpha` directly. These are estimator parameter overrides, not arbitrary `fit()` arguments. Controls are validated before fitting or opening a study. Each trial and the winner's CV use `train_params`; terminal refit adds `refit_params`, which take precedence. The resulting fitted predictor and its archive preserve the refit behavior. The same contract applies to isolated parallel candidates. Fixed training controls take precedence over candidate parameters. To search a parameter, leave it out of `train_params` and declare it in `tuning.space`. Changing either control mapping invalidates checkpoint resume before any fit. CV-weight warm starts remain unsupported. The fixed controls above are available in nirs4all 1.3.3 or newer. ### Nested local search (development main) Model-local `finetune_params` can be nested within global search for a direct multimodal model or a `by_source` base model. The local engine must select the Optuna or n4m host search profile. The global engine still selects n4m and addresses the whole model or ensemble through `tuning.space`. For example, let global search choose an image encoder dimension while local search chooses the direct model's Ridge regularization: ```python result = nirs4all.run( [GroupKFold(3), { "model": multimodal_model, "finetune_params": { "engine": "n4m", "sampler": "random", "seed": 19, "approach": "grouped", "eval_mode": "mean", "n_trials": 3, "model_params": {"model__alpha": [0.1, 1.0, 10.0]}, }, "train_params": {"model__tol": 0.003}, "refit_params": {"model__alpha": 9.0}, }], cohort, tuning={"engine": "n4m", "sampler": "random", "seed": 19, "n_trials": 3, "space": {"transformers__image__n_components": [1, 2]}}, engine="dag-ml", refit=True, save_charts=False, ) ``` Every global candidate and native outer training scope receives its own local search. Grouped local search clones the real splitter policy inside that training scope, including group constraints. Native inner tasks fit fresh encoder/preprocessing declarations on inner training rows only; an already fitted outer encoder is never reused to select local parameters. Direct models retain named raw source blocks, while base models consume only their declared source. Terminal refit performs a separate training-only local search; the step's fixed train/refit controls retain their documented precedence. A parameter must have one search owner. Overlapping global paths and local model, trial-training or forced parameters refuse before an optimizer opens; dotted and sklearn double-underscore paths name the same parameter. Changing the local configuration, upstream recipe, splitter, controls or training content invalidates global resume. Export retains the fitted selected predictor and its evidence; archive prediction performs no HPO or fit. Meta-model local HPO remains refused because searching an already computed OOF matrix would expose inner selection targets. This nested profile also refuses generated views, augmentation, feature/prediction joins and all-observation preprocessing policies, incomplete sources/targets and missing-source policies. The closed `engine="native"` profile and CV-weight warm starts keep their separate capability restrictions. This nested composition is available on development `main`; it is not part of the published 1.3.3 fixed-control qualification. `tuning.progress_callback(event)` may return `False` between trials. Cancellation raises `MultimodalTuningStopped`; a model failure is checkpointed and its original error propagates. `resume=True` continues from terminal history. A crash during an unfinished trial returns to the preceding checkpoint and may repeat that unfinished trial. Changing training buffers, labels, schemas, folds or search semantics refuses resume. Target names, task type and training masks are part of the search identity. Source presence is also fingerprinted. Hidden target/source values and held-out test data are excluded from search; modifying them does not create a new search objective. For the native multimodal N4M random-search path, set `tuning.n_jobs=2` or another positive worker count to evaluate independent candidates concurrently. Specify `sampler="random"` for new parallel searches; explicit nonrandom samplers and pruning require `n_jobs=1`. Each candidate owns a Python process, its provider snapshot, fitted models and Python/NumPy RNG. `n_jobs=1` retains the existing sequential checkpoint identity. Parallel checkpoints may contain proposals already opened for other candidates; `resume=True` reuses those proposals and verifies saved generated views before new evaluation. This setting does not enable parallelism in the general `PipelineObjective` tuning adapter. For generated views it requires a DAG-ML binding that advertises candidate-local generated views, whether the outer job runs in-process or in an isolated Python worker. Older installed bindings fail before the provider `PLAN`. For generated views with `N4A_DAGML_INPROCESS=0`, HPO uses an isolated Python worker. `tuning.progress_callback` runs in the calling process after each checkpoint event; returning `False` stops at the saved trial boundary, and returning `True` or `None` continues. `resume=True` continues from that checkpoint. The parent's `should_stop` may terminate the worker during a trial; the last complete paired checkpoint remains available, and the interrupted trial may repeat on resume. The host resets the Python/NumPy RNG for each native task using a stable seed derived from `run(random_state=...)`, otherwise the tuning seed, otherwise zero. This makes supported sklearn operators using the global RNG reproducible across resume even when a progress callback consumes randomness. Explicit operator seeds remain effective. Custom generators and GPU determinism are not guaranteed by this policy. ## Select independent source outputs When a `by_source` pipeline exposes an independent prediction output per source, native selection ranks candidate recipes using the first source in dataset order, signed as `selection_output_id="output:source_0"`. All source outputs, validation scores and OOF arrays are retained. Other source scores do not implicitly become a fused ranking metric; an explicit fusion model has its own output and selection evidence. Exactly equal selection scores use the original variant ID as the deterministic tie break. Ranked refits retain this same native ordering and their own genuine captures. ## Tune a late-fusion ensemble Use the existing `by_source` branches followed by `{"merge": "predictions"}` and a meta-model. Pass a global `tuning` mapping to `nirs4all.run()`: ```python tuning = { "engine": "n4m", "sampler": "random", "seed": 17, "n_trials": 4, "space": { "branches.image.0.n_components": [2, 4], "branches.nir.1.alpha": [0.1, 1.0], "branches.metadata.0.numeric.with_mean": [False, True], "meta.alpha": [0.1, 1.0], }, "storage": "file:///absolute/late-study", "study_name": "late-fusion", } ``` `branches...` addresses the named source and zero-based position in its public step list. Positions include `None` entries; those entries themselves cannot be tuned. `meta.` addresses the final estimator. Nested sklearn parameters accept dots or `__`, as with `branches.metadata.0.numeric__with_mean`. The example assumes a PCA at image position 0, Ridge at NIRS position 1 and a metadata `ColumnTransformer` with a transformer named `numeric` at position 0. Every candidate recomputes all branch encoders, base models and inner OOF predictions through the native scheduler. The meta-model fits those inner predictions and is scored on the outer validation folds. Training and test identities remain separate. Search paths and their destination nodes are part of the checkpoint identity; changing their routing refuses resume. The returned `RunResult` represents the selected **ensemble**. Its direct `export()` saves that ensemble even if an individual base model scores better. The profile requires instantiated sklearn operators. Sources and targets must be complete unless the branch policies below explicitly declare missingness. Development `main` also accepts the fixed model training controls described above; base-model local `finetune_params` may be nested in this global search profile. Meta-model local HPO remains refused. Single-target classification and complete multi-target regression use the same graph. The synthetic U10 example writes the recipe, cohort, checkpoint, report and archive, then predicts twelve new observations with fit forbidden: ```bash python examples/user/02_data_handling/U10_multimodal_late_tuning.py --output /tmp/mm-late --stop-after 2 python examples/user/02_data_handling/U10_multimodal_late_tuning.py --output /tmp/mm-late --resume ``` ## Late fusion with absent modalities and partial regression targets Declare `missing_source_policy="zero_with_indicator"` on the named source branch: ```python pipeline = [ GroupKFold(3), {"branch": { "by_source": True, "missing_source_policy": "zero_with_indicator", "steps": { "nir": [StandardScaler(), Ridge()], "series": [SequenceSummary(), StandardScaler(), Ridge()], }, }}, {"merge": "predictions"}, Ridge(), ] ``` Every branch encoder and base model fits only the present rows of its current native training scope, including the inner folds that supply meta-model inputs. The meta-model retains every OOF observation ID. Each branch contributes its predictions followed by one presence indicator. For an absent source, prediction columns are zero in the original target space and the indicator is zero; that source's encoders and predictor are not called. The indicator remains a column when every source observation is present. Two single-target branches therefore produce four meta-model features. Complete multiple targets are also supported. Zeros encode an absent branch for the meta-model; they do not reconstruct raw measurements. Native meta-model scores cover the requested scored rows; base-model predictions remain restricted to observations actually present for that source and observed for that target. An entirely unobserved source in any inner, outer or final training scope is rejected before model callbacks, rather than borrowing fitted state from another fold. The development profile also accepts complete single-target classification and partially observed regression targets. For partial targets, add `target_policy="per_target"` beside the branch's `missing_source_policy`; each target gets a fresh encoder/model fitted on the intersection of present source rows and observed target rows in that actual outer, inner or REFIT scope. Classification requires classifiers exposing `predict_proba`. Each branch supplies all K columns in the shared training vocabulary order, followed by its presence indicator. An absent observation supplies no class distribution; the native joined features retain explicit validity and presence masks. Every true training scope must contain the complete class vocabulary. The native scheduler validates availability before model callbacks or global HPO. This partial late profile is serial, with two to four named branches and a single meta-model. Global whole-stack HPO is supported; branch-local HPO with partial observations, external preprocessing before the source branch, augmentation, generated views, separate target transformations and already fitted preprocessing are refused. Classification targets must remain complete. See [the partial late-fusion guide](multimodal_late_partial.md) for the public pipeline declaration and the archive contract. At prediction, presence patterns and series lengths may change. An entire series modality can be absent: retain its declared channels, dtype, time unit and coordinate schema in a zero-row `RaggedSeriesSource`, and use `source_alignment="left"`. The archive records and validates source order, prediction widths, indicators and the fitted meta-model width. As with other ragged inputs, `SequenceSummary` uses equal weights for time points and ignores time coordinates; per-cell masks are unsupported. U13 generates 24 training and eight test observations, with separate groups, dense NIR and partially absent variable-length series. Its three native trials tune `branches.series.0.include_length`, `branches.nir.1.alpha` and `meta.alpha`: ```bash python examples/user/02_data_handling/U13_multimodal_late_missing_sources.py --output /tmp/mm-late-missing --stop-after 1 python examples/user/02_data_handling/U13_multimodal_late_missing_sources.py --output /tmp/mm-late-missing --resume ``` Use a fresh directory and omit both flags for a continuous run. The example writes provider/pipeline/tuning recipes, typed datasets, durable checkpoints, `late-fusion.n4a` and JSON/text reports. On resume it checks that the completed prefix is retained and only remaining trial numbers are reported as new. It then replays eight new observations with some series absent and eight with all series absent, while fitting and provider materialization are forbidden. The all-absent case additionally forbids calling the series encoder. The standalone prediction datasets and report retain both sets of predictions. ## Executable synthetic data providers Pass an IO `DataProvider` directly as `dataset`. DAG-ML executes its source node once in `PLAN`, before constructing cross-validation folds. The callback returns an IO `MultimodalDataset`; IO validates raw shapes, identities and alignment. The provider does not learn from observations. Learned augmentation belongs in a fold-scoped training controller. ```python from nirs4all_io import DataProvider def generate_cohort(*, seed, params, context): # Reuse a scientific generator and return aligned raw TensorSource objects # inside a MultimodalDataset, with explicit groups and partitions. return make_synthetic_cohort(seed=seed, n_samples=params["n_samples"]) provider = DataProvider( generate_cohort, provider_id="my.synthetic.cohort", provider_version="1", params={"n_samples": 64}, seed=17, ) result = nirs4all.run(pipeline, provider, save_charts=False) ``` The provider's root seed is separate from the model's `random_state`. Native PLAN derives and records the effective task seed passed to the callback. `provider.materialize()` is also available outside a run and uses the provider's root seed directly; pass an explicit `seed` to reproduce a native realization. Callbacks must use their seed and parameters deterministically. Change `provider_version` when changing the recipe's implementation. For partial production, supply `base=fixed_cohort` and return a mapping with `sample_ids` plus `sources` and/or `y`. IO aligns it to the base's fixed universe. Adding sources is allowed; replacing existing sources or targets requires `replace_sources=("image",)` or `replace_targets=True`. Groups and partitions remain explicit and the base is not mutated. After materialization, `provider.get(sample_ids)` and `provider.batches(batch_size, sample_ids=training_ids)` return aligned cohort views without calling the generator again. A batch iterator retains its cursor: ```python batches = provider.batches(16) first = next(batches) state = batches.state_dict() # JSON-serializable restored = DataProvider(generate_cohort, provider_id="my.synthetic.cohort", provider_version="1", params={"n_samples": 64}, seed=17) restored.load_state_dict(state["provider"]) remaining = restored.batches(16) remaining.load_state_dict(state) ``` Restoring the provider explicitly regenerates and verifies the recorded content before publishing it. Restoring the iterator checks the recipe, content, selection and batch configuration before accepting its cursor. This is a finite, materialized provider: batching does not make generation itself out-of-core. One run uses one fixed realization across folds and HPO trials; a changed recipe invalidates HPO resume. Dynamic row creation, learned generators, infinite streams and native per-epoch regeneration are outside this profile. A model controller may iterate the fixed cohort for its epochs. `nirs4all_io.provider_adapters.SklearnProviderAdapter(provider, source="nir")` exposes `.arrays(training_ids)` as `(X, y)` for an ordinary estimator's `fit`. Its `.batches(...)` supplies arrays for caller-managed `partial_fit` when the estimator actually provides that method. N-D sources require an explicit encoder; omitting `source` returns a mapping for source-aware estimators. The optional `TorchMapDataset` and `TorchIterableDataset` classes in the same module implement PyTorch's real dataset interfaces. Pass `sample_ids=training_ids` to constrain a loader to a fold. Iterable workers shard sample positions without duplication. `return_metadata=True` retains IDs, groups, partitions and masks; tuple mode refuses to discard missingness masks. Use `collate_fn=collate_provider_samples` for mixed categorical/numeric sources. Adapters capture the realized cohort, not the generator callback. Their dataset values are independent of worker count; iterable delivery order may differ. The provider cursor checkpoint does not capture DataLoader prefetch queues or a model/optimizer checkpoint. PyTorch is imported only when explicitly requested. The source recipe, effective seed, native lineage and content fingerprint are retained in run metadata and exported host models. Prediction uses the fitted model and explicitly supplied new inputs; it never executes the training provider. U11 demonstrates full and partial production, reuses the existing NIRS synthesis API, restores a batch cursor and exports/replays a model: ```bash python examples/user/02_data_handling/U11_multimodal_data_provider.py --output /tmp/mm-provider # With the optional CPU/GPU PyTorch dependency installed: python examples/user/02_data_handling/U11_multimodal_data_provider.py --output /tmp/mm-provider-torch --torch-workers 2 ``` ### Generated feature views per fold For a bounded native CV run, `generate_view` can replace declared raw sources after `PLAN`. The base cohort fixes IDs, targets, groups, partitions and source schemas before splitting. The callback receives only the scheduler-selected IDs and its deterministic view seed; return the same IDs in the same order. For example, given a fully aligned `cohort` and a source-aware concrete `model` (such as `MultimodalRegressor`): ```python import numpy as np import nirs4all from nirs4all_io import DataProvider, TensorSource from sklearn.model_selection import GroupKFold def prepare_nir(**_): return {"sample_ids": list(cohort.sample_ids), "sources": {"nir": cohort.sources["nir"]}} def nir_for_view(*, sample_ids, seed, **_): source = cohort.take(sample_ids).sources["nir"] values = np.asarray(source.values) # A deterministic fixture perturbation; replace with your own provider. values = values + np.random.default_rng(seed).normal(0, 0.001, values.shape) return {"sample_ids": sample_ids, "sources": {"nir": TensorSource( values, sample_ids, representation_id=source.representation_id, axis_units=source.axis_units, axis_coordinates=source.axis_coordinates, )}} provider = DataProvider(prepare_nir, generate_view=nir_for_view, provider_id="example.fold-nir", base=cohort, replace_sources=["nir"]) result = nirs4all.run([GroupKFold(3), {"model": model}], provider, engine="dag-ml", refit=True, save_artifacts=False, save_charts=False, random_state=19) try: print(result.best_rmse) finally: result.close() ``` The qualified profile also accepts exact `KFold`, `StratifiedKFold` and `StratifiedGroupKFold` splitters, with a single concrete estimator. A concrete sklearn `Pipeline` may learn preprocessing inside that estimator on generated training views. With **one complete raw source**, one concrete sklearn-style transformer may instead sit between the splitter and model as a separate DAG node. The node fits on the generated fold-train or full-train view; its fitted state is carried into that fold's model and, for a qualified multimodal refit, into the prediction archive. A shared transformer over multiple heterogeneous raw sources is refused before fitting; use source-specific encoders inside the multimodal model in that case. The existing `by_source` branch with a distinct preprocessing list for each named source, followed by `{"merge": {"sources": "concat"}}` and a concrete model, also fits each source chain on its generated fold-train view. This branch uses one native model task to attest all input blocks. Its refitted source chains can be exported in a prediction-only `.n4a` archive when the downstream model is a sklearn-style regressor, then replayed on a new explicit multimodal cohort, as can the source encoders inside `MultimodalRegressor` or `MultimodalClassifier`. Ragged series can be replaced as long as their declared schema is retained. For `MultimodalRegressor` or `MultimodalClassifier`, the **model-only** profile also supports N4M tuning (`engine="n4m"`) with the sequential samplers and pruners described above, or parallel random search, with a paired native/optimizer checkpoint. The checkpoint contains a manifest of each completed trial's generated views. On resume, the provider regenerates and compares every saved view before another trial starts; changing the recipe or the generated content refuses the resume. Use a stable `storage` URI and `study_name`, then call `run` again with `resume=True` and the same provider recipe. A completed search refits the selected model on generated training views. For one fitted `MultimodalRegressor` or `MultimodalClassifier`, `result.export()` now writes a prediction-only `.n4a` containing the selected REFIT model, its input schema and the verified generated-view manifest. Supply a **new explicit multimodal cohort** to `nirs4all.predict(archive, new_cohort)`; prediction does not call the training provider or fit a model. Its sources must match the training schema, including names, shapes, axes, units and feature names. The archive does not regenerate training data and cannot be used for retraining. Other generated-view estimator shapes still refuse export. Set `N4A_DAGML_INPROCESS=0` to run the qualified CV/refit profile in an isolated Python worker: it receives the completed `PLAN` provider, creates its own view store, and returns the native view manifest and fitted refit artifacts. The provider callbacks must be serializable by `cloudpickle`, and that interpreter must have the DAG-ML Python binding installed. This path does not use `dag-ml-cli`. The parent's `should_stop` callback can cancel the worker; no completed run or refit archive is published in that case. **Generated-view HPO** also runs in a Python worker when `N4A_DAGML_INPROCESS=0`, including `tuning.n_jobs>1` for the qualified N4M random-search profile. The worker owns its trial views, optimizer and selected refit; parallel candidates each run in a separate child process. It returns a usable `RunResult` for archive export, and a paired checkpoint can be resumed in a later worker. The parent stops active candidate processes when `should_stop` cancels the worker. `tuning.progress_callback` is relayed to the calling process, including its stop decision; a callback exception propagates to the caller and stops the worker. The nested parallel worker path has been qualified from installed candidate wheels on Linux, Windows x64 and macOS ARM. This does not qualify Studio installers on those platforms. Separate transform nodes on multiple raw sources remain outside this profile. Multimodal synthetic data in this example are test fixtures, not a new product generator. For a complete runnable example of fold views, HPO and prediction-only replay: ```bash python examples/user/02_data_handling/U14_multimodal_fold_provider.py --output /tmp/mm-fold-provider python examples/user/02_data_handling/U14_multimodal_fold_provider.py --output /tmp/mm-fold-provider-hpo --hpo --n-jobs 2 --worker ``` ## Archive and scope The archive contains fitted encoders, fusion and final model, raw input contracts, selected recipe and exact host dependency versions. Replay accepts new observation IDs and source mapping order but rejects source, dimension, dtype, feature name, axis, coordinate or unit changes. It performs the DAG PREDICT phase without fit and does not require the training workspace. These are Python host archives containing trusted joblib objects, not portable Core/ONNX/WASM model archives. Replay requires the declared package versions and Python major/minor version. Dense inputs have fixed trailing dimensions, and temporal sources may use the typed ragged-series contract with variable-length observations. Both forms support the source and target masks described above. Cross-language ragged replay is not qualified. Classification encodings and inferred task types depend only on training targets; a held-out class absent from that vocabulary is refused. See the [installation qualification](../../../multimodal_installation_qualification.md) for the independent wheel installation and replay with fit and source-directory access forbidden, and `scripts/verify_multimodal_archive.py` to repeat that check. The [targets and missing-source qualification](../../../multimodal_targets_and_missing_sources_qualification.md) records the U08/U09 checks and their current validation status. The [late-fusion tuning qualification](../../../multimodal_late_tuning_qualification.md) records U10, durable whole-stack search and its independent installation proof.