Native V1 migration and compatibility guide

Warning

Version 1.0.0 is published and is the compatibility baseline. Corrective V1 qualification is in progress; this page distinguishes the general Python API from the explicitly selected portable profile. A supported portable subset alone does not establish general Python or Studio feature parity.

Runtime boundary

The R3/R4 product architecture is unchanged:

• the general Python run() API chooses a DAG-ML execution profile before execution: portable requests use native; other requests use dag-ml;

• an explicit engine= or N4A_ENGINE selector is respected unchanged;

• the explicit engine="native" portable profile fails closed; no execution failure is retried through another engine;

• engine="legacy" remains available only when a direct Python caller selects the rollback profile explicitly, through the end of R4;

• Studio owns HTTP, WebSocket, store, jobs and scheduling in Rust;

• Studio may package CPython only as a bounded JSON-stdio host for an explicit library or plugin after capability preflight. CPython owns no listener, scheduler, store, job lifecycle or fallback;

• Studio, Web and strict Python product paths cannot select the legacy engine.

Installing Python is therefore the normal way to use the Python library and its optional ML integrations. It is not an instruction to install a Python backend for Studio.

Migrate a workspace

Migration is an explicit, one-way operation in the separately published nirs4all-tools 0.0.7 package. The runtime never starts it automatically and never writes a legacy source.

python -m venv n4a-migration
n4a-migration/bin/python -m pip install \
  "nirs4all-tools[duckdb,parquet]==0.0.7"

n4a-migration/bin/nirs4all-tools workspace inspect /data/workspace
n4a-migration/bin/nirs4all-tools workspace convert /data/workspace \
  --output /data/workspace-v2 --verify

The source path, inode and bytes remain unchanged. Keep the source and output at different paths. Exit code 0 means a clean verified conversion; 10 means that unsupported content was preserved opaque and needs review; 20 means unsupported input or strict refusal. There is no native-to-legacy reverse conversion.

workspace convert does not create a fitted predictor from historical result metadata or prediction arrays. Those outputs can be inspected after conversion, but they are not a model archive: load_session() expects a qualified .n4a bundle, and new-sample predict() requires a fitted model artifact. A conversion report showing preserved predictions must not be read as proof that prediction without retraining is available for that source profile. Passing the converted workspace directory to nirs4all.predict(model=..., ...) now returns an explicit converted_workspace_has_no_fitted_model capability error for both the default DAG-ML engine and explicit legacy engine. Provide a separately qualified fitted .n4a archive, or retrain before predicting new samples. This refusal is qualified on the synthetic legacy save fixture; opaque preserved bundles need their own qualification.

The authoritative read/write/migrate dispositions and retention window are in the Tools support matrix and support SLA.

Public API examples

An ordinary general Python run does not need an engine selector. The actual engine is available as result.execution_engine and in result.per_dataset. For a strict portable request, select engine="native" explicitly. Its current training contract requires a list pipeline, a splitter, a supported Methods model and dataset={"X": X, "y": y, "sample_ids": ids}. Chart generation, workspace/project/cache integration and broader host operators are separate general-profile capabilities, not portable archive guarantees.

An omitted save_charts uses the selected profile’s default (false for portable, true for general DAG-ML/explicit legacy). Explicit options are never silently discarded. Both profiles accept verbose=0..3.

The general profile restores concrete pipelines without a splitter through a single native REFIT phase: all training rows are fitted once, and test rows are never fitted. There is no cross-validation score (cv_best_score is NaN). When a test partition exists, the historical validation alias points to this test partition, with a warning and persisted evaluation metadata; it is not an independent model-selection holdout. Generator/HPO variants of this no-splitter profile remain under qualification.

Concrete pipeline chart commands are presentation-only. Spectra after a transform are generated from the captured full-training refit transform, without a new fit; they are explicitly not out-of-fold features. Saved charts include an adjacent HTML description, exact numeric CSV inputs, and scored fold membership JSON under workspace/charts. Visible-only charts print a text alternative. Branch/source/augmentation stage snapshots not yet captured are refused explicitly rather than rendered from a misleading raw substitute; this remains an open general-profile parity gap.

Captured general-profile .n4a models replay through a native DAG PREDICT phase with a Python scientific host. Public predict() and trained/loaded Session.predict() do not train again; their metadata records the selected engine, input cohort, source artifact fingerprint and training_performed=False. Inference does not manufacture validation scores. An explicit engine="native" still requires a portable Core archive and never silently loads a host model. General archives contain trusted Python/joblib objects: load them only from a trusted producer. Recorded SHA-256 digests detect corruption, not malicious producers. Older DAG exports without a recorded digest retain a visible warning and do not claim verified integrity. Loaded Sessions are bound to their source archive fingerprint and refuse replacement after loading.

General retrain(mode="full") also accepts a recorded workspace prediction dictionary, or a captured-winner archive without train_pipeline.json. It clones the selected estimator, its preprocessing pipeline and target transform, then fits them on all new training rows once through DAG-ML. It does not repeat the original generator/HPO search or reuse learned estimator objects; frozen clones retaining source estimators are refused. There is no new CV score for this full-training operation. Lineage records the source fingerprint, integrity status and training contract. Existing concrete bundles with a training specification retain their published specification-replay behavior, including its explicit splitter. Prediction-only fusion wrappers and fine-tuning remain separate unqualified retrain paths.

General retrain(mode="transfer") accepts those captured workspace predictors and host archives, including archives with a training specification. It freezes the captured X/y preprocessing in private copies and trains only a fresh final model on the new training rows, optionally replacing it with new_model. It never replays the original search, refits the captured transforms or invents a CV score. Export/replay retain that frozen preprocessing, and lineage marks its deliberate state reuse separately from the fresh model. A subsequent full retrain removes these transfer freezes and trains all steps afresh. The explicit portable native selector and the historical opt-in transfer plugin keep their separate contracts; neither is an implicit fallback from this DAG host path.

When Studio supplies store_run_id, the library attaches its pipeline results to that existing running workspace run without completing or failing the parent run; Studio retains that lifecycle. should_stop is cooperative cancellation: checked before execution, between native scientific tasks, and before publishing results. It does not interrupt an individual BLAS/estimator fit mid-call.

General sklearn-compatible model steps support train_params estimator overrides and refit_params overrides on top of them during REFIT only. These follow the historical controller’s set_params contract, not arbitrary fit(**kwargs). Controller verbose emits training diagnostics without overwriting the estimator’s verbosity setting. Unknown keys are diagnosed; the historical controller’s silent ignoring of them is not reproduced. Overrides participate in every inner HPO evaluation as well as the final fit; provenance distinguishes proposed parameters from effective parameters when an override replaces a searched value. CV preprocessing remains fold-local. For {"model": ..., "na_policy": "replace", "fill_value": v} and {"preprocessing": ..., "na_policy": "replace", "fill_value": v}, the general DAG-ML profile inserts a stateless transform immediately before the declared step. It replaces NaNs in both training and later prediction data, and the transform is retained in exported .n4a predictors. This step-local replacement is equivalent to filling the input at that point in the pipeline; other preprocessing modifiers still require separate qualification. Simple CV/full-training, export/replay, HPO, and native nested stacking controls are qualified. A closed SGD CV-weight initialization profile supports an explicitly named native fold and compatible full-train REFIT through public sklearn coefficient/intercept initializers. It resets optimization counters and does not enable general estimator or deep-learning warm starts. Specialized GPU reset, fit-influence/AOM policies, and the old by-source prediction-stacking route still require separate scientific validation.

General model-local finetune_params with approach="single", "grouped", or "individual" can use the host Optuna profile (engine="optuna" inside finetune_params, or inferred from historical sampler/trial controls). Optuna proposes parameters; DAG owns the bounded trial loop, native candidate scores and selection. Each outer training scope gets its own inner parameter search, and the final REFIT gets a separate search on the full training partition. Single/individual searches use a training-only 80/20 holdout, with separate studies per outer fold. Grouped search clones the configured splitter inside each outer-training scope, including its explicit group constraints; its candidate parameters are shared across those inner folds. Without a configured splitter, historical grouped search uses the same training-only holdout. Preprocessing and target transforms are fitted inside each inner training split. This fixes the earlier global-preprocessing/global-parameter leakage, so selected parameters and CV values need not equal that historical implementation.

Development main supports the same scoped host search inside global multimodal tuning for direct models and source-bound base models. A local subcampaign reconstructs unfitted upstream declarations with the current global candidate parameters; it never uses an outer fitted transform to select local parameters. Direct source blocks and branch source bindings stay typed and identity-aligned. Overlapping global/local parameter owners refuse before opening an optimizer, and global checkpoints bind the local search configuration and graph recipe. Meta-model local HPO remains refused: selecting parameters from a precomputed OOF matrix would leak inner selection targets. This development feature does not expand the closed portable native profiles or enable CV-weight warm starts.

Trial counts, seeds, the installed optimizer’s sampler grammar and verbosity are honored. Its historical Grid-to-TPE conversion for noncategorical ranges remains visible; provenance records the effective sampler. General exports retain the searches and the selected fitted predictor, not a resumable Optuna study or a Methods checkpoint. They remain trusted Python artifacts, not portable Core packages. Grouped best, mean (alias avg) and robust_best reduce the native per-fold scores explicitly: mean RMSE is not pooled OOF RMSE. Best honors the metric direction; robust_best never suppresses a failed trial or invents a penalty score. Source rows and the inner splitter remain in the search provenance. Progressive pruning, parallel trials, persistent studies and multi-phase host searches still require qualification; deterministic DAG parameter grids keep their separate contract.

Classification repetition voting preserves the native raw/CV selection scores. Each fitted CV/REFIT task captures its real class probabilities, when the classifier provides them, with explicit class and sample identities. The library’s existing ensemble and repetition aggregators produce the additional train/test/aggregate presentation rows without fitting again. These rows carry classification_evidence or aggregate_evidence metadata; their display scores are not new native selection reports. A classifier without predict_proba uses its real hard predictions, never fabricated one-hot probabilities. In subprocess execution the same evidence travels in a host-only capture sidecar, separate from the coordinator’s native result protocol.

Directory-backed Python models such as AutoGluon use a host-only archive extension. Native result artifacts and general .n4a bundles keep a small joblib model reference and store each predictor file under host_artifacts/. The manifest records every relative filename, byte length and SHA-256 digest. The reader verifies all declared files before deserializing the model and rejects missing, extra or changed archive sidecars. Files are copied and hashed in bounded chunks; the 512 MiB inline joblib limit does not apply to their combined size. Earlier archives with an inline predictor directory still load. These files remain trusted Python host artifacts, not portable Core packages.

General generate() and its convenience methods select the installed nirs4all.python.synthesis.v1 library adapter when no engine/plugin selector is supplied or engine="dag-ml" is requested. This reuses the scientific synthesis builder, without a legacy ML runner, HTTP backend or retry. generate.preflight() exposes that host contract; explicit engine="native" remains strict and unsupported for synthesis. An explicitly requested unknown plugin is not substituted.

General explain() similarly selects nirs4all.python.shap.v1 for the default or explicit engine="dag-ml" route. It accepts a captured general .n4a, a trained DAG result, or a persisted result.best selection and explains the full REFIT predictor without training it again. Preprocessing and inverse target transforms remain inside that predictor. output_index selects one target or class-probability column (default 0); the returned two-dimensional SHAP values describe that output, not all classes. Kernel SHAP operates on the complete input predictor and can be expensive on wide spectra: n_samples limits background rows, while sample_indices chooses the rows to explain. Recorded headers and relation aggregation levels are preserved rather than labelling an aggregate as a raw observation. CSV values, provenance and an HTML table are written before optional figures; visualizations=[] disables figures. This does not claim portable Core SHAP or saved individual CV-fold models. Explicit engine="native" remains strict.

For a qualified request:

import nirs4all

result = nirs4all.run(pipeline=pipeline, dataset=dataset)
result.export("model.n4a")

prediction = nirs4all.predict("model.n4a", new_data)
with nirs4all.session() as session:
    second = nirs4all.run(pipeline=other_pipeline, dataset=dataset, session=session)

Unsupported native shapes are rejected during preflight; they are not retried through legacy. The rollback path is deliberately different and visible:

rollback_result = nirs4all.run(
    pipeline=pipeline,
    dataset=dataset,
    engine="legacy",
)

Do not put that selector in Studio or Web configuration. See the native capability preflight and public interfaces references for the operation-by-operation capability contract.

ABI and schema identities

Surface	V1 candidate identity	Compatibility rule
Methods C ABI	libn4m ABI 2.5.0, SONAME major 2	Call n4m_check_abi_compatibility before other ABI calls; bindings remain thin.
Methods model payload	N4MM format 1 and 2	A fitted-model payload, not a complete .n4a pipeline archive.
Portable model/run archive	Core Archive V2	Validate format, bounds, Methods identity and fingerprint before replay; no refit or legacy fallback on import.
Workspace	nirs4all-workspace-v2	SQLite metadata plus Parquet array sidecars; legacy stores require explicit Tools conversion.
Runtime envelopes	rt_run_request.v1, rt_result.v1, rt_error.v1	Producers and consumers use the frozen JSON schemas; no product-specific reinterpretation.
Studio product contract	Studio V1 HTTP/OpenAPI/WS snapshots	Rust is the sole control-plane owner; plugin-host responses cross bounded JSON stdio only.

The Methods ABI reference and the ecosystem runtime schemas are the lower-level authorities. This guide does not redefine them.

Capability summary

Request	Strict product behavior
run, predict, session, save/load/export	Native for the qualified V1 matrix; general DAG-ML sessions and captured host archives support training and replay. Unsupported shapes refuse before significant work.
Full retrain	DAG-ML for concrete archive specs and captured trainable winners (workspace prediction or archive); other modes remain under qualification.
Transfer	DAG-ML for captured trainable host winners; the explicit historical Python-library plugin retains its separate preflight.
Model-local finetune_params, train_params, refit_params	Public DAG-ML execution supports deterministic generation and scoped Optuna/n4m host search, with training/refit controls. The closed Archive V2 native profile has narrower capabilities; unsupported requests refuse before execution. See pipeline keywords.
Continuing from trained weights, unavailable explain/generate shapes	Explicit refusal for unsupported requests; no implicit legacy execution.
Existing Python workflow during rollback	Direct Python call with explicit engine="legacy"; never Studio/Web.

FAQ

Does Studio need Python?

Not as a backend. Studio’s HTTP, WebSocket, store, jobs and scheduler are Rust. A packaged CPython may exist only for an explicitly selected, preflighted library/plugin call over bounded stdio.

Can Studio fall back to the old backend?

No. An unavailable native or plugin capability returns a typed refusal. The explicit legacy rollback profile is a direct Python-library feature only.

Can I open an old workspace in place?

No. Inspect it with Tools, convert to a new disjoint directory, verify the output, and retain the original for rollback.

Is exit code 10 a clean conversion?

No. It means the command completed but preserved unsupported content opaque. Read the migration and unsupported-content reports before using the output.

How long are legacy readers supported?

The Tools 0.x reader line is guaranteed through both complete releases after the R2 flip: R3 and R4. Retirement is post-V1 only, after an announced governance decision and migration notice.

Does this page mean V1 is available?

No. R4 remains held until the product lock and remaining release gates are green. Published component packages and passing candidate tests do not amount to a stable V1 release.