Native tuning and conformal calibration

This page documents the current native Python surface for combining model tuning with split conformal intervals. It is intentionally narrower than the historical finetune_params path: unsupported shapes fail closed instead of silently falling back to a different engine.

Use this path when you already have explicit arrays for development, scoring, calibration, and prediction cohorts, and you want the tuning trace plus conformal calibration artifact persisted in a nirs4all workspace.

For a runnable smoke-tested script that chains the current public subset end to end, see examples/user/04_models/U09_native_tuning_conformal.py. It runs native tuning, final conformal calibration, calibrated prediction, robustness reporting, workspace reload, and robustness artifact publication on small in-memory arrays. For the same lane with a spectroscopy-standard PLS model, see examples/user/04_models/U10_native_pls_conformal_robustness.py; it tunes PLSRegression.n_components through the public model.n_components search space path and then runs the same conformal and robustness audit stages.

What is native today

The supported public subset is:

• nirs4all.run(..., engine="dag-ml", tuning={...})

• one final sklearn-like estimator, optionally preceded by a linear chain of sklearn-like transformers;

• explicit array datasets: (X, y), [X, y], (X, y, sample_ids, groups, metadata), [X, y, sample_ids, groups, metadata], or {"X": X, "y": y};

• explicit dataset-backed mappings with a selector for fit, scoring, winner and calibration_data.dataset, using SpectroDataset, DatasetConfigs, dataset config mappings, or config/path strings;

• linear pipeline wrappers {"steps": [...]} and the legacy public alias {"pipeline": [...]} for the same supported step sequence; do not provide both keys in one wrapper;

• explicit tuning.score_data for objective scoring, as a mapping or tuple/list;

• Optuna/n4m adapter execution through the native tuning seam;

• optional tuning.winner to project a terminal prediction entry;

• optional top-level calibration or nested tuning.calibration to calibrate conformal intervals from that projected winner;

• workspace persistence for both TuningResult and CalibratedRunResult.

The following remain intentionally closed until their gates are implemented: bare/implicit dataset loaders for native tuning, implicit splitting, branch/merge graphs, model aliases, non-linear preprocessing graphs, and bit-exact resume from an interrupted optimizer checkpoint.

Native keyword/effect quick map

Use this table when wiring notebooks, CI, bindings, Studio forms, or generated configuration into the native Python runtime. It separates the public syntax from the effect of that syntax and from the evidence that downstream tools can inspect without parsing private metadata.

Public syntax	Runtime effect	Published evidence	Fail-closed boundary
run(engine="dag-ml")	Selects the DAG-ML execution backend for the current run.	RunResult metadata and backend traces.	It is not an optimizer selector; run(engine="dual") is a separate bounded parity oracle.
run(tuning={...}) / NativeTuning(...)	Runs the fixed-topology native tuning subset over one estimator or a linear sklearn-like chain.	RunResult.tuning_result, tuning_id, tuning_best_params, tuning_best_value, and optional workspace TuningResult.	Branch/merge graphs, implicit splits, aliases, non-linear graphs, and unsupported datasets fail before execution.
run.tuning.space	Defines the ordered candidate parameter contract.	nirs4all.tuning.ordered_search_space, SearchSpaceParameter, ParameterPatch, and search_space_fingerprint.	Values must stay TCV1 JSON-native and fingerprintable; Python objects, bytes, NaN/Infinity and duplicate canonical paths are rejected.
run.tuning.force_params	Enqueues a caller-provided warm-start assignment as the first optimizer trial.	First TrialResult.params, decoded best_params when it wins, and the same search_space_fingerprint.	Keys must exist in run.tuning.space; categorical values use public decoded values, not backend labels.
run.tuning.score_data	Scores candidate trials on an explicit HPO objective cohort.	Trial scores, trial diagnostics, and TuningResult.summary_artifact().	It is never reused as final conformal calibration evidence.
run.tuning.score_data.conformal_calibration plus conformal_coverage	Fits a temporary conformal calibrator per candidate only for conformal-aware objective metrics.	Trial diagnostics include score_family="conformal" and final_calibration_scope="unmodified_by_score_data".	The temporary calibrator is discarded after scoring and cannot renew the final guarantee.
run.tuning.winner	Supplies or selects the cohort used to project the terminal winner.	RunResult.best, terminal prediction metadata, and tuning_calibration_source when final calibration is requested.	Missing row-aligned identities or mixed dataset/array payloads fail before calibration.
run(..., calibration=...) or run.tuning.calibration	Fits final split-conformal intervals from the projected winner and applies them to provided prediction values.	CalibratedRunResult, PredictResult.intervals, conformal_guarantee_status, calibration_replay_source, and workspace conformal artifacts.	calibration_data cannot be injected in this combined tuning flow; final evidence comes from tuning.winner.
predict(coverage=...)	Selects already materialized conformal intervals from a calibrated result or conformal bundle sidecar.	PredictResult.interval_coverages, selected conformal_guarantee_status, and direct interval accessors.	It does not recalibrate; unsupported coverages, invalid sidecars, and conformal all_predictions=True fail closed.
robustness(..., scenarios=[...])	Produces an audit-only robustness/generalization report for frozen predictions.	RobustnessReport, summary_artifact(), summary_rows(), JSON/Markdown/HTML/Parquet/artifact exports.	It does not refit, structurally retrain, or renew conformal guarantees.
robustness.X plus robustness.predictor or robustness.predictor_bundle	Replays explicit-X spectral/OOD perturbations through an in-memory predictor or saved bundle.	metadata["spectral_replay"], replay source, bundle path, sample-id forwarding status, and spectral/OOD summary rows.	predictor and predictor_bundle are mutually exclusive; stored provenance markers such as "prediction_arrays.X" are not executable arrays.
PredictResult.spectral_replay_evidence_status	Reports whether a stored prediction can safely run spectral/OOD replay.	ready_for_spectral_replay only when row-aligned finite 2D X/spectra and a saved replay bundle are present.	needs_spectral_replay_evidence is a hard block for spectral/OOD replay, not a prompt to synthesize spectra.
nirs4all.save_workspace_predict_result(...)	Publishes a PredictResult through the workspace prediction store.	Stored prediction id plus reloadable y_pred, sample ids, model metadata, X/spectra sidecar arrays and result_metadata.	It publishes prediction evidence only; conformal artifacts/interval guarantees remain owned by save_workspace_calibrated_result(...).
predict(..., save_to_workspace=True, workspace_result_metadata=...)	Prediction-time shortcut for the same workspace prediction publisher.	Returned PredictResult.metadata["workspace_prediction_id"] plus reloadable X/spectra evidence when data carries executable arrays.	It does not persist conformal interval artifacts; use save_workspace_calibrated_result(...) for guarantees.
nirs4all.load_workspace_predict_result(...); nirs4all.load_workspace_predict_results(...); Predictions.get_predict_result_by_id(...) / to_predict_results()	Converts workspace/store rows loaded with arrays into native PredictResult objects.	Preserves intervals, calibration_replay_source, tuning_calibration_source, robustness_evidence, actual X/spectra, sample ids and model metadata.	Records loaded without arrays cannot become executable prediction results.
nirs4all.robustness_from_workspace_prediction(...)	Loads one workspace prediction, delegates to robustness(), and optionally saves the report back to the workspace.	Consumes stored executable X/spectra + predictor_bundle evidence as spectral/OOD defaults and links saved reports to prediction_id.	Does not synthesize missing spectra or bypass spectral_replay_evidence_status; provenance-only markers remain non-executable.

Minimal pattern

import numpy as np
import nirs4all
from sklearn.linear_model import Ridge
from sklearn.preprocessing import StandardScaler

X_dev = np.asarray([[0.0], [1.0], [2.0], [3.0]], dtype=float)
y_dev = np.asarray([0.0, 1.0, 2.0, 3.0], dtype=float)

X_score = np.asarray([[4.0], [5.0], [6.0]], dtype=float)
y_score = np.asarray([4.0, 5.0, 6.0], dtype=float)

X_cal = np.asarray([[7.0], [8.0], [9.0], [10.0]], dtype=float)
y_cal = np.asarray([7.2, 7.8, 9.3, 9.7], dtype=float)

result = nirs4all.run(
    pipeline=[
        {
            "name": "scale",
            "class": "sklearn.preprocessing.StandardScaler",
            "params": {"with_mean": False},
        },
        {
            "name": "ridge",
            "class": "sklearn.linear_model.Ridge",
            "params": {"fit_intercept": False},
        },
    ],
    dataset=(
        X_dev,
        y_dev,
        ["train-001", "train-002", "train-003", "train-004"],
        None,
        None,
    ),
    engine="dag-ml",
    workspace_path="workspace/",
    tuning={
        "engine": "optuna",
        "space": {"model.alpha": [0.1, 1.0]},
        "force_params": {"model.alpha": 0.1},
        "metric": "rmse",
        "direction": "minimize",
        "sampler": "grid",
        "n_trials": 2,
        "score_data": {
            "X": X_score,
            "y": y_score,
            "sample_ids": ["score-001", "score-002", "score-003"],
        },
        "workspace_tuning_id": "ridge-native-tuning",
        "winner": {
            "X": X_cal,
            "y_true": y_cal,
            "score": 0.3,
            "metric": "rmse",
            "sample_ids": ["cal-001", "cal-002", "cal-003", "cal-004"],
        },
    },
    calibration={
        "y_pred": [11.0, 12.0],
        "prediction_sample_ids": ["pred-001", "pred-002"],
        "coverage": 0.8,
        "workspace_conformal_id": "ridge-conformal",
    },
)

When top-level calibration or tuning.calibration is present, the return value is a TunedSingleEstimatorConformalResult with:

• result.run: the tuned RunResult;

• result.calibrated: the conformal CalibratedRunResult or PredictResult.

The top-level form is an alias for tuning.calibration in the current DAG-ML tuning subset, so do not provide both in the same call. Without either calibration form, the same call returns a RunResult.

DAG-ML cache namespace proofs

The Python surface does not reinterpret DAG-ML D10 cache namespace proofs. When native training, replay, bundles or prediction-cache payloads contain cache_namespace_fingerprints, nirs4all treats them as signed DAG-ML control plane data and forwards them unchanged through nirs4all.pipeline.dagml.native_client to dag_ml. DAG-ML owns validation, namespace-aware handle derivation, file-store payload naming and columnar manifest exposure.

There is no nirs4all keyword for mutating this proof after signing. Change the candidate, data identity, fold, trial or seed before creating the DAG-ML contract.

Inspecting tuning and conformal outputs

The tuned run keeps the optimizer evidence on RunResult without pretending that development objective values are test scores:

print(result.tuning_id)
print(result.tuning_best_params)
print(result.tuning_best_value)

tuning_result = result.tuning_result
if tuning_result is not None:
    print(tuning_result.optimizer)
    print(tuning_result.n_trials)
    tuning_result.save_summary("tuning-summary.json")

TuningResult.summary_artifact(), to_summary_json() and save_summary() produce a lightweight deterministic summary.json-style payload for CI, bindings and Studio cards. It includes the result fingerprint, engine, metric, direction, optimizer, best params/value, trial count, trial state counts and compact trial rows. Compact diagnostic fields are scalar-only; whitelisted diagnostics carrying arrays or mappings fail closed instead of being stringified into ambiguous card text. It also includes a safe-to-publish persistence block with resume, storage_configured, study_name, and optimizer_state_resume_supported, but deliberately omits the raw storage URI so public cards do not leak local paths or credentials. The summary does not replace the full TuningResult.to_json() tape. The matching JSON Schema is public through get_tuning_summary_schema() and tuning_summary_schema_json(). The pre-execution ordered tuning-space artifact has its own public schema through get_tuning_space_schema() and tuning_space_schema_json(). When workspace_path is supplied, the completed TuningResult is written at the optimizer-to-refit boundary, before terminal winner refit, winner projection, or final conformal calibration. If a later refit or calibration step fails, the workspace still preserves the optimizer evidence tape for inspection and resume checks. If every candidate trial fails, no terminal refit is attempted, but the all-failed TuningResult remains persistable with its trial states and summary artifact.

When calibration is requested, intervals live on the calibrated prediction container. In that case result is a TunedSingleEstimatorConformalResult: result.run is the underlying RunResult, result.calibrated is the calibrated prediction container, the tuning_* accessors above proxy to result.run, and the common conformal accessors proxy to result.calibrated. Use the accessor methods rather than reaching into metadata:

prediction = result.calibrated if hasattr(result, "calibrated") else None
if prediction is not None:
    print(result.interval_coverages)
    intervals_80 = result.interval(0.8)
    status = result.conformal_guarantee_status
    print(status["status"])
    metrics = result.metrics(y_true_prediction)
    robustness_report = result.robustness(y_true=y_true_prediction)

These accessors are part of the public Python contract. Studio, bindings and scripts should depend on them, not on private fields or serialized JSON layout.

Public Python result types

The native conformal/tuning surface exposes stable top-level containers from nirs4all:

• RunResult: returned by run(tuning=...) when no final conformal calibration is requested. Its tuning_result, tuning_id, tuning_best_params, and tuning_best_value accessors expose optimizer evidence.

• TuningResult and TrialResult: public optimizer-evidence containers returned by tuning helpers and workspace loaders. TuningResult provides deterministic JSON persistence plus summary_artifact(), to_summary_json() and save_summary() for dashboards and release bundles. Direct construction is fail-closed: SearchSpaceParameter, ParameterPatch, OrderedSearchSpaceSpec, DagMLTuningSpec, TrialResult and TuningResult canonicalize tuning paths, reject non-TCV1 values, non-finite scores, boolean integers, duplicate trial ids and params outside tuning.space before fingerprinting or summary publication. Reload through TuningResult.from_dict() or load_json() uses the same strict numeric boundary for best_value: booleans and numeric strings are rejected, not coerced.

• TunedSingleEstimatorConformalResult: returned by run(tuning=..., calibration=...) and tune_single_estimator(..., calibration=...). It keeps the tuned run in run, the calibrated prediction result in calibrated, and proxies the common tuning_* accessors.

• CalibratedRunResult: returned by calibrate(..., as_predict_result=False), conformal workspace/bundle loaders, and the tuning+calibration path when a rich calibrated artifact is requested. It carries point predictions, intervals, sample ids, the conformal artifact fingerprint, guarantee metadata, JSON persistence helpers, and metrics(y_true). Reload recomputes stored qhat values from retained non-negative residual scores, then verifies that every materialized interval is exactly derived from the embedded artifact as y_pred ± qhat; grouped conformal results use a row-aligned qhat vector selected from the calibrated group for each prediction row. Self-consistent JSON with edited intervals, quantiles, group keys or vector qhat values is rejected.

• CalibratedPredictionBlock and ConformalIntervalBlock: internal/public substrate objects used by calibrated results and prediction helpers. Direct construction is fail-closed: interval coverage must match its mapping key, interval arrays must match y_pred shape, lower bounds must be <= upper bounds, qhat must be scalar or row-aligned and non-negative, and only the supported conformal method/unit vocabulary is accepted. Positive infinity remains valid for unbounded intervals.

• ConformalCalibrationData: typed input helper for calibrate(calibration_data=...). It serializes either replayed-array evidence or a selected dataset-backed calibration cohort to the same mapping syntax used by the runtime. Dataset-backed selectors reject non-string or whitespace-padded keys plus non-JSON-native values, and include_augmented must be a boolean so the helper cannot publish a coerced calibration cohort.

• ConformalMetricSet: returned by conformal_metrics(...) and CalibratedRunResult.metrics(y_true). It is empirical post-hoc evidence on an observed prediction cohort; it does not change the conformal guarantee. The container fails closed if observed_coverage is non-finite, outside [0, 1], or inconsistent with n_covered / n_samples; if coverage_gap is non-finite or not exactly observed_coverage - coverage; or if width/interval-score diagnostics are NaN or negative. Positive infinity remains valid for unbounded intervals.

• TUNING_ENGINES, TUNING_DIRECTIONS, TUNING_CONTRACT_KEYS, TUNING_OPTIMIZER_PERSISTENCE_KEYS, TUNING_RUNTIME_KEYS, CONFORMAL_TUNING_SCORE_METRICS, TuningEngine, and TuningDirection: public discovery and typing helpers for the native run(tuning=...) contract. Use them for forms, bindings, CLI validation and Studio controls instead of hard-coding the currently executable optimizer engines, objective directions, strict contract keys, optimizer persistence keys, runtime wrapper keys and conformal-aware objective metrics.

• inspect_tuning_space(...), OrderedSearchSpaceSpec, SearchSpaceParameter, and ParameterPatch: public inspection helpers for canonical ordered tuning.space patches. Use the JSON-native nirs4all.tuning.ordered_search_space artifact in forms, bindings and Studio previews when you need the exact patch order and search-space fingerprint before execution. Use TUNING_SPACE_SCHEMA_ID, get_tuning_space_schema() and tuning_space_schema_json() to validate this artifact without executing a pipeline.

• FINETUNE_ENGINES, FINETUNE_APPROACHES, FINETUNE_EVAL_MODES, FINETUNE_OPTUNA_SAMPLERS, FINETUNE_OPTUNA_PRUNERS, FINETUNE_N4M_SAMPLERS, FINETUNE_N4M_PRUNERS, FINETUNE_DAGML_DETERMINISTIC_ENGINES, FINETUNE_DAGML_META_KEYS, FINETUNE_DAGML_SELECTION_METRICS, FINETUNE_DAGML_APPROACHES, FINETUNE_DAGML_EVAL_MODES, FINETUNE_ENGINE_ALIASES, FINETUNE_SAMPLER_KEY_ALIASES, FINETUNE_EVAL_MODE_ALIASES, FinetuneEngine, FinetuneSampler, FinetunePruner, FinetuneApproach, and FinetuneEvalMode: public discovery and typing helpers for historical model-local finetune_params. These constants separate the registry-level vocabulary from the Optuna, n4m and deterministic DAG-ML executable subsets so downstream tooling does not overclaim support.

• CONFORMAL_CALIBRATION_METHODS, CONFORMAL_CALIBRATION_UNITS, CONFORMAL_MULTI_TARGET_POLICIES, CONFORMAL_EXECUTABLE_MULTI_TARGET_POLICIES, ConformalMethod, ConformalUnit, and ConformalMultiTarget: public discovery and typing helpers for the current conformal calibration method, exchangeability unit, and reserved/executable multi-target policy vocabulary. The executable V1 policies are marginal for one-dimensional targets and joint_max for simultaneous multi-target regions on two-dimensional replayed arrays.

• ROBUSTNESS_MODES, ROBUSTNESS_EXECUTABLE_MODES, and RobustnessMode: public discovery and typing helpers for the reserved robustness reuse policies and the subset executable by the current runtime.

• RobustnessScenarioSpec: typed input helper for robustness(scenarios=...). It serializes the supported audit scenario keywords to the same mapping syntax consumed by the runtime and rejects unsupported kinds, non-finite severities, negative noise severities, and unsupported distributions before execution.

• ROBUSTNESS_SCENARIO_KINDS / RobustnessScenarioKind: public discovery and typing helpers for the exact scenario kind vocabulary exposed by the runtime, report metadata, and keyword registry.

• ROBUSTNESS_STOCHASTIC_SCENARIO_KINDS, ROBUSTNESS_SCENARIO_DISTRIBUTIONS, and RobustnessScenarioDistribution: public discovery and typing helpers for the subset of scenario kinds that accepts distribution, and for the currently supported distribution vocabulary.

• RobustnessReport: returned by robustness(...). It is an audit artifact for frozen prediction cohorts and provides deterministic JSON, Markdown, HTML and Parquet-directory exports.

These types are intended for Python users, CI automation, Studio integration and bindings. Prefer their accessors and export helpers over private implementation modules.

The keyword/effect registry is also exposed from the top-level package for generated docs, forms, Studio and bindings:

registry = nirs4all.get_keyword_registry()
registry_json = nirs4all.keyword_registry_json(indent=2)
registry_schema = nirs4all.get_keyword_registry_schema()
registry_schema_json = nirs4all.keyword_registry_schema_json(indent=2)
robustness_summary_schema = nirs4all.get_robustness_summary_schema()
robustness_summary_schema_json = nirs4all.robustness_summary_schema_json(indent=2)

scenario_keywords = [
    entry for entry in registry["entries"] if entry["path"].startswith("robustness.scenarios")
]

The registry is descriptive. It reports keyword paths, value schemas, effects, engine support and UI hints, but it does not execute or validate a run by itself. For static consumers, nirs4all keyword-registry --output keyword-registry.json and nirs4all keyword-registry --schema --output keyword-registry.schema.json emit the same artifacts as the published docs _static/keyword-registry*.json files. get_robustness_summary_schema() and robustness_summary_schema_json() expose the JSON Schema for RobustnessReport.summary_artifact() / summary.json, so bindings and Studio can validate card/dashboard payloads without loading the full report schema. Docs builds also publish this contract as _static/robustness-summary.schema.json; CI can emit the same schema with nirs4all robustness-summary-schema --output robustness-summary.schema.json.

The mapping form above is the lowest-level public syntax. The same current subset also has typed helpers:

tuning = nirs4all.NativeTuning(
    engine="optuna",
    space={"scale": [nirs4all.TuningPassthrough()], "model.alpha": [0.1, 1.0]},
    force_params={"scale": nirs4all.TuningPassthrough(), "model.alpha": 0.1},
    metric="rmse",
    direction="minimize",
    sampler="grid",
    n_trials=2,
    score_data=nirs4all.TuningScoreData(
        X=X_score,
        y=y_score,
        sample_ids=["score-001", "score-002", "score-003"],
    ),
    winner=nirs4all.TuningWinner(
        X=X_cal,
        y_true=y_cal,
        score=0.3,
        metric="rmse",
        sample_ids=["cal-001", "cal-002", "cal-003", "cal-004"],
    ),
    workspace_tuning_id="ridge-native-tuning",
)

result = nirs4all.run(
    pipeline=[
        {"name": "scale", "transform": StandardScaler()},
        {"model": Ridge()},
    ],
    dataset=(X_dev, y_dev, ["train-001", "train-002", "train-003", "train-004"]),
    engine="dag-ml",
    workspace_path="workspace/",
    tuning=tuning,
    calibration=nirs4all.TuningCalibration(
        y_pred=[11.0, 12.0],
        prediction_sample_ids=["pred-001", "pred-002"],
        coverage=0.8,
        workspace_conformal_id="ridge-conformal",
    ),
)

Typed helpers normalize to the same dictionaries consumed by the runtime. They do not expand the supported subset. NativeTuning.to_tuning_spec() returns the deterministic optimizer contract; runtime-only blocks such as score_data, winner, calibration, and workspace ids stay outside that fingerprint. force_params is inside that deterministic contract because it changes the optimizer trajectory: it warm-starts the first Optuna/n4m trial with explicit public parameter values, fails closed when a key is not present in run.tuning.space, and can invalidate calibration when it changes the selected predictor. Both optimizer adapters publish that warm-start through the same public tape: the first TrialResult.params, best_params when the forced trial wins, and the trial search_space_fingerprint are expressed in decoded nirs4all syntax. Backend-specific labels or encoded categorical tokens are never part of the public result. The tuning.space mapping is lowered to an ordered search-space contract before any optimizer sees it. Public dotted paths such as ridge.alpha and sklearn double-underscore paths such as ridge__alpha are canonicalized to the same dotted path; duplicate canonical paths are rejected instead of relying on dictionary overwrite order. Candidate assignments and force_params become ordered ParameterPatch records with canonical string paths and JSON-native values, and trial diagnostics include a search_space_fingerprint so a score tape can be tied back to the exact ordered space that produced it. In short, sklearn double-underscore paths are canonicalized to the same dotted spelling as the public patch syntax. The typed helper applies the same validation as mapping payloads: storage must be an explicit URI such as sqlite:///study.db, and study_name is trimmed and cannot contain NUL characters. TuningCalibration(...) validates coverage as a scalar in (0, 1) or a non-empty unique list of such values, normalizes method and unit to the supported lower-case spellings, and refuses values outside CONFORMAL_CALIBRATION_METHODS and CONFORMAL_CALIBRATION_UNITS before runtime execution. Its extra={...} field is reserved for additional calibration options only; it cannot override typed keys such as coverage, method, unit, prediction_sample_ids, y_pred, workspace metadata, or inject calibration_data. TuningCalibration.as_predict_result must be a boolean, and TuningCalibration.extra, TuningCalibration.workspace_metadata and NativeTuning.workspace_metadata require canonical non-empty string keys. Their values must stay strict JSON-native: finite integers/floats, strings, booleans, null, lists, or nested mappings with canonical string keys. Non-finite numbers, bytes, tuples, sets and arbitrary Python objects fail before publication. The same strict boundary is enforced again when workspace tuning or conformal rows are persisted through save_workspace_tuning_result(...), run(tuning=..., workspace_path=...), TuningCalibration.workspace_metadata, or save_workspace_calibrated_result(...); the workspace store does not stringify Python objects into metadata. Prediction workspace publication applies the same rule to predict(..., save_to_workspace=True, workspace_metadata=..., workspace_result_metadata=...) and to save_workspace_predict_result(..., metadata=..., result_metadata=...) before writing the prediction sidecar. NativeTuning.space and force_params canonicalize string patch keys before serializing. Raw NativeTuning(score_data={...}), winner={...} and calibration={...} mappings use the same fail-closed boundary before to_dict() publishes them: dataset selectors require canonical string keys, include_augmented and as_predict_result are strict booleans, winner scores are finite numbers, and nested calibration_data is rejected. Raw calibration coverage, method, unit, workspace_metadata and workspace_conformal_id are validated and canonicalized like the typed TuningCalibration(...) helper. Public calibrate(..., result_metadata=...) and predict_calibrated(..., result_metadata=...) also require strict JSON-native mappings before generated guarantee metadata is merged. Top-level NativeTuning core fields are validated through DagMLTuningSpec before publication: engine, metric and direction are canonicalized, n_trials/seed/resume reject coercive values, and storage, study and workspace tuning ids must be canonical strings. TuningScoreData.metric/score_metric and raw NativeTuning.score_data.metric/score_metric must be real non-empty strings; non-string values, blank strings and NUL-containing strings fail closed, and the published score metric is lowercase canonical metric. TuningWinner.metric, dataset_name, model_name and task_type, plus the same raw NativeTuning.winner fields and aliases, use the same no-stringify rule; winner metric and task_type publish lowercase values, while dataset_name and model_name publish trimmed labels. TuningPassthrough() is the typed form of the structured {"kind": "passthrough"} marker for optional non-final preprocessing steps; forms, bindings and Studio should use it instead of inventing a separate placeholder. The structured marker is exact: kind must be the literal string "passthrough" and host objects that only stringify to that value are rejected before native tuning can run. For named categorical choices, tuning.space also accepts {"type": "categorical", "options": {"label": value}}: Optuna/n4m see the stable label, while best_params, trial rows and terminal refit receive the public JSON-native value. When force_params targets such a named categorical choice, callers still pass the public decoded value; Optuna and n4m may encode it to a stable backend label internally, including for optimizer.enqueue(...), but both adapters publish the decoded value back in TrialResult.params and best_params. They reject internal labels or unknown keys instead of silently changing the warm-start trial. The internal categorical codec is also fail-closed on direct construction: choices must be non-empty, unique, optimizer-native without a decoder, and TCV1 JSON-native; decoder keys must match encoded choices and decoded public values must be unique before Optuna/n4m can use them. The pruner key is transported to the selected optimizer for the supported backend names (none, median, successive_halving, hyperband, plus native n4m aliases such as asha/racing where available). The current shared objective reports one final score per trial; it does not invent intermediate fold/resource signals, so pruning is limited to what that backend can infer from the available objective calls. If an Optuna objective raises optuna.exceptions.TrialPruned, the adapter preserves that optimizer state as TrialResult(state="PRUNED", value=None) with compact diagnostics score_extractor="pruned" instead of collapsing it into FAIL; in the reference contract, the adapter records a distinct TrialResult(state="PRUNED", value=None, diagnostics={...}). The n4m adapter uses the same portable row when the shared objective raises a prune exception, but only if the installed native binding can terminalize the trial through optimizer.tell_result(..., TrialStatus.PRUNED). Older n4m bindings that cannot record a pruned terminal state fail closed instead of converting the candidate to FAIL or to a worst completed score. sampler and pruner strings are normalized case-insensitively before adapter dispatch, so forms such as " TPE " or " HyperBand " resolve to their canonical lower-case values. The legacy Optuna and n4m managers apply the same canonicalization to approach, eval_mode, and explicit direction, including the sample alias for sampler and the eval_mode="avg" alias for eval_mode="mean"; Optuna also canonicalizes per-phase sampler values before multi-phase dispatch. Logs, traces and optimizer dispatch therefore use stable values after validation. The shared Optuna and n4m adapters both accept sampler="grid" for the portable categorical/grid subset; n4m fails closed with an explicit enum error if the installed native optimizer bindings do not expose their GRID sampler yet. Unknown sampler or pruner spellings are rejected before optimizer construction; they do not silently fall back to TPE or to a no-pruner study. For Optuna persistence, storage must be an explicit URI string such as sqlite:///study.db; a bare path like study.db is rejected. study_name is trimmed and cannot contain NUL characters. When resume=True, the Optuna adapter requires both storage and study_name; anonymous or in-memory resume is rejected before optimizer execution. If force_params is also supplied while resuming a non-empty study, it must match the already materialized warm-start trial; the adapter does not enqueue a duplicate trial and fails closed if the caller changes the warm-start assignment under the same study_name/storage. Existing materialized trial params must also match the current tuning.space keys exactly; changing the search-space keys under the same persisted study fails closed before optimizer execution. Existing categorical values must also still be present in the current choices for their key; removing or renaming a choice under the same persisted study fails closed before optimizer execution. Existing numeric values must also remain inside the current range for their key; narrowing a range so that a stored trial falls outside it fails closed before optimizer execution. If the current numeric space declares a step, restored values must also lie on that grid. During Optuna storage resume, n_trials is treated as the target total trial count: an already completed one-trial study with n_trials=1 runs no extra trial, while n_trials=2 runs one remaining trial. New Optuna storage-backed studies also persist nirs4all study fingerprints in study.user_attrs: a format marker, schema version, optimizer contract fingerprint and search-space fingerprint. Resume fails closed if those fingerprints are missing from a non-empty study or no longer match. Trials restored from Optuna storage keep compact summary diagnostics even when they were produced by a previous process: completed rows use score_extractor="optuna_storage", failed rows use score_extractor="failed", and pruned rows use score_extractor="pruned". Restored Optuna COMPLETE rows must carry a finite numeric value; missing or non-finite storage values fail closed instead of becoming completed trials with no usable objective value. Restored non-COMPLETE rows must not carry a final storage value; failed, pruned or in-flight rows with final values are rejected as corrupted optimizer history. Restored RUNNING rows fail closed during resume because interrupted active trials cannot be safely recovered into a terminal HPO tape. Restored terminal Optuna rows must keep exactly the current tuning.space parameter keys when the search space is non-empty; rows whose stored parameter table was removed fail closed instead of becoming completed trials with empty public params. Restored queued Optuna WAITING rows that already carry materialized params or fixed_params must also satisfy the current tuning.space; incompatible values are rejected before Optuna can consume them. Restored Optuna trial numbers must be canonical unique integers; corrupt storage that yields non-integer or duplicate trial numbers fails closed before the HPO tape is projected. n4m optimizer-state persistence uses native N4MOPT checkpoints in this shared objective seam: set storage="file:///absolute/checkpoint-dir" and study_name="filename-safe-name". After each terminal trial, nirs4all writes a JSON manifest containing the native checkpoint bytes plus tuning/search-space fingerprints. With resume=True, the adapter reloads that checkpoint only when the optimizer contract still matches; changing the search space, sampler, pruner, seed, direction, metric or warm-start assignment fails closed. The n_trials value is treated as the target total trial count, so a checkpoint with one completed trial and n_trials=2 runs one remaining trial. Restored n4m trial rows are decoded back to the public parameter syntax before they enter TuningResult.trials, and are ordered canonically by numeric trial id even if a native binding returns checkpoint records in another order. Duplicate restored trial ids, or restored ids that are not canonical integers, fail closed before TuningResult construction; optimizer-only categorical labels used for JSON-native values or named options are never exposed as resumed TrialResult.params. The restored checkpoint row params must still match the current tuning.space keys and value domains; edited or incompatible checkpoint keys, categorical choices, numeric ranges or numeric steps fail closed before they can contribute optimizer history. Restored COMPLETE rows must carry a finite numeric score; missing, boolean or non-finite scores fail closed instead of becoming complete trials with no usable value. Restored non-success rows also keep compact diagnostics aligned with their state: score_extractor="failed" for failed candidates and score_extractor="pruned" for pruned candidates. If a native checkpoint contains a terminal cancelled row, it is restored as CANCELLED with score_extractor="cancelled", so resumed summary cards do not collapse these states into a generic checkpoint row. Non-terminal or unsupported checkpoint rows such as RUNNING fail closed during resume instead of being counted as completed optimizer history. Restored n4m non-COMPLETE rows must not carry a final score; failed, pruned or cancelled checkpoint rows with scores are rejected as corrupted optimizer history. When an individual candidate fails during fit or scoring, the tuning tape keeps a FAIL trial with value=None and diagnostic error metadata. Other candidates can still complete; if all candidates fail, nirs4all returns the failed tuning tape, uses a finite direction-worst best_value for fingerprint/summary compatibility, and skips terminal refit because there is no winner. nirs4all distinguishes this from Optuna- or n4m-pruned trials, which stay visible as PRUNED rows in TuningResult.trials, trial_states and compact summary rows. n4m bindings must expose either optimizer.tell_result(...) with a failed trial status or optimizer.tell(...) so the adapter can advance the native optimizer after a candidate failure. Older bindings that cannot record failed candidates are rejected fail-closed instead of silently losing the trial tape. n4m prune-aware tapes additionally require optimizer.tell_result(..., TrialStatus.PRUNED). Linear preprocessing steps may also use explicit sklearn.* class-path mappings with params, or direct sklearn.* string steps when no constructor parameters are needed. String imports work as direct model steps, named tuple steps, and transform/model mapping values. When constructor parameters are required inside a linear chain, preprocessing mappings may use {"transform": "sklearn.preprocessing.StandardScaler", "params": {"with_mean": False}} and final model mappings may use {"model": "sklearn.linear_model.Ridge", "params": {"fit_intercept": False}}. Constructor params must use canonical string keys and TCV1-compatible JSON-native values; Python objects, tuples, bytes and non-finite numbers are rejected before import or instantiation. The {"class": ..., "params": ...} form remains available for preprocessing steps and the direct single-model form. Short aliases and arbitrary imports remain unsupported in the native tuning compiler. A single model can use the same declarative form directly, for example pipeline={"name": "ridge", "class": "sklearn.linear_model.Ridge", "params": {"fit_intercept": False}} or pipeline={"name": "ridge", "model": "sklearn.linear_model.Ridge", "params": {"fit_intercept": False}} with space={"ridge.alpha": [...]}.

The lower-level nirs4all.tune_single_estimator() helper also accepts NativeTuning(score_data=..., winner=..., calibration=...) for explicit array/tuple/list scoring and winner cohorts. It uses the same single-estimator and linear compiler forms described above, including direct sklearn.* string imports when no constructor parameters are needed and {"class": ..., "params": ...} steps when they are, plus the {"steps": [...]} and {"pipeline": [...]} wrappers. That includes TuningConformalScoreCalibration for temporary conformal-aware objective scoring, TuningWinner(...) for the projected final winner entry, and TuningCalibration(...) for the final integrated tune → calibrate call. Dataset-backed score_data and winner still belong to run(tuning=...), which owns dataset loading and selector resolution.

Important keywords and effects

Keyword	Required now	Effect
engine="dag-ml"	yes	Selects the native public subset. Other engines reject tuning here.
tuning.engine	yes	Selects the optimizer adapter, currently optuna or n4m for the covered subset.
tuning.space	yes	Parameter search space. Linear chains use paths such as scale.factor or model.alpha; sklearn-style scale__factor is canonicalized to the same ordered dotted patch path.
tuning.score_data	yes	Explicit scoring cohort used by the objective. nirs4all does not derive it implicitly. Only mapping and tuple/list forms are accepted. Mapping form accepts X/y or X_score/y_score, optional metric/score_metric, sample id aliases, groups/score_groups, and metadata/score_metadata; tuple/list form is (X_score, y_score, sample_ids, groups, metadata) with exactly 2–5 fields and strict metadata keys. Identity fields must align with y; compatible estimators receive them in predict().
tuning.score_data.conformal_calibration	optional	Explicit development calibration cohort for conformal-aware objective scoring. It fits a temporary calibrator per candidate and never replaces the final run(..., calibration=...) result.
tuning.winner	required for calibration	Explicit terminal prediction cohort used to create RunResult.best.
tuning.calibration	optional	Applies split conformal calibration after tuning and terminal winner projection.
calibration	optional	Top-level alias for tuning.calibration; requires run(tuning=..., engine="dag-ml") and must not be combined with nested tuning.calibration.
workspace_path	optional	Enables workspace persistence. The same path is inherited by tuning.calibration or top-level calibration unless overridden.
workspace_tuning_id	optional	Stable id for the persisted TuningResult. The raw alias tuning_id is accepted, but must not be combined with workspace_tuning_id; caller-provided ids must be canonical non-empty strings without surrounding whitespace or NULs.
workspace_conformal_id	optional	Stable id for the persisted conformal result; caller-provided ids must be canonical non-empty strings without surrounding whitespace or NULs.
resume=True	optional	Reuses an already completed workspace tuning result for this subset; it is not interrupted optimizer-state resume.

Caller-provided conformal ids must be canonical non-empty strings without surrounding whitespace or NULs. Caller-provided robustness ids must also be canonical non-empty strings without surrounding whitespace or NULs. Omitting these ids asks the workspace store to generate ids that remain outside the scientific result fingerprints.

Registry coverage reference

The public keyword registry is the canonical machine-readable contract for forms, bindings, Studio and docs generation. The table below mirrors the native tuning/conformal/robustness entries that this guide covers. Changes names the state affected by the keyword; Invalidates calibration explains whether a previous conformal result must be treated as stale.

Registry path	Lifecycle	Changes	Invalidates calibration	Summary
run.engine	execution	execution_backend	if_predictor_changes	Selects the pipeline execution backend; native is the fail-closed Archive V2 Methods subset, while dual is a bounded native-first, no-fallback oracle only for explicit array regression with KFold(shuffle=False) and PLSRegression; neither selects the HPO algorithm.
run.tuning	search	candidate_fit, selection, final_predictor	always	Fixed-topology full-DAG HPO argument. The public subset currently executes only engine="dag-ml" single-estimator array pipelines with explicit score_data; broader DAG shapes remain fail-closed.
run.tuning.engine	search	optimizer_algorithm	if_predictor_changes	Optimizer selector inside full-DAG tuning; it is distinct from run.engine and currently executes only the single-estimator array subset.
run.tuning.space	search	parameter_patches, candidate_predictors, selection	always	Fixed-topology parameter search space. The public subset supports value patches on a single estimator; structural axes and full pipeline recompilation remain planned.
run.tuning.force_params	search	trial_sequence, candidate_fit, selection	if_predictor_changes	Enqueues a caller-provided parameter assignment as the first optimizer trial. Keys must be a subset of run.tuning.space; values use the public decoded syntax, not optimizer-internal categorical labels.
run.tuning.seed	search	trial_sequence, selected_predictor	if_predictor_changes	Seeds the HPO trajectory. A different seed invalidates calibration only when it changes the selected predictor.
run.tuning.storage	storage	optimizer_state	not_applicable	Optuna storage URI for storage-backed studies, or n4m file:///absolute/checkpoint-dir storage for native N4MOPT checkpoints. Bare paths are rejected.
run.tuning.study_name	storage	optimizer_state	not_applicable	Optuna study name or n4m checkpoint filename stem. Whitespace is trimmed, NUL characters are rejected, and n4m also requires a filename-safe value.
run.tuning.score_data	search	objective_scores, trial_ranking, selected_predictor	if_predictor_changes	Supplies the explicit scoring cohort used by run(tuning) optimizer-driving. It is mandatory in the public subset and may be mapping, tuple/list, or explicit dataset-backed selector from SpectroDataset, DatasetConfigs, config mapping, or config/path string.
run.tuning.score_data.conformal_calibration	search	objective_scores, trial_ranking	if_predictor_changes	Provides an explicit development calibration cohort used only for conformal-aware objective scoring. It fits a temporary calibrator per candidate and never replaces the final run(calibration=...) result.
run.tuning.score_data.conformal_coverage	search	objective_scores, trial_ranking	if_predictor_changes	Selects the nominal coverage for the temporary conformal scorer inside score_data. It requires score_data.conformal_calibration and affects only trial ranking, not the final run(calibration=...) result.
run.tuning.winner	refit	run_result_best, terminal_prediction_entry	replaces_existing	Projects the refit winner to an explicit terminal prediction entry. It is required before run(tuning).calibration and can use arrays or an explicit dataset-backed selector from SpectroDataset, DatasetConfigs, config mapping, or config/path string.
run.tuning.calibration	calibration	calibrator, calibrated_result	replaces_existing	Runs conformal calibration immediately after the explicit winner projection. calibration_data is derived from winner and cannot be supplied here.
run.calibration	calibration	calibrator, calibrated_result	replaces_existing	Top-level alias for run(tuning).calibration in the public DAG-ML tuning subset. It requires run(tuning=..., engine="dag-ml") and derives calibration_data from tuning.winner.
run.tuning.workspace_tuning_id	storage	workspace_tuning_results	not_applicable	Names the workspace TuningResult record and is required with resume=True in the public run(tuning) subset.
run.tuning.workspace_metadata	storage	workspace_tuning_results	not_applicable	Persists caller-provided strict JSON-native metadata alongside the workspace TuningResult without changing optimizer behavior.
run.tuning.resume	search	terminal_refit, terminal_prediction_entry	if_predictor_changes	Reuses an already completed matching workspace TuningResult in the public subset. It is not interrupted optimizer checkpoint resume.
run.tuning.calibration.workspace_conformal_id	storage	workspace_conformal_results	not_applicable	Names the conformal workspace record produced by run(tuning).calibration; the top-level run(workspace_path=...) is inherited unless overridden in the calibration payload.
calibrate.calibration_data	calibration	calibrator	replaces_existing	Exchangeable calibration cohort. The public API supports replayed-array mappings and explicit dataset-backed selectors with replayed predictions, an in-memory predictor, a saved predictor bundle, a RunResult.best-like prediction entry, or a stored workspace chain plus physical sample ids.
calibrate.calibration_data.dataset	calibration	calibration_targets, calibration_identity_source	replaces_existing	Selects explicit dataset-backed calibration evidence from a SpectroDataset, DatasetConfigs object, dataset config mapping, or config/path string.
calibrate.calibration_data.selector	calibration	calibration_rows	replaces_existing	Filters the explicit dataset-backed calibration cohort. It is mandatory so nirs4all never guesses a calibration partition.
calibrate.calibration_data.y_pred	calibration	nonconformity_scores, calibrator	replaces_existing	Provides already replayed point predictions for the calibration cohort. It does not trigger model replay.
calibrate.calibration_data.predictor	calibration	calibration_predictions, nonconformity_scores, calibrator	replaces_existing	Replays an in-memory sklearn-like predictor on the selected dataset-backed calibration cohort. sample_ids/groups/metadata are forwarded when predict() accepts them.
calibrate.calibration_data.predictor_bundle	calibration	calibration_predictions, nonconformity_scores, calibrator	replaces_existing	Replays a saved predictor path through nirs4all.predict() on the selected dataset-backed calibration cohort.
calibrate.calibration_data.predictor_result	calibration	calibration_predictions, nonconformity_scores, calibrator	replaces_existing	Replays a RunResult.best-like prediction entry through nirs4all.predict() on the selected dataset-backed calibration cohort.
calibrate.calibration_data.predictor_chain_id	calibration	calibration_predictions, nonconformity_scores, calibrator	replaces_existing	Replays a stored workspace chain through nirs4all.predict(chain_id=...) on the selected dataset-backed calibration cohort. Requires calibration_data.workspace_path.
calibrate.calibration_data.workspace_path	calibration	calibration_predictions	replaces_existing	Provides the explicit workspace root used by predictor_result or predictor_chain_id replay during dataset-backed calibration. It is not the conformal result persistence workspace.
calibrate.calibration_data.sample_ids	calibration	calibration_cohort_manifest, coverage_claim_unit	replaces_existing	Supplies explicit row-aligned physical sample ids for calibration. calibration_sample_ids and physical_sample_ids are raw aliases; missing or duplicated ids remain fail-closed.
calibrate.calibration_data.groups	calibration	calibration_cohort_manifest, coverage_claim_scope	replaces_existing	Supplies optional row-aligned group labels for raw replayed-array calibration mappings. group_by="group" consumes these labels for grouped conformal calibration.
calibrate.calibration_data.metadata	calibration	calibration_cohort_manifest, coverage_claim_scope	replaces_existing	Supplies optional row-aligned metadata for raw replayed-array calibration mappings. Metadata keys used by group_by select grouped conformal partitions.
calibrate.calibration_data.sample_id_column	calibration	calibration_cohort_manifest, coverage_claim_unit	replaces_existing	Extracts physical sample ids from the selected SpectroDataset metadata column when sample_ids are not supplied directly.
calibrate.calibration_data.group_column	calibration	calibration_cohort_manifest	replaces_existing	Extracts optional row-aligned group labels into the calibration cohort manifest; it does not enable grouped conformal guarantees by itself.
calibrate.calibration_data.metadata_columns	calibration	calibration_cohort_manifest	replaces_existing	Copies selected SpectroDataset metadata fields into the calibration cohort manifest for audit and diagnostics.
calibrate.method	calibration	calibrator	replaces_existing	Conformal score and interval construction method. V1 currently supports split absolute residuals on replayed regression predictions.
calibrate.coverage	calibration	quantile, prediction_bounds	extends_existing	Nominal coverage used to select the finite-sample conformal quantile; retained scores allow deterministic interval materialization for replayed-array calibration.
calibrate.unit	calibration	score_aggregation, coverage_claim	replaces_existing	V1 exchangeability unit. The current public surface requires explicit physical sample ids for replayed-array calibration.
calibrate.group_by	calibration	calibration_partitions, coverage_claim_scope	replaces_existing	Grouped split conformal is executable for replayed-array calibration when calibration and prediction rows provide matching group evidence. Report slices alone never create a grouped conformal guarantee.
calibrate.prediction_groups	prediction	selected_prediction_bounds, coverage_claim_scope	not_applicable	Row-aligned prediction labels for group_by="group"; missing or unseen groups fail closed without global fallback.
calibrate.prediction_metadata	prediction	selected_prediction_bounds, coverage_claim_scope	not_applicable	Row-aligned prediction metadata for non-group group_by keys; every key must be present, non-null and JSON-compatible.
calibrate.multi_target	calibration	nonconformity_score, prediction_region, coverage_claim_scope	replaces_existing	marginal supports one-dimensional targets; joint_max supports simultaneous multi-target regions for two-dimensional replayed arrays.
calibrate.result_metadata	calibration	calibrated_result_metadata, calibrated_result_fingerprint	not_applicable	Stores strict JSON-native result metadata before nirs4all merges generated guarantee metadata. It changes the calibrated result fingerprint, not the fitted conformal artifact.
predict_calibrated.result_metadata	prediction	calibrated_result_metadata, calibrated_result_fingerprint, source_calibrated_result_fingerprint	not_applicable	Stores strict JSON-native result metadata when applying an existing calibrated result to new replayed predictions; nirs4all appends guarantee/source metadata after validation.
predict.coverage	prediction	selected_prediction_bounds	not_applicable	Selects already materialized calibrated coverages at prediction time for calibrated replayed-array results/stores/bundles and model .n4a bundles with a conformal sidecar.
predict.all_predictions	prediction	prediction_entries	not_applicable	Requests all model prediction entries. With a conformal sidecar and coverage=..., all_predictions=True remains fail-closed until each entry carries calibrated identity mapping.
predict.save_to_workspace	prediction_storage	workspace_prediction_rows, prediction_arrays, result_metadata, workspace_prediction_id	not_applicable	Publishes the returned PredictResult through the workspace prediction store, including executable X/spectra when available from data; it does not persist conformal artifacts or renew guarantees.
predict.workspace_metadata	prediction_storage	prediction_sample_metadata	not_applicable	Stores caller-provided strict JSON-native sample-level metadata alongside the workspace prediction sidecar when predict.save_to_workspace is true.
predict.workspace_result_metadata	prediction_storage	result_metadata, robustness_evidence	not_applicable	Stores strict JSON-native result-level metadata for the workspace prediction row, including robustness_evidence.predictor_bundle for later spectral/OOD replay.

Workspace prediction ids are canonical non-empty strings without surrounding whitespace or NULs. The public prediction publisher generates the id it returns as metadata["workspace_prediction_id"]; lower-level explicit prediction_id writes fail closed on invalid ids instead of silently generating or stringifying a replacement. Workspace link ids stored beside tuning, conformal and robustness rows follow the same rule when supplied: run_id, pipeline_id, chain_id, and source prediction_id/conformal_id must be canonical non-empty strings without surrounding whitespace or NULs.

| robustness.mode | robustness | robustness_report | mode_dependent | Selects the robustness reuse policy. The current public API supports clean_frozen audit-only reports on already replayed predictions; matched recalibration and structural refit remain planned. | | robustness.scenarios | robustness | robustness_results | mode_dependent | Defines report cells for robustness diagnostics. The current public API accepts RobustnessScenarioSpec helpers or equivalent mappings for observed, prediction-space and explicit-X spectral scenarios. | | robustness.scenarios[].kind | robustness | robustness_results | mode_dependent | Selects the audit-only scenario cell. observed leaves predictions unchanged; prediction-space scenarios alter already materialized predictions; spectral scenarios replay an explicit frozen predictor on perturbed X. | | robustness.scenarios[].severity | robustness | robustness_results | mode_dependent | Controls scenario magnitude: offset, noise standard deviation, scale/ramp amplitude or fractional feature-axis shift depending on the scenario kind. | | robustness.scenarios[].distribution | robustness | robustness_results | mode_dependent | Selects the stochastic scenario distribution. The current executable subset accepts distribution only for prediction_noise and spectral_noise, with normal and centered uniform noise supported fail-closed. | | robustness.X | robustness | robustness_results | mode_dependent | Supplies the explicit input matrix used by spectral scenarios. It is never fitted on; it is perturbed for evaluation-only frozen predictor replay. | | robustness.predictor | robustness | robustness_results | mode_dependent | Frozen predictor replay hook for spectral scenarios. The current Python surface accepts predictor.predict(X) or a callable and does not refit or recalibrate it. | | robustness.predictor_bundle | robustness | robustness_results | mode_dependent | Saved predictor path replay hook for spectral scenarios. The current Python surface routes perturbed X through nirs4all.predict(model=predictor_bundle, data={"X": X, "sample_ids": ...}) and does not refit or recalibrate it. | | robustness.slice_by | reporting | reported_metrics | not_applicable | Adds diagnostic report slices over supplied metadata. A sliced empirical coverage is not a conditional conformal guarantee unless a separate valid grouped calibrator supports it. | | robustness.seed | robustness | robustness_results | mode_dependent | Controls reproducible audit-only stochastic scenarios such as prediction_noise. It does not change a frozen predictor or calibrator; omitted seeds use effective_seed=0. | | robustness.workspace_path | reporting | workspace_robustness_results | not_applicable | Persists the verified RobustnessReport in the workspace after the audit is computed. This does not change metrics, fingerprints, predictions, or conformal guarantees. | | robustness.workspace_robustness_id | reporting | workspace_robustness_results | not_applicable | Stable id for the workspace robustness row produced by robustness(workspace_path=...). If omitted, the store generates an id that is not injected into the report fingerprint; if provided, it must be a canonical non-empty string without surrounding whitespace or NULs. | | robustness.workspace_name | reporting | workspace_robustness_results | not_applicable | Human-readable workspace label for a persisted robustness report. It affects only workspace inventory and does not enter the report fingerprint. | | robustness.workspace_metadata | reporting | workspace_robustness_results | not_applicable | Strict JSON-native workspace metadata stored alongside the robustness report row for CI/Studio indexing. It does not enter the report fingerprint. |

For tuning, workspace_metadata is constrained by the published keyword registry schema to strict JSON-native values with canonical string keys; it is display/storage metadata only and never changes optimizer behavior. Robustness workspace metadata follows the same strict JSON-native published schema and the workspace store rejects invalid values before inserting the robustness_results row.

The calibration payload cannot provide its own calibration_data inside run(tuning=...), whether calibration is nested under tuning.calibration or passed as top-level run(..., calibration=...). Calibration evidence is derived from tuning.winner so the order remains auditable: tune, terminal refit/projection, then conformal calibration. The HPO scoring cohort from tuning.score_data is never reused as final calibration evidence; if the final intervals differ from score-data residuals, the tuning.winner cohort is authoritative. In the typed helper, TuningCalibration.extra follows the same rule and also refuses overrides of typed calibration keys. When workspace_conformal_id is supplied, the workspace row stores the winner-derived calibrated result, so reloading it preserves those intervals and does not recompute them from tuning.score_data. Reloaded CalibratedRunResult objects can be converted back to the public prediction surface with calibrated.to_predict_result(), preserving intervals, calibrated_result_fingerprint, calibration_replay_source, and tuning_calibration_source for notebooks, bindings, CLI views and Studio. The conversion and reload path validates conformal_guarantee_status against the embedded conformal artifact; stale fingerprints, method/unit mismatches or non-materialized coverage selections fail closed rather than displaying a false guarantee. The calibrated prediction metadata also includes tuning_calibration_source={"source": "tuning.winner", "score_data_role": "hpo_objective_only", "score_data_used": false}. Studio, Python bindings and audit tools can use this field to show that tuning.score_data influenced optimizer ranking only, not the final conformal evidence. If the calibration payload supplies result_metadata, nirs4all keeps those user fields and adds tuning_calibration_source only when it is absent. Use prediction.tuning_calibration_source or calibrated.tuning_calibration_source for the direct accessor instead of parsing metadata manually.

The tuple/list dataset order is fixed: (X, y, sample_ids, groups, metadata). Use None for an omitted middle field. Tuple/list forms with extra fields are rejected instead of guessed, and all identity fields must align with y.

The score_data tuple/list follows the same rule for the scoring cohort: (X_score, y_score, sample_ids, groups, metadata). Use mapping form when you need to override the scoring metric for that cohort. NativeTuning.to_dict() validates tuple/list score_data before publication: it must contain between two and five fields, metadata must use canonical non-empty string keys in either column-style mapping or row-style sequence form, metadata values must stay strict JSON-native and finite, and tuple inputs publish as JSON-native lists. Any other score_data type, including scalar strings, bytes, numbers and arbitrary objects, is rejected before publication.

The replayed-array calibration_data tuple is similarly explicit: (y_true, y_pred, sample_ids, groups, metadata). sample_ids are still required for conformal calibration evidence; use mapping form when you need named fields or dataset-backed selectors. Raw replayed-array mappings accept the same canonical payload plus the public aliases y_pred_calibration or calibration_predictions for y_pred, calibration_sample_ids or physical_sample_ids for sample_ids, calibration_groups for groups, and calibration_metadata for metadata. Providing more than one alias for the same field is rejected as ambiguous before calibration.

You can also use the typed helper when building forms or bindings:

calibration_data = nirs4all.ConformalCalibrationData(
    y_true=y_cal,
    y_pred=y_cal_pred,
    sample_ids=calibration_sample_ids,
    groups=calibration_groups,
    metadata=calibration_metadata,
)

calibrated = nirs4all.calibrate(
    calibration_data=calibration_data,
    y_pred=y_pred,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

Grouped split conformal, also called Mondrian conformal, is executable for the same replayed-array substrate when both calibration and prediction rows carry the declared grouping evidence. Use group_by="group" to consume calibration_groups and prediction_groups; use metadata keys in group_by when the row-aligned calibration_metadata and prediction_metadata mappings contain those keys. Missing prediction groups, null metadata values, or groups not seen during calibration fail closed; nirs4all never falls back to the global calibrator for a grouped request. Group labels are strict strings too: whitespace-padded labels, NUL-containing labels and non-string labels fail before group quantiles are fitted or selected.

calibrated = nirs4all.calibrate(
    y_true=y_cal,
    y_pred_calibration=y_cal_pred,
    y_pred=y_pred,
    calibration_sample_ids=calibration_sample_ids,
    prediction_sample_ids=prediction_sample_ids,
    calibration_groups=calibration_groups,
    prediction_groups=prediction_groups,
    group_by="group",
    coverage=0.9,
)

status = calibrated.conformal_guarantee_status
print(status["guarantee"])  # split_conformal_group_marginal_coverage

calibrate.group_by is therefore partial: it changes calibration_partitions and coverage_claim_scope, and replaces any existing calibrator. calibrate.prediction_groups and calibrate.prediction_metadata are partial, not_applicable to calibration invalidation, and change selected_prediction_bounds plus coverage_claim_scope because they route prediction rows to already fitted group quantiles.

For multi-target regression, multi_target="joint_max" is executable on two-dimensional replayed arrays. nirs4all stores one nonconformity score per physical sample, max(abs(y_true - y_pred)) across target columns, and applies the resulting scalar qhat to every target column of the prediction row. The guarantee is simultaneous for the full target vector, not a separate conditional claim per target.

calibrated = nirs4all.calibrate(
    y_true=y_cal_2d,
    y_pred_calibration=y_cal_pred_2d,
    y_pred=y_pred_2d,
    calibration_sample_ids=calibration_sample_ids,
    prediction_sample_ids=prediction_sample_ids,
    multi_target="joint_max",
    coverage=0.9,
)

print(calibrated.conformal_guarantee_status["guarantee"])
# split_conformal_joint_max_simultaneous_coverage

For conformal-aware development scoring, use mapping form with conformal_calibration and a conformal metric:

"score_data": {
    "X": X_score,
    "y": y_score,
    "metric": "conformal_mean_width",
    "conformal_coverage": 0.9,
    "conformal_calibration": {
        "X": X_dev_calibration,
        "y_true": y_dev_calibration,
        "sample_ids": dev_calibration_ids,
    },
}

Or use the typed helper when building public configs in Python:

tuning = nirs4all.NativeTuning(
    engine="optuna",
    space={"alpha": [0.2, 0.9]},
    metric="conformal_mean_width",
    direction="minimize",
    score_data=nirs4all.TuningScoreData(
        X=X_score,
        y=y_score,
        conformal_coverage=0.9,
        conformal_calibration=nirs4all.TuningConformalScoreCalibration(
            X=X_dev_calibration,
            y_true=y_dev_calibration,
            physical_sample_ids=dev_calibration_ids,
        ),
    ),
)

The typed form is checked early: conformal_coverage is valid only with conformal_calibration, must be a finite numeric scalar in (0, 1), and conformal score metrics require that temporary calibration cohort. Numeric strings are rejected; coverage values must be real numeric scalars, not host values that can be converted with float(...). TuningScoreData(...) accepts either X/y or the score-cohort aliases X_score/y_score, plus either metric or its input alias score_metric, then emits canonical X/y and metric; it also accepts groups/score_groups and metadata/score_metadata, emitted canonically as groups/metadata. The score metric must be a real non-empty string, not an integer, boolean, blank string, or NUL-containing value; it is published as lowercase canonical metric before objective scoring. Metadata keys must be canonical non-empty strings for both column-style mappings and row-style sequences of mappings; raw NativeTuning.score_data.metadata follows the same rule. Metadata values must be strict JSON-native and finite: lists and nested mappings are accepted, but non-finite numbers, bytes, tuples and Python objects are rejected before runtime. Providing multiple aliases in any group is rejected as ambiguous. TuningConformalScoreCalibration(...) accepts exactly one feature alias among X, X_calibration, and features, plus exactly one target alias among y_true, y, y_calibration, target, and targets; it emits canonical X/y_true. It also accepts sample_ids, calibration_sample_ids, or physical_sample_ids, groups/calibration_groups, and metadata/calibration_metadata, then emits canonical sample_ids, groups, and metadata for the runtime. Metadata keys must be canonical non-empty strings for both column-style mappings and row-style sequences of mappings; raw score_data.conformal_calibration.metadata follows the same rule before NativeTuning.to_dict() can publish it, including the same strict JSON-native value boundary. The same one-alias rule is used by raw replayed-array calibrate(calibration_data={...}) mappings for y_pred_calibration / calibration_predictions, calibration_sample_ids / physical_sample_ids, calibration_groups, and calibration_metadata. Mapping payloads may use the read-only alias coverage, but conformal_coverage is the canonical spelling used by the registry. Multi-coverage lists belong to final calibration/prediction, not to the temporary tuning scorer. Final TuningCalibration.coverage accepts one numeric coverage or a non-empty unique list of numeric coverages; list elements such as "0.8" fail closed instead of being coerced. Raw mapping payloads follow the same alias exclusivity rules as the typed helpers for both standard and conformal-aware score data; nirs4all rejects ambiguous mappings before building the scorer.

This calibrator is temporary and candidate-local. It is used only to rank HPO trials; final intervals still come from tuning.winner plus top-level calibration or nested tuning.calibration.

The completed tuning_result.trials[*].diagnostics also records this boundary for audit: score_family="conformal", score_extractor="conformal_temporary_calibration", and final_calibration_scope="unmodified_by_score_data". The lightweight TuningResult.summary_artifact() mirrors a compact subset of those diagnostics under trials[*].diagnostics so bindings, Studio and CI cards can explain FAIL or conformal-aware trials without loading the full HPO tape. It keeps scalar audit keys such as error_type, score_family, score_extractor, search_space_fingerprint and tuning_fingerprint, but does not include candidate params or raw exception messages.

score_data can also use an explicit dataset-backed selector:

"score_data": {
    "dataset": spectro_dataset,  # or DatasetConfigs / config mapping / "dataset.json"
    "selector": {"partition": "val"},
    "sample_id_column": "Sample_ID",
    "group_column": "Batch",
    "metadata_columns": ["Site"],
}

The selector is mandatory here too. This keeps objective scoring auditable and prevents nirs4all from silently deriving a validation cohort. If the source is a DatasetConfigs, config mapping, or path, it must resolve to exactly one dataset. The Python helper form TuningScoreData(dataset=..., selector=...) executes this same selected cohort and transports sample_id_column, group_column, and metadata_columns to compatible predict() methods. Those dataset-backed column selectors are strict identifiers: sample_id_column and group_column must be canonical non-empty strings without surrounding whitespace or NULs, and metadata_columns must be either one canonical string or a duplicate-free sequence of canonical strings. Raw NativeTuning.score_data and NativeTuning.winner dataset-backed mappings use the same column-selector rule before publication. It also accepts sample_ids, score_sample_ids, prediction_sample_ids, or physical_sample_ids, serializing exactly one provided alias to canonical sample_ids. It rejects mixed dataset + explicit X/y arrays, matching the mapping runtime boundary.

Dataset-backed inputs must be explicit mappings. nirs4all does not guess a training partition:

dataset = {
    "dataset": spectro_dataset,  # or DatasetConfigs / config mapping / "dataset.json"
    "selector": {"partition": "train"},
    "sample_id_column": "Sample_ID",
    "group_column": "Batch",
    "metadata_columns": ["Site"],
}

include_augmented defaults to False for this native tuning subset. Set "include_augmented": True only when augmented samples are intended to be part of the fit cohort. Dataset-backed tuning selectors reject non-string or whitespace-padded keys plus non-JSON-native values, and include_augmented must be a boolean before TuningScoreData or TuningWinner can publish a runtime mapping. Config/path sources must resolve to exactly one dataset.

The same mapping form is available for tuning.winner, which is the cohort used for RunResult.best and conformal calibration:

"winner": {
    "dataset": spectro_dataset,  # or DatasetConfigs / config mapping / "dataset.json"
    "selector": {"partition": "test"},
    "sample_id_column": "Sample_ID",
    "group_column": "Batch",
    "metadata_columns": ["Site"],
    "score": 0.42,
    "metric": "rmse",
}

Do not combine this with explicit X/y_true arrays in the same winner payload. The typed helper form TuningWinner(dataset=..., selector=...) projects the same selected cohort into RunResult.best, preserving physical sample ids, optional groups, and selected metadata columns for calibration and audit, and rejects the same mixed dataset + explicit X/y_true payloads. Use one representation per cohort. TuningWinner.score must be a finite number, not a boolean or numeric string. TuningWinner.metadata and raw NativeTuning.winner.metadata use the same strict metadata key rule as TuningScoreData; their values must also stay strict JSON-native and finite. TuningWinner.metric, dataset_name, model_name and task_type are strict string fields: nirs4all rejects non-string values, blanks and NULs instead of calling str(...). metric and task_type are published lowercase; dataset and model labels are published trimmed.

For the winner cohort, sample_ids, prediction_sample_ids, and physical_sample_ids are accepted aliases. The typed TuningWinner(...) helper also accepts these aliases plus winner_sample_ids, and serializes exactly one provided alias to canonical sample_ids. Raw winner mappings follow the same fail-closed rule for X/winner_x, y_true/winner_y_true, score, metric, sample id, dataset name, model name, task type, and metadata aliases; provide one spelling per field. They are stored as physical sample ids in the prediction metadata and are required before the winner can be used as conformal calibration evidence. When the selected SpectroDataset target is a single-column matrix, the native tuning adapter normalizes it to a one-dimensional target vector before conformal calibration; multi-target matrices remain outside this public subset.

Calibrating from an explicit dataset cohort

nirs4all.calibrate() can also use a selected dataset cohort directly when you already have replayed point predictions for that exact cohort. The dataset value may be a SpectroDataset, a DatasetConfigs object, a dataset config mapping, or a config/path string accepted by DatasetConfigs. The dataset supplies the calibration targets and identities; either y_pred inside the mapping or top-level y_pred_calibration= supplies the replayed calibration predictions. Alternatively, predictor may provide an in-memory sklearn-like object; nirs4all extracts X from the same selected cohort and calls predict(X), forwarding sample_ids, groups and metadata when accepted by the predictor signature. For saved predictors, use predictor_bundle to route through nirs4all.predict() on the extracted X. Mapping aliases model_bundle, predictor_path and model_path are accepted for this same source; the typed helper emits canonical predictor_bundle. For previous nirs4all training outputs, use predictor_result=train_result.best; for a stored workspace chain, use predictor_chain_id together with an explicit nested workspace_path. The typed helper also accepts workspace_chain_id as a read alias and serializes it as canonical predictor_chain_id. Both spellings must carry a canonical non-empty chain id without surrounding whitespace or NULs; nirs4all rejects invalid values before calling predict(chain_id=...) or publishing replay provenance.

calibrated = nirs4all.calibrate(
    calibration_data={
        "dataset": spectro_dataset,  # or "calibration_dataset.json"
        "selector": {"partition": "calibration"},
        "sample_id_column": "Sample_ID",
        "group_column": "Batch",
        "metadata_columns": ["Site"],
        "y_pred": y_cal_pred,
    },
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

The typed helper serializes to the same payload and is the preferred shape for forms, bindings and Studio payload builders:

calibrated = nirs4all.calibrate(
    calibration_data=nirs4all.ConformalCalibrationData(
        dataset=spectro_dataset,
        selector={"partition": "calibration"},
        sample_id_column="Sample_ID",
        group_column="Batch",
        metadata_columns=["Site"],
        y_pred=y_cal_pred,
    ),
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

For in-memory replay, replace y_pred= with predictor=fitted_model. The selected calibration X, sample ids, groups and metadata are forwarded to predict() when the predictor accepts them. For saved model bundles, use predictor_bundle="model.n4a" in the same typed helper. For nirs4all replay sources, use predictor_result=train_result.best or predictor_chain_id="chain-id" with workspace_path="workspace/".

If you prefer to keep all prediction arrays at the top level:

calibrated = nirs4all.calibrate(
    calibration_data={
        "dataset": spectro_dataset,
        "selector": {"partition": "calibration"},
        "sample_id_column": "Sample_ID",
    },
    y_pred_calibration=y_cal_pred,
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

In-memory predictor replay:

calibrated = nirs4all.calibrate(
    calibration_data={
        "dataset": "calibration_dataset.json",
        "selector": {"partition": "calibration"},
        "sample_id_column": "Sample_ID",
        "group_column": "Batch",
        "metadata_columns": ["Site"],
        "predictor": fitted_model,
    },
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

Saved predictor replay:

calibrated = nirs4all.calibrate(
    calibration_data={
        "dataset": "calibration_dataset.json",
        "selector": {"partition": "calibration"},
        "sample_id_column": "Sample_ID",
        "predictor_bundle": "model.n4a",
    },
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

Previous-result replay:

calibrated = nirs4all.calibrate(
    calibration_data={
        "dataset": "calibration_dataset.json",
        "selector": {"partition": "calibration"},
        "sample_id_column": "Sample_ID",
        "predictor_result": train_result.best,
        "workspace_path": "workspace",
    },
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

Workspace-chain replay:

calibrated = nirs4all.calibrate(
    calibration_data={
        "dataset": "calibration_dataset.json",
        "selector": {"partition": "calibration"},
        "sample_id_column": "Sample_ID",
        "predictor_chain_id": "chain-abc123",
        "workspace_path": "workspace",
    },
    y_pred=y_pred_new,
    prediction_sample_ids=prediction_sample_ids,
    coverage=0.9,
)

Provide exactly one calibration replay source inside the mapping: y_pred, predictor, predictor_bundle, predictor_result, or predictor_chain_id. The selector is mandatory, physical sample ids must be explicit through sample_ids, calibration_sample_ids, physical_sample_ids, or sample_id_column, and mixed dataset + y_true/X payloads are rejected. Physical sample id values for calibration and prediction cohorts must be canonical non-empty strings without surrounding whitespace or NULs. Supplying more than one explicit sample-id alias for a dataset-backed calibration mapping is rejected as ambiguous; use sample_id_column only when the ids should be read from dataset metadata. Optional row-aligned groups and metadata can be supplied explicitly with groups/calibration_groups and metadata/calibration_metadata, or derived from group_column and metadata_columns. Dataset-backed selectors reject non-string or whitespace-padded keys plus non-JSON-native values, and raw mapping include_augmented must be a boolean. Dataset-backed predictor_chain_id/workspace_chain_id values are strict workspace chain ids, not labels; blanks, whitespace-padded strings, NULs and non-strings fail before replay. Dataset-backed calibration column selectors are canonical too: sample_id_column and group_column must be non-empty strings without surrounding whitespace or NULs, and metadata_columns must be one canonical string or a duplicate-free sequence of canonical strings. The same rule applies to ConformalCalibrationData(...) and raw calibration_data={...} mappings before cohort extraction. The nested workspace_path is used for predictor replay only; it is separate from the top-level workspace path used to save conformal results.

Applying a stored calibrator

Reload the conformal artifact from the workspace and apply it to new point predictions:

calibrated = nirs4all.load_workspace_calibrated_result(
    "workspace/",
    "ridge-conformal",
)

prediction = nirs4all.predict_calibrated(
    calibrated,
    y_pred=[13.0, 14.0],
    prediction_sample_ids=["pred-003", "pred-004"],
)

interval = prediction.interval(0.8)
print(interval.lower, interval.upper)

The direct predict_calibrated() example above does not replay a model. It applies the stored conformal calibrator to already computed point predictions. The prediction_sample_ids are required for every non-empty conformal prediction cohort and must align with y_pred. They must be non-empty canonical strings, unique physical sample identifiers, and disjoint from the calibration cohort physical sample ids embedded in the conformal artifact; empty ids, surrounding whitespace, NUL-containing ids, duplicating a prediction id, or reusing a calibration sample as a prediction row fail closed instead of displaying a leaked guarantee. The returned result preserves the original calibration_replay_source from the stored calibrated result; caller-supplied result_metadata cannot replace that source or the generated conformal_guarantee_status. The top-level source_calibrated_result_fingerprint must match the same value inside conformal_guarantee_status, so corrupted or mixed application metadata fails closed on reload. If you pass coverage=..., the value selects one or more already materialized coverages: it must be finite, unique when passed as a list, and strictly between 0 and 1. It does not recalibrate the conformal artifact. If the requested coverage was not materialized during calibration, predict() fails closed and reports the available coverages.

The same identity contract is rechecked when loading filesystem stores, workspace conformal rows, or .n4a conformal sidecars. A stored calibrated_result.json whose non-empty prediction cohort lacks canonical physical sample_ids fails reload instead of becoming a partial PredictResult. A corrupted conformal_results workspace row with the same missing prediction identity is rejected by both load_workspace_calibrated_result(...) and load_workspace_calibrated_predict_result(...). Workspace conformal metadata is also validated before insertion, so invalid metadata cannot be hidden in the row and then trusted on reload. Result-level metadata must also be strict JSON-compatible before construction, reload, fingerprinting or persistence: non-finite floats, Python objects, non-string keys, whitespace-padded keys, nested non-string mapping keys and tuple values that would otherwise be silently coerced fail closed. Grouped conformal results keep the same contract through these persistence paths: filesystem stores, workspace conformal_results rows, conformal-only .n4a bundles and model .n4a sidecars preserve group_keys, group_calibrators and row-aligned qhat vectors, then revalidate them and strict non-boolean integer grouped n_samples summaries against the embedded artifact before exposing a CalibratedRunResult or PredictResult. Version fields on conformal cohort manifests, calibration artifacts and calibrated results are strict integer contract tags: boolean true/false and numeric strings are rejected on direct construction and reload instead of being coerced to schema version 1. Optional conformal artifact identity strings (target_name, predictor_fingerprint, calibration_data_fingerprint) must be either None or non-empty strings without surrounding whitespace or NULs; reload rejects the same invalid values instead of publishing ambiguous provenance. ConformalCalibrationSpec validates direct construction the same way as parse_conformal_calibration_spec(): coverage values must be real numeric scalars, not booleans or numeric strings, and method, unit, group keys and multi-target mode are canonicalized before fingerprinting. Reloaded cohort manifest unit and calibrated prediction method/unit fields must already be strings; objects with a helpful __str__ are rejected instead of being stringified into valid contract values. Conformal calibration cohort rows built directly or reloaded from JSON now require a strict non-boolean integer row_index, canonical non-whitespace sample_id, role and optional group, plus strict JSON-native metadata with string keys, before the cohort manifest can be fingerprinted. The optional serialized n_samples summary must also be a strict non-boolean integer matching the row count. Row-aligned calibration metadata supplied as either column mappings or per-row mappings uses the same strict key rule: non-string or whitespace-padded metadata keys fail before they can be coerced into manifest JSON. Conformal numeric arrays (y_true, y_pred, interval bounds and qhat) must contain real numeric values; boolean payloads fail closed instead of being coerced to 0.0/1.0, and numeric strings such as "1.0" are rejected instead of being parsed as floats. The same bool rejection applies to Python-side from_dict(...) payloads carrying NumPy boolean scalars in serialized scores or quantiles, and serialized numeric fields reject NumPy ndarray scalars instead of coercing them to JSON numbers. Direct ConformalIntervalBlock / CalibratedPredictionBlock construction is also fail-closed for mismatched coverage keys, mismatched interval shapes, inverted bounds, invalid method/unit values, invalid group-key lengths, and negative or non-row-aligned qhat values. Direct SplitConformalCalibrator construction validates retained residual scores, coverage keys, recomputed quantiles, method and unit before apply(); negative scores, edited qhat values or unsupported vocabulary fail closed instead of materializing invalid intervals. Empirical ConformalMetricSet diagnostics are checked the same way at construction: observed_coverage must be finite in [0, 1] and match n_covered / n_samples, coverage_gap must equal observed_coverage - coverage, and widths/interval scores must be non-negative or positive infinity for unbounded intervals.

For model .n4a bundles carrying a conformal sidecar, predict(..., coverage=...) replays the model bundle first, then applies the attached calibrator. This currently applies intervals only to the single selected prediction entry (all_predictions=False). all_predictions=True remains fail-closed until each returned prediction entry can carry its own calibrated identity mapping. If the bundle contains a conformal/ sidecar but that sidecar is incomplete, duplicated, or contains unexpected members, predict() fails sidecar validation instead of falling back to an uncalibrated prediction path. The same fail-closed path applies when the sidecar looks structurally complete but its calibrated_result.json has a non-empty prediction cohort without canonical physical sample_ids: predict() rejects the sidecar before running the raw model prediction.

Audit-only robustness report

When observed targets are available for an external prediction cohort, nirs4all.robustness() can produce the current audit-only robustness report:

report = nirs4all.robustness(
    prediction,
    y_true=[13.2, 13.7],
    X=X_external,
    predictor=frozen_predictor,
    mode="clean_frozen",
    scenarios=[
        nirs4all.RobustnessScenarioSpec(kind="observed"),
        nirs4all.RobustnessScenarioSpec(kind="prediction_bias", severity=0.2),
        nirs4all.RobustnessScenarioSpec(kind="prediction_noise", severity=0.05),
        nirs4all.RobustnessScenarioSpec(kind="spectral_noise", severity=0.001),
    ],
    metadata={"instrument": ["inst-a", "inst-b"], "batch": ["b1", "b2"]},
    slice_by=["instrument"],
    seed=123,
    workspace_path="workspace/",
    workspace_robustness_id="external-audit",
    workspace_name="External robustness audit",
)

print(report.scenarios[0].metrics.rmse)
print(report.scenarios[0].conformal_metrics[0.8].observed_coverage)
print(report.scenarios[0].slices[0].metrics.mae)
print(report.summary_rows())
print(report.degradation_rows())
print(report.worst_slices(metric="rmse", top_k=3))

# Convenience syntax when you already hold a result object:
same_report_from_prediction = prediction.robustness(y_true=[13.2, 13.7])

report.save_json("robustness-report.json")
report.save_summary("robustness-summary.json")
report.save_markdown("robustness-report.md")
report.save_html("robustness-report.html")
report.save_parquet("robustness-report.parquet")
report.save_artifacts("robustness-artifacts")
same_report = nirs4all.RobustnessReport.load_json("robustness-report.json")
same_report_from_tables = nirs4all.RobustnessReport.load_parquet("robustness-report.parquet")
same_report_from_bundle = nirs4all.RobustnessReport.load_artifacts("robustness-artifacts")

robustness_id = nirs4all.save_workspace_robustness_report(
    "workspace/",
    report,
    robustness_id="external-audit-copy",
    name="External robustness audit",
)
same_report_from_workspace = nirs4all.load_workspace_robustness_report(
    "workspace/",
    "external-audit",
)

The same pattern composes directly with calibrated predictions produced by the native tuning example:

prediction = nirs4all.predict_calibrated(
    restored_conformal,
    y_pred=[13.0, 14.0],
    prediction_sample_ids=["pred-003", "pred-004"],
)

report = prediction.robustness(
    y_true=[13.2, 13.7],
    scenarios=[
        nirs4all.RobustnessScenarioSpec(kind="observed"),
        nirs4all.RobustnessScenarioSpec(kind="prediction_bias", severity=0.1),
        nirs4all.RobustnessScenarioSpec(kind="prediction_noise", severity=0.05),
    ],
    metadata={"batch": ["external-a", "external-b"]},
    slice_by=["batch"],
    seed=7,
    workspace_path="workspace/",
    workspace_robustness_id="u09-robustness",
    workspace_name="U09 robustness audit",
)

report.save_artifacts("workspace/u09-robustness-artifacts")
same_report = nirs4all.load_workspace_robustness_report("workspace/", "u09-robustness")
same_bundle = nirs4all.RobustnessReport.load_artifacts("workspace/u09-robustness-artifacts")

This is the current public Python E2E lane: run(tuning=...) → calibrate/predict_calibrated() → PredictResult.robustness() with explicit external y_true. It remains audit-only: robustness scenarios do not refit, recalibrate, or renew the conformal guarantee.

For CI/release publication, an already verified JSON report can also be republished without re-running the audit:

nirs4all robustness-report robustness-report.json --format markdown --output robustness-report.md
nirs4all robustness-report robustness-report.json --format summary --output robustness-summary.json
nirs4all robustness-report robustness-report.json --format html --output robustness-report.html
nirs4all robustness-report robustness-report.json --format parquet --output robustness-report.parquet
nirs4all robustness-report robustness-report.json --format artifacts --output robustness-artifacts/
nirs4all robustness-report robustness-artifacts/ --format markdown --output reviewed-report.md
nirs4all workspace robustness list --workspace workspace --json
nirs4all workspace robustness show external-audit --workspace workspace --json
nirs4all workspace robustness from-prediction --workspace workspace --prediction-id pred-001 --y-true "1.0,2.0,3.0" --scenarios-json '[{"kind":"spectral_offset","severity":0.01}]' --save-to-workspace --workspace-robustness-id pred-001-spectral-audit --format summary --output robustness-summary.json
nirs4all workspace robustness export external-audit --workspace workspace --format summary --output robustness-summary.json
nirs4all workspace robustness export external-audit --workspace workspace --format artifacts --output robustness-artifacts/
nirs4all workspace robustness export external-audit --workspace workspace --format html --output robustness-report.html

workspace robustness from-prediction is the CLI form of nirs4all.robustness_from_workspace_prediction(...). It keeps the same fail-closed boundary: stored provenance markers do not become executable spectra, but actual row-aligned X/spectra plus predictor_bundle evidence are consumed as defaults for spectral/OOD scenarios.

This does not recalibrate or refit. Post-prediction scenarios compute point metrics and conformal diagnostics on already materialized predictions: prediction_bias adds severity to y_pred, and prediction_noise draws seeded normal or centered uniform noise. normal uses severity as standard deviation; uniform samples from [-severity, +severity]. In both cases existing conformal interval bounds are shifted with the prediction delta. If seed is omitted, nirs4all records and uses effective_seed=0, so repeated reports are still deterministic. The explicit-X spectral cells require X and exactly one frozen replay source: either an in-memory predictor or a saved predictor_bundle. The in-memory form accepts predictor.predict(X) or a callable. The saved-bundle form routes the perturbed matrix through public nirs4all.predict(model=predictor_bundle, data={"X": X, "sample_ids": ...}, all_predictions=False), preserving row identity when the prediction result carries sample ids. Both forms replay without fit or recalibration and recenter existing conformal intervals by the prediction delta. predictor and predictor_bundle are mutually exclusive. Spectral reports include metadata["spectral_replay"] so auditors can distinguish in-memory predictor replay from saved-bundle replay, see the saved bundle path, confirm the nirs4all.predict route, and verify whether sample ids were forwarded. If a PredictResult already carries actual row-aligned metadata["X"] or metadata["spectra"] plus published robustness_evidence.predictor_bundle or model_path, nirs4all.robustness(result, ...) and result.robustness(...) use those values as the spectral replay defaults. This only consumes real arrays stored on the result; strings such as "prediction_arrays.X" are provenance markers and remain fail-closed unless the array itself is present. For workspace-loaded predictions, convert the stored record explicitly:

from nirs4all.data.predictions import Predictions

predictions = Predictions.from_workspace("workspace/", load_arrays=True)
result = predictions.get_predict_result_by_id("pred-001")
record = predictions.get_prediction_by_id("pred-001", load_arrays=True)

# Direct public one-record shortcut:
result = nirs4all.load_workspace_predict_result("workspace/", "pred-001")

# Direct public bulk shortcut:
results = nirs4all.load_workspace_predict_results("workspace/")

# Direct workspace prediction -> robustness report shortcut:
report = nirs4all.robustness_from_workspace_prediction(
    "workspace/",
    "pred-001",
    y_true=record["y_true"],
    scenarios=[{"kind": "spectral_offset", "severity": 0.01}],
    save_to_workspace=True,
    workspace_robustness_id="pred-001-spectral-audit",
)

if result.spectral_replay_evidence_status["status"] == "ready_for_spectral_replay":
    report = result.robustness(
        y_true=record["y_true"],
        scenarios=[{"kind": "spectral_noise", "severity": 0.01}],
    )

nirs4all.load_workspace_predict_result() is the direct public one-record workspace path. nirs4all.load_workspace_predict_results() is the direct public bulk workspace path, with optional dataset_name=... filtering. Predictions.get_predict_result_by_id() is the lower-level buffer equivalent; Predictions.to_predict_results() performs the same conversion for the whole buffer. All four call PredictResult.from_prediction_record() internally and preserve result_metadata, robustness_evidence, executable X/spectra, sample_indices, model name, preprocessing metadata, calibration_replay_source and tuning_calibration_source. They intentionally require load_arrays=True; a metadata-only record is not enough to replay spectral/OOD scenarios. nirs4all.robustness_from_workspace_prediction() is the direct report path for one persisted prediction: it performs the same conversion and then calls robustness(), optionally saving the report back into the workspace with a prediction_id link. Python publishers can make the evidence executable by writing it through Predictions.add_prediction(..., X=..., spectra=..., result_metadata=...) before flush(). X/spectra are stored in the array sidecar, result_metadata.robustness_evidence records the predictor_bundle or model_path, and Predictions.merge_stores() preserves those fields during workspace consolidation. This is the supported Python publisher path for spectral/OOD replay evidence; provenance-only strings remain non-executable. spectral_noise adds seeded noise to X; normal uses severity as standard deviation and uniform samples from [-severity, +severity]. spectral_offset applies a deterministic additive offset; spectral_scale multiplies every feature by 1.0 + severity; spectral_slope adds a centered linear ramp across the feature axis; and spectral_shift shifts spectra by fractional feature units with interpolation. These stress cells are diagnostic only and do not renew a conformal coverage guarantee. Matched recalibration and structural refit remain planned gates.

scenarios accepts either the typed RobustnessScenarioSpec helper shown above or the equivalent mapping form, for example {"kind": "prediction_noise", "severity": 0.05}. The supported kind values are currently observed, prediction_bias, prediction_noise, spectral_noise, spectral_offset, spectral_scale, spectral_slope, and spectral_shift. severity is an offset for prediction_bias, a standard deviation for normal noise, a centered half-width for uniform noise, and the deterministic magnitude for the other spectral cells. distribution is optional and currently accepts "normal" or "uniform" for the two noise cells. extra={...} on RobustnessScenarioSpec can carry stable labels or audit metadata into the scenario payload, but it cannot override kind, severity, or distribution. Direct RobustnessScenarioSpec construction is fail-closed: kind and distribution must be real strings, not host objects that stringify to supported values; severity must be a real numeric scalar, not a boolean or numeric string, and extra keys must be canonical non-empty strings without NULs before to_dict() can publish the scenario. Scenario mappings, including extra={...} on the typed helper, use the same strict boundary: keys must be canonical non-empty strings without NULs, kind and distribution must be real strings, severity must be a real finite numeric scalar, and the payload must stay TCV1 JSON-native and fingerprintable. Plain strings, finite numbers, booleans, null, arrays, and objects with text keys are accepted; Python objects, NaN/Infinity, bytes, and other opaque values are rejected before report execution. The same no-stringify rule is applied again to published result payloads: RobustnessScenarioResult.scenario and RobustnessSliceResult.slice_key keys must be canonical non-empty strings without NULs, their values must stay TCV1 JSON-native, and RobustnessScenarioResult.severity must be a real numeric scalar. These mappings enter report JSON, fingerprints, summary_rows(), degradation_rows(), worst_slices(), tabular_records() and Studio/CI summary cards, so corrupted reloads fail closed instead of silently converting Python keys with str(...).

The JSON export is deterministic and verified on reload through the report fingerprint, which makes it suitable for CI artifacts and release audits. save_markdown() writes the same report as a deterministic human-readable summary for release notes, audit packets, or CI job summaries. The Markdown summary includes conformal guarantee status details when present, including requested/effective engine, method, unit, selected and calibrated coverages, scope, invalidation reasons and limitations; this is descriptive metadata and does not renew the guarantee. It also includes degradation versus the reference scenario and the worst diagnostic slices when slices are available. summary_rows(reference=0) returns one compact row per scenario for CI dashboards and Studio cards: point metrics, deltas/ratios versus the reference scenario, conformal coverage/width summary, the worst diagnostic slice, and an execution-scope hint. execution_scope is baseline, prediction_replay or spectral_replay; requires_spectral_replay=true marks explicit-X spectral/OOD replay rows and is metadata for orchestration, not permission for bindings or Studio to replay perturbations locally. Markdown and HTML exports render this same compact scenario summary, including scope and replay-evidence columns, before the detailed tables. save_html() writes a standalone escaped HTML artifact with the same report tables. save_parquet() writes a portable directory containing manifest.json and one Parquet file per non-empty flat report table, such as compact scenario summary, scenario metrics, degradation rows, conformal diagnostics and slices. It also stores report.json; manifest.json records table filenames, row counts, and fingerprints, so RobustnessReport.load_parquet() can reload the full report and verify that the exported table files still match the manifest. save_artifacts() writes a publication directory with manifest.json plus selected report.json, summary.json, report.md, report.html, and report.parquet/ artifacts. Use it when CI, release automation, bindings, or Studio need one stable directory to attach or inspect. summary.json is the lightweight UI/CI payload: fingerprint, mode, report version, slice keys, optional conformal_guarantee_status, optional spectral_replay, and compact summary_rows() with execution-scope hints. The guarantee and spectral replay blocks are copied from report metadata when present so cards and dashboards can show engine, coverage, invalidation details, replay source, bundle path and sample-id forwarding status without parsing the full report. Because report metadata enters RobustnessReport.fingerprint, its keys must be canonical non-empty strings without NULs and its values must stay TCV1 JSON-native; arbitrary Python objects or non-string keys fail before JSON publication. The same payload is available directly with summary_artifact(), to_summary_json() and save_summary(), and from CLI publication with --format summary. Its JSON Schema is available from get_robustness_summary_schema() and robustness_summary_schema_json(). The full bundle can be emitted from existing report/workspace CLI entry points with --format artifacts --output <directory>. RobustnessReport.load_artifacts() reloads a bundle from its manifest and verifies the embedded JSON or Parquet report fingerprint. When Markdown, HTML or Parquet artifacts are present, it compares them against the deterministic report rendering or table-level Parquet manifest. When summary.json is present, it is also compared against the deterministic summary payload. Missing or edited files fail closed before downstream CI, bindings or Studio consume the directory. The nirs4all robustness-report CLI accepts either a report JSON file or one of these verified artifact directories as input.

Workspace CLI

The workspace commands are read-only and useful in CI or release audits:

nirs4all workspace tuning list --workspace workspace/ --json
nirs4all workspace tuning show ridge-native-tuning --workspace workspace/ --json
nirs4all workspace conformal list --workspace workspace/ --json
nirs4all workspace conformal show ridge-conformal --workspace workspace/ --json
nirs4all workspace conformal predict ridge-conformal \
  --workspace workspace/ \
  --y-pred "13.0,14.0" \
  --sample-ids "pred-003,pred-004" \
  --json

show reloads and verifies the typed artifact before printing it. conformal predict emits calibrated intervals and guarantee metadata without modifying the workspace.

Executable example

See examples/user/04_models/U09_native_tuning_conformal.py for a complete smoke-tested script. It covers typed conformal-aware objective scoring with NativeTuning, TuningScoreData, and TuningConformalScoreCalibration, then keeps final conformal calibration on the projected winner, reloads the workspace artifacts, applies the stored calibrator to new point predictions, and includes a standalone ConformalCalibrationData calibration smoke path.

Reading the guarantee metadata

Both calibrated result containers and prediction results expose conformal_guarantee_status:

status = prediction.conformal_guarantee_status
print(status["status"])
print(status["coverage"])
print(status["limitations"])
print(prediction.calibration_replay_source["kind"])

The metadata records the method, unit, requested/effective coverage, fingerprints and known limitations. It also includes calibration_replay_source, a compact provenance block that tells whether calibration predictions were provided as arrays, a PredictResult, dataset-backed y_pred, an in-memory predictor, a saved predictor_bundle, a predictor_result, or a workspace predictor_chain_id; the block records the route (provided_arrays, predictor.predict, or nirs4all.predict) and bundle/chain details when applicable. Guarantee metadata string fields are not coerced: effective_engine, requested_engine, source_calibrated_result_fingerprint, and each invalidation_reasons entry must be non-empty strings without surrounding whitespace, and persisted conformal_guarantee_status.version must be the strict integer 1. Status predictor_fingerprint, calibration_data_fingerprint, guarantee, and scope are also rechecked against the embedded artifact on construction and reload. A persisted status must include the complete generated field set, and status must be active exactly when invalidation_reasons is empty, otherwise invalidated. The generated limitations list must also match the embedded artifact’s guarantee mode exactly; edited, shortened, empty or non-string limitation payloads fail closed before publication. Use prediction.calibration_replay_source or calibrated.calibration_replay_source for the direct accessor; it reads the nested status block first and falls back to the top-level result metadata for older or partially projected payloads. New CalibratedRunResult payloads fail closed if both locations carry calibration_replay_source but disagree, so notebooks, Studio and bindings cannot observe contradictory replay provenance. Empirical diagnostics such as observed coverage and interval width can be computed separately with nirs4all.conformal_metrics() when observed y_true values are available for the prediction cohort. Invalid or inconsistent metric payloads fail closed before fingerprinting, persistence or UI publication.