nirs4all.pipeline.execution.step_cache module

In-memory preprocessed data snapshots for pipeline step caching.

Stores lightweight feature snapshots after each preprocessing step, keyed by (chain_path_hash, data_hash, selector_fingerprint). For generator sweeps with shared preprocessing prefixes and for the refit pass replaying the same chain, caching eliminates redundant computation.

Uses copy-on-write (CoW) SharedBlocks references instead of copy.deepcopy() for near-zero-cost cache hits. The cached SharedBlocks are never mutated; when the next pipeline step writes to the dataset, _prepare_for_mutation() triggers a CoW detach that copies only once, on demand.

class nirs4all.pipeline.execution.step_cache.CachedStepState(source_snapshots: list[tuple[~nirs4all.data._features.array_storage.SharedBlocks, list[str], list[str] | None, str]] = <factory>, processing_names: list[list[str]] = <factory>, content_hash: str = '', bytes_estimate: int = 0)

Bases: object

Lightweight CoW snapshot of dataset feature state after a preprocessing step.

Uses SharedBlocks copy-on-write references instead of deep-copied FeatureSource objects. Restore is O(n_sources) — near-free — because it only acquires an additional reference to each shared array. The actual copy is deferred until the next pipeline step mutates the data.

Variables:

• source_snapshots (list[tuple[nirs4all.data._features.array_storage.SharedBlocks, list[str], list[str] | None, str]]) – Per-source CoW snapshot tuples.

• processing_names (list[list[str]]) – Processing chain per source (list of lists of str).

• content_hash (str) – Post-step content hash of the dataset.

• bytes_estimate (int) – Estimated memory footprint (computed once at store time).

bytes_estimate: int = 0

content_hash: str = ''

processing_names: list[list[str]]

release() → None

Release all SharedBlocks references held by this state.

Called when the entry is evicted from the cache or on clear().

source_snapshots: list[tuple[SharedBlocks, list[str], list[str] | None, str]]