nirs4all.data.schema.config module

Schema definitions for dataset configuration.

This module defines Pydantic models for validating and normalizing dataset configurations. It supports both the legacy format (train_x, test_x, etc.) and the planned new format (files, sources, variations).

The models provide: - Type validation and coercion - Default value handling - Clear documentation via Field descriptions - Serialization/deserialization

class nirs4all.data.schema.config.AggregateMethod(value)

Bases: StrEnum

Method for aggregating predictions.

MEAN = 'mean'

MEDIAN = 'median'

VOTE = 'vote'

class nirs4all.data.schema.config.CategoricalMode(value)

Bases: StrEnum

Mode for handling categorical columns in Y data.

AUTO = 'auto'

NONE = 'none'

PRESERVE = 'preserve'

class nirs4all.data.schema.config.FoldConfig(*, folds: list[FoldDefinition] | None = None, file: str | None = None, format: Literal['auto', 'csv', 'json', 'yaml', 'txt'] | None = 'auto', column: str | None = None)

Bases: BaseModel

Configuration for cross-validation fold definitions.

Supports multiple ways to specify folds: - Inline: List of FoldDefinition objects - File: Path to a fold file (CSV, JSON, YAML) - Column: Column name in metadata containing fold assignments

Examples

# Inline fold definitions folds:

• train: [0, 1, 2, 3, 4] val: [5, 6, 7, 8, 9]

• train: [5, 6, 7, 8, 9] val: [0, 1, 2, 3, 4]

# File reference folds:

file: “path/to/folds.csv” format: auto

# Column in metadata folds:

column: “cv_fold”

column: str | None

file: str | None

folds: list[FoldDefinition] | None

format: Literal['auto', 'csv', 'json', 'yaml', 'txt'] | None

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

to_fold_list() → list[tuple[list[int], list[int]]] | None

Convert inline fold definitions to fold list.

Returns:

List of (train_indices, val_indices) tuples, or None if not inline.

validate_fold_source() → FoldConfig

Validate that exactly one fold source is specified.

class nirs4all.data.schema.config.FoldDefinition(*, train: list[int], val: list[int] = <factory>)

Bases: BaseModel

Definition of a single cross-validation fold.

Specifies which sample indices belong to training and validation sets.

model_config = {'extra': 'forbid'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

train: list[int]

val: list[int]

class nirs4all.data.schema.config.HeaderUnit(value)

Bases: StrEnum

Unit type for spectral headers.

INDEX = 'index'

NONE = 'none'

TEXT = 'text'

WAVELENGTH = 'nm'

WAVENUMBER = 'cm-1'

class nirs4all.data.schema.config.NAPolicy(value)

Bases: StrEnum

Policy for handling NA/missing values during data loading and pipeline execution.

ABORT = 'abort'

AUTO = 'auto'

IGNORE = 'ignore'

REMOVE_FEATURE = 'remove_feature'

REMOVE_SAMPLE = 'remove_sample'

REPLACE = 'replace'

class nirs4all.data.schema.config.PartitionConfig(*, type: PartitionType | None = None, column: str | None = None, train_values: list[str] | None = None, test_values: list[str] | None = None, predict_values: list[str] | None = None, unknown_policy: Literal['error', 'ignore', 'train'] | None = None, train: str | list[int] | None = None, test: str | list[int] | None = None, predict: str | list[int] | None = None, shuffle: bool | None = None, random_state: int | None = None, stratify: str | None = None, train_file: str | None = None, test_file: str | None = None, predict_file: str | None = None, **extra_data: Any)

Bases: BaseModel

Configuration for partition assignment.

Supports multiple partition methods: - Static: Assign entire file to a partition (use type) - Column-based: Partition based on column values (use column) - Percentage-based: Split by percentage (use train, test with percentages) - Index-based: Explicit index lists (use train, test with lists) - Index file: Load indices from external files (use train_file, test_file)

Examples

# Static partition (entire file) partition:

type: train

# Column-based partition partition:

column: “split” train_values: [“train”, “training”] test_values: [“test”, “validation”]

# Percentage-based partition partition:

train: “80%” test: “80%:100%” shuffle: true random_state: 42

# Index-based partition partition:

train: [0, 1, 2, 3, 4] test: [5, 6, 7, 8, 9]

# Index file partition partition:

train_file: “train_indices.txt” test_file: “test_indices.txt”

column: str | None

model_config = {'extra': 'allow'}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

predict: str | list[int] | None

predict_file: str | None

predict_values: list[str] | None

random_state: int | None

shuffle: bool | None

stratify: str | None

test: str | list[int] | None

test_file: str | None

test_values: list[str] | None

train: str | list[int] | None

train_file: str | None

train_values: list[str] | None

type: PartitionType | None

unknown_policy: Literal['error', 'ignore', 'train'] | None

validate_partition_method() → PartitionConfig

Validate that partition specification is consistent.

class nirs4all.data.schema.config.SignalTypeEnum(value)

Bases: StrEnum

Signal type for spectral data.

ABSORBANCE = 'absorbance'

AUTO = 'auto'

KUBELKA_MUNK = 'kubelka-munk'

LOG_1_R = 'log(1/R)'

REFLECTANCE = 'reflectance'

REFLECTANCE_PERCENT = 'reflectance%'

TRANSMITTANCE = 'transmittance'

TRANSMITTANCE_PERCENT = 'transmittance%'

class nirs4all.data.schema.config.TaskType(value)

Bases: StrEnum

Task type for the dataset.

AUTO = 'auto'

BINARY_CLASSIFICATION = 'binary_classification'

MULTICLASS_CLASSIFICATION = 'multiclass_classification'

REGRESSION = 'regression'