nirs4all.data.parsers.files_parser module

Files parser for dataset configuration.

This parser handles the new ‘files’ syntax defined in the specification. Implemented in Phase 4 to support partition assignment.

The files syntax supports whole files with explicit train/test partitions. Column/row selectors, key joins, and column-based partition rules are refused by the positional loader. Use explicit train_x/train_y/train_group inputs or the declared relation materialization API for those requests.

Example

files:

  • path: data/measurements.csv partition: train

The sources syntax (Phase 6) allows specifying multiple feature sources for sensor fusion or multi-instrument datasets:

Example

sources:

  • name: “NIR” files:

    • path: data/NIR_train.csv partition: train

    • path: data/NIR_test.csv partition: test

    params:

    header_unit: nm signal_type: absorbance

  • name: “MIR” train_x: data/MIR_train.csv test_x: data/MIR_test.csv params:

    header_unit: cm-1 signal_type: absorbance

targets:

path: data/targets.csv partition: train

class nirs4all.data.parsers.files_parser.FilesParser[source]

Bases: BaseParser

Parser for new ‘files’ syntax configuration.

The files syntax provides: - Flexible column selection (by index, name, regex, range) - Row selection and filtering - Partition assignment per file or via partition config - Key-based sample linking across files

can_parse(input_data: Any) → bool[source]

Check if this is a files-format configuration.

Parameters:

input_data – The input to check.

Returns:

True if input has ‘files’ key with non-empty list.

parse(input_data: dict[str, Any]) → ParserResult[source]

Parse a files-format configuration.

Parameters:

input_data – Dictionary configuration to parse.

Returns:

ParserResult with parsed configuration.

class nirs4all.data.parsers.files_parser.SourcesParser[source]

Bases: BaseParser

Parser for multi-source ‘sources’ syntax configuration.

The sources syntax provides: - Named feature sources (e.g., NIR, MIR spectrometers) - Per-source loading parameters - Automatic source alignment by sample key - Shared targets and metadata across sources

Example configuration:
sources:

  • name: “NIR” files:

    • path: data/NIR_train.csv partition: train

    • path: data/NIR_test.csv partition: test

    params:

    header_unit: nm signal_type: absorbance

  • name: “MIR” train_x: data/MIR_train.csv test_x: data/MIR_test.csv params:

    header_unit: cm-1

targets:

path: data/targets.csv partition: train

metadata:

path: data/metadata.csv partition: train

can_parse(input_data: Any) → bool[source]

Check if this is a sources-format configuration.

Parameters:

input_data – The input to check.

Returns:

True if input has ‘sources’ key with non-empty list.

parse(input_data: dict[str, Any]) → ParserResult[source]

Parse a sources-format configuration.

Converts the sources syntax to a DatasetConfigSchema that can be further converted to legacy format for backward compatibility.

Parameters:

input_data – Dictionary configuration to parse.

Returns:

ParserResult with parsed configuration.

class nirs4all.data.parsers.files_parser.VariationsParser[source]

Bases: BaseParser

Parser for feature variations ‘variations’ syntax configuration.

The variations syntax provides: - Named feature variations (e.g., raw, snv, derivative) - Per-variation loading parameters - Preprocessing provenance tracking - Multiple variation modes (separate, concat, select, compare)

Example configuration:
variations:

  • name: “raw” files:

    • path: data/spectra_raw.csv partition: train

    • path: data/spectra_raw_test.csv partition: test

  • name: “snv” description: “SNV preprocessed spectra” preprocessing_applied:

    • type: “SNV” software: “OPUS 8.0”

    train_x: data/spectra_snv_train.csv test_x: data/spectra_snv_test.csv

variation_mode: separate # or concat, select, compare variation_select: [“raw”, “snv”] # only for mode=select

targets:

path: data/targets.csv partition: train

can_parse(input_data: Any) → bool[source]

Check if this is a variations-format configuration.

Parameters:

input_data – The input to check.

Returns:

True if input has ‘variations’ key with non-empty list.

parse(input_data: dict[str, Any]) → ParserResult[source]

Parse a variations-format configuration.

Converts the variations syntax to a DatasetConfigSchema that can be further converted to legacy format for backward compatibility.

Parameters:

input_data – Dictionary configuration to parse.

Returns:

ParserResult with parsed configuration.