nirs4all.operators.models.pytorch.spectral_transformer module

Spectral Transformer - Modern Transformer Architecture for NIR Spectral Data.

This module provides a state-of-the-art transformer model specifically designed for Near-Infrared (NIR) spectroscopy data classification and regression tasks.

Key Features: - Learnable positional encoding for spectral data - Pre-LayerNorm transformer blocks for stable training - Patch embedding for efficient sequence handling - Multi-scale attention with configurable heads - Class token for aggregation (CLS token approach) - Proper regularization (dropout, stochastic depth) - Support for binary, multi-class classification, and regression

Architecture inspired by: - Vision Transformer (ViT) for patch-based processing - Pre-LN Transformer for training stability - DeBERTa’s disentangled attention concepts

Designed for ~4k samples with typical NIR spectral dimensions (100-2000 wavelengths).

class nirs4all.operators.models.pytorch.spectral_transformer.SpectralTransformer(*args: Any, **kwargs: Any)

Bases: Module

Modern Transformer for NIR Spectral Classification.

Designed specifically for spectroscopy data with: - Patch-based processing for efficiency - CLS token for classification - Pre-LayerNorm for stable training - Stochastic depth for regularization - Support for 2-class, multi-class, and regression

Parameters:

• input_shape – Tuple of (channels, seq_len) for input spectra.

• num_classes – Number of output classes (1 for regression, 2+ for classification).

• embed_dim – Transformer embedding dimension.

• depth – Number of transformer blocks.

• num_heads – Number of attention heads.

• patch_size – Size of spectral patches.

• ff_mult – Feed-forward expansion multiplier.

• dropout – General dropout rate.

• drop_path – Maximum stochastic depth rate.

• pool – Pooling method - ‘cls’ for CLS token, ‘mean’ for mean pooling.

forward(x: torch.Tensor) → torch.Tensor

Forward pass.

Parameters:

x – Input tensor of shape (batch, channels, seq_len)

Returns:

Output predictions of shape (batch, num_classes) or (batch, 1)

nirs4all.operators.models.pytorch.spectral_transformer.spectral_transformer(input_shape: tuple[int, int], params: dict[str, Any] | None = None) → SpectralTransformer

Create a SpectralTransformer for regression.

Suitable for ~4k samples of NIR spectral data.

Parameters:

• input_shape – Tuple of (channels, seq_len).

• params – Configuration dictionary with keys: - embed_dim (int): Embedding dimension (default: 64) - depth (int): Number of transformer blocks (default: 4) - num_heads (int): Number of attention heads (default: 4) - patch_size (int): Spectral patch size (default: auto) - ff_mult (float): Feed-forward multiplier (default: 4.0) - dropout (float): Dropout rate (default: 0.15) - drop_path (float): Stochastic depth rate (default: 0.1) - pool (str): Pooling method ‘cls’ or ‘mean’ (default: ‘cls’)

Returns:

SpectralTransformer model instance.

nirs4all.operators.models.pytorch.spectral_transformer.spectral_transformer_classification(input_shape: tuple[int, int], num_classes: int = 2, params: dict[str, Any] | None = None) → SpectralTransformer

Create a SpectralTransformer for classification.

Suitable for binary and multi-class classification on NIR spectral data.

Parameters:

• input_shape – Tuple of (channels, seq_len).

• num_classes – Number of classes (2 for binary, 3+ for multi-class).

• params – Configuration dictionary (see spectral_transformer).

Returns:

SpectralTransformer model instance for classification.

nirs4all.operators.models.pytorch.spectral_transformer.spectral_transformer_large(input_shape: tuple[int, int], params: dict[str, Any] | None = None) → SpectralTransformer

Large SpectralTransformer variant - higher capacity for larger datasets.

Recommended for datasets with >5k samples.

Parameters:

• input_shape – Tuple of (channels, seq_len).

• params – Additional configuration parameters.

Returns:

Large SpectralTransformer model instance.

nirs4all.operators.models.pytorch.spectral_transformer.spectral_transformer_large_classification(input_shape: tuple[int, int], num_classes: int = 2, params: dict[str, Any] | None = None) → SpectralTransformer

Large SpectralTransformer for classification.

Parameters:

• input_shape – Tuple of (channels, seq_len).

• num_classes – Number of classes.

• params – Additional configuration parameters.

Returns:

Large SpectralTransformer model instance for classification.

nirs4all.operators.models.pytorch.spectral_transformer.spectral_transformer_small(input_shape: tuple[int, int], params: dict[str, Any] | None = None) → SpectralTransformer

Small SpectralTransformer variant - faster training, less overfitting risk.

Recommended for smaller datasets (<1k samples).

Parameters:

• input_shape – Tuple of (channels, seq_len).

• params – Additional configuration parameters.

Returns:

Small SpectralTransformer model instance.

nirs4all.operators.models.pytorch.spectral_transformer.spectral_transformer_small_classification(input_shape: tuple[int, int], num_classes: int = 2, params: dict[str, Any] | None = None) → SpectralTransformer

Small SpectralTransformer for classification.

Parameters:

• input_shape – Tuple of (channels, seq_len).

• num_classes – Number of classes.

• params – Additional configuration parameters.

Returns:

Small SpectralTransformer model instance for classification.