Preprocessing

This section covers spectral preprocessing techniques for NIRS data.

• Preprocessing Overview
  • Why Preprocessing Matters
  • Common NIRS Preprocessing Workflow
  • Quick Example
  • Available Operators
  • Recommended Preprocessing by Model Type
  • Preprocessing Order Rules
  • Task-Specific Preprocessing
  • Multi-Layer Preprocessing for Deep Learning
  • SciPy Functions Reference
  • Best Practices
  • See Also
• Preprocessing Cheatsheet
  • Classical ML (scikit-learn + libraries)
  • Neural Networks
  • Quick Rules of Thumb
  • Preprocessing Chains
  • Parameter Recommendations
  • Warning Signs
  • See Also
• Preprocessing Handbook
  • Philosophy: Multi-Layer Preprocessing
  • Preprocessing Categories
  • Preprocessing Order
  • Multi-Layer Preprocessing for Deep Learning
  • Task-Specific Preprocessing
  • Advanced Techniques
  • Validation & Quality Control
  • Summary
  • See Also
• Standard Normal Variate (SNV) Transformation
  • Overview
  • Implementation
  • Parameters
  • Use Cases
  • Examples
  • Technical Details
  • Comparison with StandardScaler
  • Migration from Previous Implementation
  • References
  • See Also
• Resampler - Wavelength Grid Resampling
  • Basic Usage
  • Parameters
  • Examples
  • How It Works
  • Controller Integration
  • Warnings and Errors
  • Best Practices
  • Technical Details
  • See Also
  • References
• TransferPreprocessingSelector Cheat Sheet
  • Purpose
  • Quick Start
  • Stages Overview
  • Presets
  • Generator Mode (preprocessing_spec)
  • Key Parameters
  • Common Patterns
  • Results API
  • Metrics Explained
  • ⚠️ Gotchas

Overview

Preprocessing is a critical step in NIRS data analysis. NIRS4ALL provides a comprehensive set of preprocessing operators that are sklearn-compatible and can be chained in pipelines.

📖 Preprocessing Overview

Comprehensive guide to available preprocessing techniques, operators, and when to use them.

Start Here

Preprocessing Overview

📋 Cheatsheet

Quick reference for preprocessing selection by model type.

Quick Ref

Preprocessing Cheatsheet

📚 Handbook

In-depth guide with theory, advanced techniques, and multi-layer preprocessing.

Complete

Preprocessing Handbook

📐 Resampling

Wavelength resampling and interpolation techniques.

Advanced

Resampler - Wavelength Grid Resampling

Available Preprocessing Methods

Method	Description	Use Case
SNV	Standard Normal Variate	Scatter correction
MSC / EMSC	Multiplicative Scatter Correction	Reference-based scatter correction
Derivatives	Savitzky-Golay, First/Second	Baseline removal, peak enhancement
NorrisWilliams	Gap derivative with segment smoothing	Noise-robust derivatives
WaveletDenoise	Multi-level wavelet denoising	Noise reduction with frequency selectivity
OSC	Orthogonal Signal Correction	Remove Y-orthogonal variation
EPO	External Parameter Orthogonalization	Remove external parameter influence
Detrend	Polynomial detrending	Linear/quadratic baseline removal
Baseline	ALS, AirPLS, ArPLS, SNIP, etc.	Advanced baseline correction
Normalization	Min-max, area, vector	Scale standardization
Smoothing	Gaussian, moving average	Noise reduction

Quick Example

from nirs4all.operators.transforms import SNV, SavitzkyGolay, NorrisWilliams, WaveletDenoise

pipeline = [
    SNV(),                                    # Scatter correction
    SavitzkyGolay(window_length=15, deriv=1), # First derivative
    # ... rest of pipeline
]

# Or use newer transforms
pipeline = [
    WaveletDenoise(wavelet='db4', level=3),    # Wavelet denoising
    NorrisWilliams(gap=5, segment=5, deriv=1), # Norris-Williams gap derivative
    # ... rest of pipeline
]

See Also

• Operator catalog: choose a job, then inspect the method - Complete operator reference

• Standard Normal Variate (SNV) Transformation - Detailed SNV documentation

• TransferPreprocessingSelector Cheat Sheet - Transfer learning preprocessing