Data Handling

This section covers loading and managing spectroscopic data in NIRS4ALL.

• Loading Data
  • Overview
  • Quick Start
  • Supported Formats
  • File Keys Reference
  • Wavelength Headers
  • Signal Type
  • Task Type
  • Multi-Source Datasets
  • Sample Aggregation
  • Multiple Datasets
  • Using SpectroDataset Directly
  • Complete Example
  • Common Patterns
  • Troubleshooting
  • See Also
• Synthetic Data Generation
  • Overview
  • Quick Start
  • Convenience Functions
  • Builder API
  • Non-Linear Target Complexity
  • Environmental and Matrix Effects
  • Validation and Benchmarking
  • Configuration Options
  • Exporting Synthetic Data
  • Matching Real Data
  • Advanced: Custom Component Library
  • Integration with Pipelines
  • Performance
  • API Reference
  • See Also
• Signal Types and Conversion
  • Overview
  • Supported Signal Types
  • Automatic Detection
  • Manual Signal Type Setting
  • Signal Conversion
  • Working with Multiple Sources
  • Complete Example
  • API Reference
  • See Also
• Sample Repetition & Aggregation
  • Overview
  • Quick Start
  • Setting Repetition
  • Pipeline Output
  • Visualization with Aggregation
  • Multi-Dataset Configuration
  • Priority Resolution
  • Aggregation Algorithm
  • Complete Example
  • Repetition Details
  • Advanced Aggregation Configuration
  • Complete Example
  • Properties and Methods Reference
  • See Also
• Heterogeneous Source Repetitions
  • Required Blocks
  • Representation Choices
  • Fit Influence
  • Stacking And Missing Sources
  • Replay And Explainability
  • Minimal Examples
• Tutorial: Heterogeneous Source Repetitions
  • 1. Start From A Relation Config
  • 2. Parse The Contract
  • 3. Build A Raw Multi-source Dataset
  • 4. Materialize A Sample-level Matrix
  • 5. Try A Cartesian Representation
  • 6. Use The Pipeline Boundary
  • 7. Compare The Shipped Examples
• Raw multimodal pipelines
  • Run the demonstration
  • Describe inputs
  • Declare independent experimental units
  • Encode variable-length series
  • Handle absent modalities
  • Declare targets
  • Encode, fuse and tune
  • Select independent source outputs
  • Tune a late-fusion ensemble
  • Late fusion with absent modalities and partial regression targets
  • Executable synthetic data providers
  • Archive and scope
• Fusion tardive avec observations partielles
• Native multimodal early fusion
• Octave models in a Python multimodal workflow
  • Prepare the runtime
  • Train and save all five models
  • Predict in a new process
  • Use the same SDK entry points with another controller
• Sample Filtering User Guide
  • Overview
  • Quick Start
  • Available Filters
  • Combining Filters
  • Pipeline Integration Options
  • Managing Exclusions
  • Filtering Reports
  • Visualization
  • Best Practices
  • Edge Cases and Warnings
  • Troubleshooting
  • API Reference
  • See Also

Overview

NIRS4ALL provides flexible data handling capabilities for loading spectroscopic data from various formats, generating synthetic data, filtering samples, and aggregating predictions.

📂 Loading Data

Load data from CSV, Excel, MATLAB, NumPy, and Parquet formats.

Essential

Loading Data

🧪 Synthetic Data

Generate realistic synthetic NIRS spectra for testing and prototyping.

New in 0.6

Synthetic Data Generation

🔬 Signal Types

Detect and convert between absorbance, reflectance, and other spectral representations.

Essential

Signal Types and Conversion

🔁 Repetitions & Aggregation

Handle repeated scans of the same physical sample: group replicates with DatasetConfigs(repetition=...) and aggregate them (aggregate, aggregate_method, outlier exclusion). Core to NIRS.

NIRS

Sample Repetition & Aggregation

Source-aware Repetitions

Declare heterogeneous source repetitions such as MIR=2, Raman=3, NIR=2 with explicit sample identity, representation, reducer, fit influence, and replay contracts.

Experimental

Heterogeneous Source Repetitions

Heterogeneous Repetitions Tutorial

Build the shipped MIR/Raman/NIRS relation fixture, materialize sample-level and cartesian representations, and connect the staged data to rep_fusion.

Experimental Tutorial

Tutorial: Heterogeneous Source Repetitions

🔍 Sample Filtering

Filter samples based on metadata, outliers, or custom criteria.

Preprocessing

Sample Filtering User Guide

Coming Soon

• Loading Data - Complete guide to DatasetConfigs and supported formats (CSV, Excel, MATLAB, NumPy, Parquet)

See Also

• Getting Started - Quick start guide

• Writing a Pipeline in nirs4all - Pipeline syntax reference