Signal Types and Conversion

Overview

NIRS data can be in different physical representations depending on the instrument and measurement technique. nirs4all automatically detects and converts between these signal types to ensure consistent analysis.

Understanding signal types is crucial because:

• Different instruments output different signal types

• Some algorithms perform better with specific representations

• Conversion affects the scale and interpretation of spectral features

• Proper signal type handling is essential for multi-source data fusion

Supported Signal Types

Type	Range	Common Use	Description
absorbance	[0, ∞)	Lab spectrometers	-log₁₀(I/I₀), proportional to concentration
reflectance	[0, 1]	Portable devices	I/I₀ as fraction
reflectance%	[0, 100]	Older instruments	100×(I/I₀) as percentage
transmittance	[0, 1]	Thin samples	I/I₀ for transmitted light
transmittance%	[0, 100]	Older instruments	100×(I/I₀) as percentage
log_1_r	(-∞, 0]	Pre-converted	log₁₀(1/R) - pseudo-absorbance
log_1_t	(-∞, 0]	Pre-converted	log₁₀(1/T) - absorbance from transmittance
kubelka_munk	[0, ∞)	Diffuse reflectance	(1-R)²/(2R) for scattering media

Special Signal Types

Type	Description	Use Case
auto	Automatic detection	Default for new datasets
unknown	Cannot be determined	Ambiguous data ranges
preprocessed	Already preprocessed	SNV, derivatives, mean-centered data

Automatic Detection

nirs4all can automatically detect the signal type using value range analysis and wavelength band characteristics:

from nirs4all.data import DatasetConfigs

# Load data (signal type detected automatically)
dataset = DatasetConfigs("path/to/data")

# Check detected signal type
signal_type, confidence, reason = dataset.detect_signal_type(src=0)
print(f"Detected: {signal_type.value}")
print(f"Confidence: {confidence:.1%}")
print(f"Reason: {reason}")

Detection Logic

The detection algorithm uses multiple heuristics:

1. Value Range Analysis: Examines min/max/mean values

   • [0, 1] → likely reflectance/transmittance

   • [0, 100] → likely percent reflectance/transmittance

   • [0, 3+] → likely absorbance

   • Negative values → likely preprocessed

2. Water Band Direction (if wavelengths available):

   • Peaks at 1450, 1940, 2500 nm → absorbance

   • Dips at water bands → reflectance/transmittance

3. Statistical Indicators:

   • Mean near 0, std near 1 → preprocessed (SNV)

   • Symmetric distribution around 0 → preprocessed (derivative)

Manual Signal Type Setting

Override automatic detection when you know the signal type:

from nirs4all.data.signal_type import SignalType

# Force signal type
dataset.set_signal_type("absorbance", src=0, forced=True)

# Using enum
dataset.set_signal_type(SignalType.REFLECTANCE, src=0, forced=True)

# Using common abbreviations
dataset.set_signal_type("R", src=0, forced=True)  # Reflectance
dataset.set_signal_type("A", src=0, forced=True)  # Absorbance
dataset.set_signal_type("%R", src=0, forced=True) # Reflectance percent

Signal Type Aliases

Many string aliases are supported for convenience:

Signal Type	Accepted Aliases
Absorbance	"a", "abs", "absorbance", "A"
Reflectance	"r", "ref", "refl", "R"
Reflectance %	"%r", "r%", "reflectance%"
Transmittance	"t", "trans", "T"
Kubelka-Munk	"km", "kubelka_munk", "f(r)"

Signal Conversion

Convert between signal types using sklearn-compatible transformers:

from nirs4all.operators.transforms.signal_conversion import (
    ToAbsorbance,
    FromAbsorbance,
    KubelkaMunk,
    SignalTypeConverter
)
from sklearn.preprocessing import StandardScaler
from sklearn.cross_decomposition import PLSRegression

# Pipeline 1: Convert reflectance to absorbance
pipeline_abs = [
    ToAbsorbance(source_type="reflectance"),
    StandardScaler(),
    {"model": PLSRegression(n_components=10)}
]

# Pipeline 2: Convert to Kubelka-Munk (for powders)
pipeline_km = [
    KubelkaMunk(source_type="reflectance"),
    StandardScaler(),
    {"model": PLSRegression(n_components=10)}
]

# Pipeline 3: General converter
from nirs4all.operators.transforms.signal_conversion import SignalTypeConverter

pipeline_general = [
    SignalTypeConverter(source_type="reflectance%", target_type="absorbance"),
    StandardScaler(),
    {"model": PLSRegression(n_components=10)}
]

Common Conversions

Reflectance to Absorbance

from nirs4all.operators.transforms.signal_conversion import ToAbsorbance
import numpy as np

# Reflectance data (0-1 range)
R = np.array([[0.5, 0.4, 0.3], [0.6, 0.5, 0.4]])

# Convert to absorbance
converter = ToAbsorbance(source_type="reflectance")
A = converter.fit_transform(R)
# A ≈ [[0.301, 0.398, 0.523], [0.222, 0.301, 0.398]]

# Round-trip (verify conversion)
R_back = converter.inverse_transform(A)
print(f"Round-trip error: {np.abs(R - R_back).max():.2e}")

Absorbance to Reflectance

from nirs4all.operators.transforms.signal_conversion import FromAbsorbance

# Absorbance data
A = np.array([[0.3, 0.4, 0.5], [0.2, 0.3, 0.4]])

# Convert to reflectance
converter = FromAbsorbance(target_type="reflectance")
R = converter.fit_transform(A)

Percent to Fraction

from nirs4all.operators.transforms.signal_conversion import PercentToFraction

# Reflectance percent (0-100 range)
R_pct = np.array([[50, 40, 30], [60, 50, 40]])

# Convert to fraction (0-1 range)
converter = PercentToFraction()
R_frac = converter.fit_transform(R_pct)
# R_frac ≈ [[0.5, 0.4, 0.3], [0.6, 0.5, 0.4]]

Kubelka-Munk Transformation

from nirs4all.operators.transforms.signal_conversion import KubelkaMunk

# Reflectance data for diffuse reflectance (powders, granules)
R = np.array([[0.5, 0.4, 0.3], [0.6, 0.5, 0.4]])

# Apply Kubelka-Munk transformation: F(R) = (1-R)² / (2R)
converter = KubelkaMunk(source_type="reflectance")
F_R = converter.fit_transform(R)

# Inverse transform
R_back = converter.inverse_transform(F_R)

When to Use Each Conversion

Use Absorbance when:

• Working with liquid samples (Beer-Lambert law applies)

• You need proportionality to concentration

• Comparing with reference methods that use absorbance

• Using algorithms designed for absorbance spectra

Use Reflectance when:

• Raw instrument output is reflectance

• Working with solid samples

• Preserving original measurement scale

Use Kubelka-Munk when:

• Analyzing diffuse reflectance from scattering media

• Working with powders, granules, or tablets

• You need a linearized relationship for thick samples

• Sample thickness is effectively infinite

Working with Multiple Sources

Different data sources can have different signal types:

from nirs4all.data import DatasetConfigs

# Multi-source dataset
dataset = DatasetConfigs({
    "train_x": ["nir_spectra.csv", "vis_spectra.csv"],
    "train_y": "targets.csv"
})

# Check signal types for each source
for src_idx in range(2):
    signal_type, confidence, reason = dataset.detect_signal_type(src=src_idx)
    print(f"Source {src_idx}: {signal_type.value} ({confidence:.0%})")

# Set different signal types per source
dataset.set_signal_type("absorbance", src=0)  # NIR is absorbance
dataset.set_signal_type("reflectance", src=1)  # VIS is reflectance

# Access signal types
signal_types = dataset.signal_types
print(f"All signal types: {[st.value for st in signal_types]}")

Complete Example

Here’s a complete workflow demonstrating signal type detection and conversion:

import nirs4all
import numpy as np
from nirs4all.data.signal_type import detect_signal_type, SignalType
from nirs4all.operators.transforms.signal_conversion import ToAbsorbance
from sklearn.preprocessing import StandardScaler
from sklearn.cross_decomposition import PLSRegression

# Generate synthetic reflectance data
np.random.seed(42)
n_samples, n_wavelengths = 100, 200
R = np.random.uniform(0.3, 0.7, size=(n_samples, n_wavelengths))
y = R[:, 100] + np.random.normal(0, 0.02, n_samples)  # Target related to 1 band

# Detect signal type
signal_type, confidence, reason = detect_signal_type(R)
print(f"Detected: {signal_type.value} ({confidence:.1%})")
print(f"Reason: {reason}")

# Compare pipelines with different signal representations

# Run both
result_r = nirs4all.run(pipeline=[
    StandardScaler(),
    {"model": PLSRegression(n_components=10)}
], dataset=(R, y), name="Reflectance", verbose=0)
result_a = nirs4all.run(pipeline=[
    ToAbsorbance(source_type="reflectance"),
    StandardScaler(),
    {"model": PLSRegression(n_components=10)}
], dataset=(R, y), name="Absorbance", verbose=0)

print(f"\nResults:")
print(f"Reflectance:  RMSE = {result_r.best_rmse:.4f}")
print(f"Absorbance:   RMSE = {result_a.best_rmse:.4f}")

API Reference

SpectroDataset Methods

• dataset.detect_signal_type(src=0, force_redetect=False) → Tuple[SignalType, float, str]

  Detect signal type using heuristics. Returns (SignalType, confidence, reason_string).

• dataset.set_signal_type(signal_type, src=0, forced=True) → None

  Set the signal type for a data source. Use forced=True to prevent auto-detection from overriding.

• dataset.signal_type(src=0) → SignalType

  Get the current signal type for a source.

• dataset.signal_types → List[SignalType]

  Get signal types for all sources.

Signal Converters

All converters follow the sklearn TransformerMixin pattern with fit(), transform(), and inverse_transform() methods:

• ToAbsorbance(source_type="reflectance", epsilon=1e-10, clip_negative=True)

  Convert reflectance/transmittance to absorbance: A = -log₁₀(X)

• FromAbsorbance(target_type="reflectance")

  Convert absorbance to reflectance/transmittance: X = 10^(-A)

• KubelkaMunk(source_type="reflectance", epsilon=1e-10)

  Apply Kubelka-Munk transformation: F(R) = (1-R)² / (2R)

• PercentToFraction()

  Convert percentage (0-100) to fraction (0-1)

• FractionToPercent()

  Convert fraction (0-1) to percentage (0-100)

• SignalTypeConverter(source_type, target_type)

  General-purpose converter that automatically determines the conversion path

Utility Functions

• detect_signal_type(spectra, wavelengths=None, wavelength_unit="nm") → Tuple[SignalType, float, str]

  Standalone function to detect signal type from numpy array

• normalize_signal_type(signal_type) → SignalType

  Convert string or enum to SignalType enum

See Also

• Loading Data - Data loading fundamentals

• Sample Repetition & Aggregation - Sample repetition and prediction aggregation

• Preprocessing Handbook - Signal-specific preprocessing strategies

• Getting Started - Quick start guide

See also

Related Examples:

• U04: Signal Conversion - Signal type detection and conversion operators

• U01: Flexible Inputs - Data loading with signal type handling