nirs4all.operators.transforms.orthogonalization module

Orthogonal Signal Correction (OSC) and External Parameter Orthogonalization (EPO).

This module provides advanced preprocessing methods for removing unwanted variation from spectral data while preserving information relevant to the target variable.

Classes:

OSC: Orthogonal Signal Correction (Direct OSC / DOSC) EPO: External Parameter Orthogonalization

References

• Wold, S., et al. (1998). Orthogonal signal correction of near-infrared spectra. Chemometrics and Intelligent Laboratory Systems, 44(1-2), 175-185.

• Westerhuis, J.A., et al. (2001). Direct orthogonal signal correction. Chemometrics and Intelligent Laboratory Systems, 56(1), 13-25.

• Roger, J.M., et al. (2003). EPO-PLS external parameter orthogonalisation of PLS application to temperature-independent measurement of sugar content of intact fruits. Chemometrics and Intelligent Laboratory Systems, 66(2), 191-204.

class nirs4all.operators.transforms.orthogonalization.EPO(scale: bool = True, *, copy: bool = True)

Bases: TransformerMixin, BaseEstimator

External Parameter Orthogonalization (EPO).

Removes variation in X that is correlated with external parameters (e.g., temperature, humidity, batch effects, instrumental drift) without using the target variable Y. EPO is useful when you know specific sources of unwanted variation and want to remove them while retaining spectral directions orthogonal to the learned interference.

Scientific Basis:

EPO differs from OSC in that it uses external parameter measurements rather than the target Y. This makes EPO a semi-supervised method that can remove known interference sources. Signal sharing those spectral directions is also removed.

Common applications include: - Temperature-independent sugar content measurement in fruits - Removing batch effects in manufacturing processes - Correcting for instrumental drift over time - Handling moisture effects in soil spectroscopy

Algorithm:

1. Center X and external parameter d

2. Regress X on d to estimate interference loadings in feature space

3. Store the orthogonal complement of those loadings

4. Transform applies learned orthogonalization without needing d

Parameters:

• scale (bool, default=True) – Whether to center X and external parameter before EPO.

• copy (bool, default=True) – Whether to copy input data.

Variables:

• projection (ndarray of shape (n_features, n_features)) – Orthogonal projector onto the complement of the fitted interference directions.

• projection_coefs (ndarray) – Regression coefficients for orthogonalizing each feature against external parameter(s).

• X_mean (ndarray of shape (n_features,)) – Mean of X from training data (if scale=True).

• d_mean (float or ndarray) – Mean of external parameter(s) from training data (if scale=True).

• n_features_in (int) – Number of features seen during fit.

References

[1]

Roger, J.M., Chauchard, F., & Bellon-Maurel, V. (2003). EPO-PLS external parameter orthogonalisation of PLS application to temperature-independent measurement of sugar content of intact fruits. Chemometrics and Intelligent Laboratory Systems, 66(2), 191-204.

Examples

>>> from nirs4all.operators.transforms import EPO
>>> import numpy as np
>>> # Generate synthetic data with temperature effect
>>> X = np.random.randn(100, 50)
>>> temperature = np.random.randn(100)
>>> X = X + np.outer(temperature, np.ones(50)) * 0.5  # Add temperature effect
>>> # Apply EPO to remove temperature effect
>>> epo = EPO()
>>> X_filtered = epo.fit_transform(X, temperature)
>>> # Temperature effect is removed

Notes

• EPO uses external parameter during fit, not Y

• The second argument to fit() is the external parameter (d), not the target (y)

• Variation sharing the interference subspace is removed even when it is Y-related

• For purely Y-orthogonal variation removal, use OSC instead

fit(X, d)

Fit EPO filter using external parameter.

Parameters:

• X (array-like of shape (n_samples, n_features)) – Training spectra or feature matrix.

• d (array-like of shape (n_samples,) or (n_samples, n_params)) – External parameter(s) to orthogonalize against (e.g., temperature, batch ID). This is NOT the target variable Y.

Returns:

self – Fitted transformer instance.

Return type:

EPO

Raises:

• TypeError – If X is a sparse matrix.

• ValueError – If X and d have incompatible shapes.

fit_transform(X, d)

Fit on external parameters and apply the reusable spectral projection.

set_fit_request(*, d: bool | None | str = '$UNCHANGED$') → EPO

Configure whether metadata should be requested to be passed to the fit method.

Note that this method is only relevant when this estimator is used as a sub-estimator within a meta-estimator and metadata routing is enabled with enable_metadata_routing=True (see sklearn.set_config()). Please check the User Guide on how the routing mechanism works.

The options for each parameter are:

• True: metadata is requested, and passed to fit if provided. The request is ignored if metadata is not provided.

• False: metadata is not requested and the meta-estimator will not pass it to fit.

• None: metadata is not requested, and the meta-estimator will raise an error if the user provides it.

• str: metadata should be passed to the meta-estimator with this given alias instead of the original name.

The default (sklearn.utils.metadata_routing.UNCHANGED) retains the existing request. This allows you to change the request for some parameters and not others.

Added in version 1.3.

Parameters:

d (str, True, False, or None, default=sklearn.utils.metadata_routing.UNCHANGED) – Metadata routing for d parameter in fit.

Returns:

self – The updated object.

Return type:

object

transform(X)

Apply EPO filter to data.

Parameters:

X (array-like of shape (n_samples, n_features)) – Spectra or feature matrix to transform.

Returns:

X_filtered – EPO-filtered data with external parameter effects removed.

Return type:

ndarray of shape (n_samples, n_features)

Raises:

• NotFittedError – If transform is called before fit.

• ValueError – If X has incompatible number of features.

Notes

The transform applies the learned orthogonalization without requiring the external parameter. The projection learned during fit is applied to remove the systematic variation pattern identified in the calibration set.

class nirs4all.operators.transforms.orthogonalization.OSC(n_components: int = 1, scale: bool = True, method: str = 'dosc', *, copy: bool = True)

Bases: TransformerMixin, BaseEstimator

Orthogonal Signal Correction (Direct OSC / DOSC).

Removes systematic variation in X that is orthogonal (unrelated) to Y, improving model interpretability and potentially prediction performance. OSC filters out variation unrelated to the target while preserving Y-predictive information.

This implementation uses the Direct OSC (DOSC) algorithm, which is mathematically more rigorous than the original iterative Wold’s OSC. DOSC provides a theoretically exact solution using simple least squares operations.

Scientific Basis:

OSC is particularly useful for NIRS data where systematic variations (temperature, scattering effects, instrumental drift) can obscure chemical information. By removing Y-orthogonal variation, OSC can:

• Reduce model complexity (fewer PLS components needed)

• Improve model interpretability (focus on Y-relevant features)

• Enhance calibration transfer between instruments

• Potentially improve prediction performance (though not guaranteed)

Algorithm:

1. Center and scale X and Y

2. Compute Y-predictive direction: w_pls = X^T @ y / ||X^T @ y||

3. For each orthogonal component: a. Compute PLS direction on residual X b. Orthogonalize to Y-direction using Gram-Schmidt projection c. Compute scores and loadings d. Deflate X by removing orthogonal component

4. Store orthogonal weights and loadings for transform

Parameters:

• n_components (int, default=1) – Number of orthogonal components to remove. Typical values are 1-3. Larger values remove more variation but risk removing Y-relevant information. Use cross-validation to select optimal value.

• scale (bool, default=True) – Whether to center and scale X and Y before OSC. Recommended for spectral data.

• method (str, default='dosc') – OSC algorithm variant. Currently only ‘dosc’ (Direct OSC) is supported. Reserved for future extensions (e.g., ‘wold’, ‘posc’).

• copy (bool, default=True) – Whether to copy input data. If False, input arrays may be modified.

Variables:

• W_ortho (ndarray of shape (n_features, n_components)) – Orthogonal weight vectors. These define the directions in X space that are orthogonal to Y and contain systematic variation to be removed.

• P_ortho (ndarray of shape (n_features, n_components)) – Orthogonal loading vectors. These define how the orthogonal scores relate back to the original features.

• X_mean (ndarray of shape (n_features,)) – Mean of X from training data (if scale=True).

• X_std (ndarray of shape (n_features,)) – Standard deviation of X from training data (if scale=True).

• y_mean (float or ndarray) – Mean of y from training data (if scale=True).

• y_std (float or ndarray) – Standard deviation of y from training data (if scale=True).

• n_features_in (int) – Number of features seen during fit.

• n_components (int) – Actual number of components used (may be less than requested if limited by data dimensions).

References

[1]

Wold, S., Antti, H., Lindgren, F., & Öhman, J. (1998). Orthogonal signal correction of near-infrared spectra. Chemometrics and Intelligent Laboratory Systems, 44(1-2), 175-185.

[2]

Westerhuis, J.A., de Jong, S., & Smilde, A.K. (2001). Direct orthogonal signal correction. Chemometrics and Intelligent Laboratory Systems, 56(1), 13-25.

[3]

Fearn, T. (2000). On orthogonal signal correction. Chemometrics and Intelligent Laboratory Systems, 50(1), 47-52.

Examples

>>> from nirs4all.operators.transforms import OSC
>>> from sklearn.cross_decomposition import PLSRegression
>>> import numpy as np
>>> # Generate synthetic data
>>> X = np.random.randn(100, 50)
>>> y = X[:, :5].mean(axis=1) + np.random.randn(100) * 0.1
>>> # Apply OSC
>>> osc = OSC(n_components=2)
>>> X_filtered = osc.fit_transform(X, y)
>>> # Use in pipeline
>>> from sklearn.pipeline import Pipeline
>>> pipe = Pipeline([
...     ('osc', OSC(n_components=2)),
...     ('pls', PLSRegression(n_components=5))
... ])
>>> pipe.fit(X, y)
Pipeline(steps=[...])

Notes

• OSC is a supervised preprocessing method that requires Y during fit

• OSC does not guarantee improved prediction performance; use cross-validation

• Removing too many components can remove Y-relevant information

• For temperature/batch correction with known external parameters, consider EPO instead

fit(X, y)

Fit OSC filter to training data.

Parameters:

• X (array-like of shape (n_samples, n_features)) – Training spectra or feature matrix.

• y (array-like of shape (n_samples,) or (n_samples, n_targets)) – Target values. OSC uses Y to determine which variation to preserve.

Returns:

self – Fitted transformer instance.

Return type:

OSC

Raises:

• TypeError – If X is a sparse matrix (not supported).

• ValueError – If n_components is invalid or data dimensions are incompatible.

fit_transform(X, y)

Fit OSC and transform X in one step.

Parameters:

• X (array-like of shape (n_samples, n_features)) – Training spectra.

• y (array-like of shape (n_samples,) or (n_samples, n_targets)) – Target values.

Returns:

X_filtered – OSC-filtered training data.

Return type:

ndarray of shape (n_samples, n_features)

transform(X)

Apply OSC filter to data.

Parameters:

X (array-like of shape (n_samples, n_features)) – Spectra or feature matrix to transform.

Returns:

X_filtered – OSC-filtered data with orthogonal components removed.

Return type:

ndarray of shape (n_samples, n_features)

Raises:

• NotFittedError – If transform is called before fit.

• ValueError – If X has incompatible number of features.