TabPFN for NIRS — fixed recipe (no HPO)

TL;DR

from nirs4all.operators.models import TabPFNNIRSRegressor

est = TabPFNNIRSRegressor(n_estimators=16)
est.fit(X_train, y_train)
y_pred = est.predict(X_test)

One TabPFN training per dataset. No hyperparameter tuning, no preprocessing CV, no stacking. The preprocessing pipeline is fixed and was selected empirically on a 57-dataset NIRS regression cohort.

On that cohort the median test-set RMSE lands at ~+2.5 % relative to the TabPFN paper’s 72-chain per-dataset HPO baseline TabPFN_opt — making this the simplest recipe that gets close to the paper without the HPO cost.

What does it do?

TabPFNNIRSRegressor chains the following steps inside fit:

Step

Operator

Why

1

SavitzkyGolay(window_length=11, polyorder=2, deriv=1)

Smoothing + first derivative — the most universally robust SG kernel in the TabPFN-paper winners

2

OSC(n_components=1)

Removes one orthogonal-to-y signal direction. Net win on the cohort (helps more than it hurts)

3

linspace subsample to max_features

TabPFN v3 has a 2000-feature hard limit. Uniformly-spaced column indices keep spectral structure

4

StandardScaler(with_mean=True, with_std=False)

Center, do not scale (TabPFN is scale-aware)

5

TabPFNRegressor(n_estimators=16, ignore_pretraining_limits=True)

TabPFN ensemble. n_estimators=16 is the sweet spot from a sweep on the cohort

At predict time the same pipeline is applied to X_test (re-using the fitted SG / OSC / scaler / subsample indices).

When to use this vs the alternatives

Recipe

When

Cost per dataset

TabPFNNIRSRegressor (this class)

Default choice. No HPO available, want to match TabPFN_opt cheaply

1× TabPFN

TabPFN_opt (paper, 72-chain HPO)

Need absolute best test RMSE, willing to pay the HPO cost

~72× TabPFN

Mini-HPO over 5 chains via TabPFN-OOF

More accurate than fixed recipe on some datasets, but introduces new catastrophes on SPXY / Kennard-Stone splits (OOF picks the wrong chain when train/test distributions diverge)

~26× TabPFN

AOMPLSRegressor / POPPLSRegressor

Pure linear baselines, no GPU needed, interpretable

~1× PLS

nicon / decon CNN

Lots of data (n > 1000), heavy non-linearity

Long training

Hyperparameter knobs

Parameter

Default

What it does

n_estimators

16

TabPFN ensemble size at fit time. 4 is faster but noisier; 16 is the empirical sweet spot

max_features

2000

TabPFN v3 cap. Wider spectra get uniformly subsampled

sg_window_length

11

SG smoothing window

sg_polyorder

2

SG polynomial order

sg_deriv

1

SG derivative order

osc_n_components

1

Number of OSC directions to remove

random_state

0

Seed for TabPFN

device

"auto"

TabPFN device. "auto" / "cpu" / "cuda"

model_path

"auto"

TabPFN checkpoint. "auto" loads the default v3

After .fit, three book-keeping attributes are populated:

  • n_features_in_ — number of input features seen at fit time

  • n_features_used_ — number of features actually fed to TabPFN (after the max_features cap)

  • subsample_idx_ — column indices used (or None when no subsample)

Performance on a 57-dataset NIRS cohort

The recipe was validated on the full TabPFN-paper regression cohort (60 datasets minus a few too-large ones for HPO comparisons):

Metric

Value

Median test RMSE Δ vs TabPFN_opt

~+2.5 %

Wins vs TabPFN_opt (lower RMSE)

~21 / 57 datasets

Worst case

+47 % on Brix_spxy70 (n_train = 35, structural limit)

Datasets within ±10 % of TabPFN_opt

majority

The full sweep is in bench/nicon_v2/ (see test_full_50_results.csv and test_minihpo5_untested_results.csv for raw numbers).

Use in a nirs4all.run pipeline

import nirs4all
from sklearn.model_selection import ShuffleSplit
from nirs4all.operators.models import TabPFNNIRSRegressor

result = nirs4all.run(pipeline=[
    ShuffleSplit(n_splits=3, test_size=0.2, random_state=42),
    {"model": TabPFNNIRSRegressor(n_estimators=16)},
], dataset="path/to/data", verbose=1)
print(f"best RMSE: {result.best_rmse:.4f}")

The estimator is sklearn-compatible (get_params, set_params, clone), so it integrates with nirs4all’s cross-validation, hyperparameter search, and result-tracking machinery the same way as any other regressor.

Requirements

  • Install the optional dependency: pip install tabpfn

  • A GPU is strongly recommended for n_train > 200. TabPFN runs on CPU but is much slower (the import is done lazily inside fit, so the module itself imports without the dependency).

Known limitations

  • On a handful of datasets (Brix_spxy70, Corn_Oil_80, Biscuit_Sucrose, TIC_spxy70, DIESEL_hlb-a) the fixed recipe is +20 to +47 % above TabPFN_opt. These are typically n_train < 100 with very specific per-dataset preprocessing requirements that no fixed recipe can match.

  • On SPXY / Kennard-Stone test splits where the test set is designed to be far from the train set, cheap CV-based preprocessing selectors (the alternative to this fixed recipe) frequently mis-rank chains. Sticking with the fixed recipe is safer on those splits.

See also