Repetitions: several measurements are still one physical sample

You scan each of 30 materials four times. The file has 120 rows, but the experiment has 30 independent materials. A random row split could put one material’s scans in both training and validation, producing misleadingly easy predictions.

If you only need safe splitting and one reported prediction per material, start with dataset repetition: Sample_ID and Sample Repetition & Aggregation. The nodes here instead reshape repeated measurements into input blocks.

Choose where repetitions belong

Repetition reshaping changes the row meaning from scan to physical material. Thirty materials remain thirty independent materials.

Node	Output for 30 materials × four scans × 31 features	Choose when
rep_to_sources	Four sources, each 30 × 31	Each repetition needs its own source path
rep_to_pp	One source, 30 × 4 × 31	A view-aware model should retain repeated scans
rep_fusion	An explicit relation-aware representation	Sources have unequal counts or missing observations

Worked example: one feature block per repetition

The Python example creates 24 training materials and six held-out materials, each with four scans. Every material’s scans stay in the same partition. Reshape before the splitter because reshaping changes row identities.

Note

These tabs express the same Python SDK workflow. JSON and YAML are recipe files; Python can also use sklearn/nirs4all objects. R, Octave and WASM native pipeline facades do not execute this host-controller node directly. For a recipe that runs in those languages, use 10. Run the same recipe in your language.

JSON

{
  "pipeline": [
    {
      "rep_to_sources": {
        "column": "Sample_ID",
        "expected_reps": 4,
        "on_unequal": "error",
        "source_names": "rep_{i}"
      }
    },
    {
      "merge": {
        "sources": "concat"
      }
    },
    {
      "split": {
        "class": "sklearn.model_selection.KFold",
        "params": {
          "n_splits": 3,
          "shuffle": true,
          "random_state": 17
        }
      }
    },
    {
      "model": {
        "class": "sklearn.cross_decomposition.PLSRegression",
        "params": {
          "n_components": 3
        }
      }
    }
  ]
}

YAML

pipeline:
- rep_to_sources:
    column: Sample_ID
    expected_reps: 4
    on_unequal: error
    source_names: rep_{i}
- merge:
    sources: concat
- split:
    class: sklearn.model_selection.KFold
    params:
      n_splits: 3
      shuffle: true
      random_state: 17
- model:
    class: sklearn.cross_decomposition.PLSRegression
    params:
      n_components: 3

Python

import tempfile
import numpy as np
import nirs4all
from nirs4all.data.dataset import SpectroDataset
from sklearn.model_selection import KFold
from sklearn.cross_decomposition import PLSRegression

rng = np.random.default_rng(17)
base = rng.normal(size=(30, 31))
X = np.repeat(base, 4, axis=0) + rng.normal(scale=0.05, size=(120, 31))
y = np.repeat(2 * base[:, 10] - base[:, 20], 4)
ids = np.repeat(np.arange(30), 4)
dataset = SpectroDataset("four_scans_per_material")
dataset.add_samples(X[:96], {"partition": "train"})
dataset.add_samples(X[96:], {"partition": "test"})
dataset.add_metadata(ids.reshape(-1, 1), headers=["Sample_ID"])
dataset.add_targets(y.reshape(-1, 1))

pipeline = [
    {"rep_to_sources": {
        "column": "Sample_ID", "expected_reps": 4,
        "on_unequal": "error", "source_names": "rep_{i}",
    }},
    {"merge": {"sources": "concat"}},
    KFold(n_splits=3, shuffle=True, random_state=17),
    {"model": PLSRegression(n_components=3)},
]
with tempfile.TemporaryDirectory() as workspace:
    result = nirs4all.run(pipeline, dataset, engine="legacy",
                         workspace_path=workspace, save_charts=False, verbose=0)
    print("CV validation score:", result.cv_best_score)
    print("Selected model test RMSE:", result.best_rmse)
    result.close()

What to expect: 120 scan rows become 30 material rows. Four 31-column sources are joined into 124 columns. CV now splits the 24 training materials, not the 96 scans. The six held-out materials receive six predictions. rep_0 means the first stored scan of each material; it does not mean instrument 0.

The historical rep_to_sources and rep_to_pp controllers skip reshaping during prediction. New input must already have the trained source/view representation. Do not submit a fresh 120-row scan file and assume inference will group it.

Keep scans on a view axis instead

This fragment needs the same repetition-aware dataset. It creates four named views per material. A matrix learner flattens four equal-width views to 124 columns; a tensor model can preserve their separate axis.

JSON

{
  "pipeline": [
    {
      "rep_to_pp": {
        "column": "Sample_ID",
        "expected_reps": 4,
        "on_unequal": "error",
        "pp_names": "{pp}_scan{i}"
      }
    },
    {
      "split": {
        "class": "sklearn.model_selection.KFold",
        "params": {
          "n_splits": 3,
          "shuffle": true,
          "random_state": 17
        }
      }
    },
    {
      "model": {
        "class": "sklearn.cross_decomposition.PLSRegression",
        "params": {
          "n_components": 3
        }
      }
    }
  ]
}

YAML

pipeline:
- rep_to_pp:
    column: Sample_ID
    expected_reps: 4
    on_unequal: error
    pp_names: '{pp}_scan{i}'
- split:
    class: sklearn.model_selection.KFold
    params:
      n_splits: 3
      shuffle: true
      random_state: 17
- model:
    class: sklearn.cross_decomposition.PLSRegression
    params:
      n_components: 3

Python

from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import KFold

pipeline = [{'rep_to_pp': {'column': 'Sample_ID',
                'expected_reps': 4,
                'on_unequal': 'error',
                'pp_names': '{pp}_scan{i}'}},
 {'split': KFold(n_splits=3, shuffle=True, random_state=17)},
 {'model': PLSRegression(n_components=3)}]

Expected result: one source, 30 material rows, four views, 31 features per view. This is a structural transformation, not an average over scans.

Enumerate repetition options

Field	Choices / default	Meaning
column	Metadata name; omitted uses dataset aggregation column	Which scans describe the same material
expected_reps	Positive integer; omitted infers count	Validate your acquisition protocol
on_unequal	error (default), pad, drop, truncate	Policy for unequal counts
source_names	Template rep_{i} or explicit list	Labels for rep_to_sources
pp_names	Template with {i} and/or {pp}	Labels for rep_to_pp; lists unsupported
preserve_order	True by default	Stored row order determines repetition slots
aggregate_metadata	first, validate, drop	Handle metadata differences within a material

Unequal counts are a scientific choice

For counts 4, 4 and 3 across three materials:

Policy	What happens	Consequence
error	Stop and report mismatch	You can inspect the acquisition issue
pad	Add missing values to the short group	Model must handle missing input
drop	Remove incomplete groups	Population changes
truncate	Keep three scans per group	Two acquired scans are discarded

Start with error. Do not silently pad or drop observations just to make array shapes agree. Grouping by equal target values (column: y) exists historically, but different materials can share a concentration and new targets are unknown. Use a physical-sample identifier.

When rep_fusion is appropriate

A NIR source may have four scans while a chemistry source has one assay. An ordinary scan-row matrix cannot express that relationship safely. rep_fusion materializes a representation from raw relation-aware multisource data. It requires RawMultiSourceDataset staging and a RepresentationPlan; the ordinary SpectroDataset used above is deliberately rejected.

A representation selection fragment is:

JSON

{
  "rep_fusion": {
    "representation": "per_source_aggregate"
  }
}

YAML

rep_fusion:
  representation: per_source_aggregate

Python

fusion_step = {'rep_fusion': {'representation': 'per_source_aggregate'}}

Expected result: the staged source observations are aggregated per physical sample with lineage retained. masked_stack preserves slots and presence masks; other supported representations and source policies are explained in Heterogeneous Source Repetitions. Run this materialization before branching, and retain its plan for prediction. Do not combine it with another repetition reshaping mechanism in the same recipe.

Implementation-backed examples: repetition tests, relation-fusion tests.