branch: let the same samples follow different paths
A spectrum may contain useful information after normalization and smoothing.
branch keeps both paths. You decide later whether to compare them, join their
features, or combine their predictions.
A branch is a named list of steps. Each path starts from the incoming data; preprocessing in one path does not overwrite another path.
Worked example: two views, one Ridge model
The merge joins columns: 31 normalized + 31 smoothed = 62 features. Both paths preserve observation identity.
Note
These tabs express the same Python SDK workflow. JSON and YAML are recipe files; Python can also use sklearn/nirs4all objects. R, Octave and WASM native pipeline facades do not execute this host-controller node directly. For a recipe that runs in those languages, use 10. Run the same recipe in your language.
{
"pipeline": [
{
"split": {
"class": "sklearn.model_selection.KFold",
"params": {
"n_splits": 3,
"shuffle": true,
"random_state": 17
}
}
},
{
"branch": {
"normalized": [
{
"class": "nirs4all.operators.transforms.StandardNormalVariate"
}
],
"smoothed": [
{
"class": "nirs4all.operators.transforms.SavitzkyGolay",
"params": {
"window_length": 7,
"polyorder": 2,
"deriv": 0
}
}
]
}
},
{
"merge": "features"
},
{
"model": {
"class": "sklearn.linear_model.Ridge",
"params": {
"alpha": 1.0
}
}
}
]
}
pipeline:
- split:
class: sklearn.model_selection.KFold
params:
n_splits: 3
shuffle: true
random_state: 17
- branch:
normalized:
- class: nirs4all.operators.transforms.StandardNormalVariate
smoothed:
- class: nirs4all.operators.transforms.SavitzkyGolay
params:
window_length: 7
polyorder: 2
deriv: 0
- merge: features
- model:
class: sklearn.linear_model.Ridge
params:
alpha: 1.0
import tempfile
import numpy as np
import nirs4all
from nirs4all.data.dataset import SpectroDataset
from sklearn.model_selection import KFold
from sklearn.linear_model import Ridge
from nirs4all.operators.transforms import SNV, SavitzkyGolay
rng = np.random.default_rng(17)
X = rng.normal(size=(48, 31))
y = 2 * X[:, 10] - X[:, 20] + rng.normal(scale=0.1, size=48)
dataset = SpectroDataset("worked_node")
dataset.add_samples(X[:40], {"partition": "train"},
headers=[str(i) for i in range(31)])
dataset.add_samples(X[40:], {"partition": "test"})
dataset.add_targets(y.reshape(-1, 1))
pipeline = [
KFold(n_splits=3, shuffle=True, random_state=17),
{"branch": {
"normalized": [SNV()],
"smoothed": [SavitzkyGolay(window_length=7, polyorder=2, deriv=0)],
}},
{"merge": "features"},
{"model": Ridge(alpha=1.0)},
]
with tempfile.TemporaryDirectory() as workspace:
result = nirs4all.run(pipeline, dataset, engine="dag-ml",
workspace_path=workspace, save_charts=False, verbose=0)
print("CV validation score:", result.cv_best_score)
print("Selected model test RMSE:", result.best_rmse)
result.close()
What to expect: one fused model recipe. Ridge is fitted once per fold on a
62-column input. The eight test observations receive predictions. SNV and
smoothing both see the incoming spectra; this is not SNV followed by smoothing.
The example selects engine="dag-ml" and requires that graph runtime installed.
See 1. Run your first complete experiment for setup.
Three reasons to branch
Mode |
Input per path |
Purpose |
Join |
|---|---|---|---|
Named/list duplication |
Same observations and sources |
Complementary preprocessing or models |
|
|
Different observation subsets |
Separate sites or flagged cohorts |
|
|
Modalities of the same sample |
Spectra and laboratory markers need different transforms |
|
Without a merge, subsequent steps continue on each active branch. A model immediately after two preprocessing branches fits separate models. With a feature merge before the model, it fits one model on both blocks.
Compare paths instead of fusing them
Omit the merge to apply the same Ridge estimator to both representations:
{
"pipeline": [
{
"split": {
"class": "sklearn.model_selection.KFold",
"params": {
"n_splits": 3,
"shuffle": true,
"random_state": 17
}
}
},
{
"branch": {
"normalized": [
{
"class": "nirs4all.operators.transforms.StandardNormalVariate"
}
],
"smoothed": [
{
"class": "nirs4all.operators.transforms.SavitzkyGolay",
"params": {
"window_length": 7,
"polyorder": 2,
"deriv": 0
}
}
]
}
},
{
"model": {
"class": "sklearn.linear_model.Ridge",
"params": {
"alpha": 1.0
}
}
}
]
}
pipeline:
- split:
class: sklearn.model_selection.KFold
params:
n_splits: 3
shuffle: true
random_state: 17
- branch:
normalized:
- class: nirs4all.operators.transforms.StandardNormalVariate
smoothed:
- class: nirs4all.operators.transforms.SavitzkyGolay
params:
window_length: 7
polyorder: 2
deriv: 0
- model:
class: sklearn.linear_model.Ridge
params:
alpha: 1.0
from nirs4all.operators.transforms import SavitzkyGolay
from nirs4all.operators.transforms import StandardNormalVariate as SNV
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold
pipeline = [{'split': KFold(n_splits=3, shuffle=True, random_state=17)},
{'branch': {'normalized': [SNV()],
'smoothed': [SavitzkyGolay(window_length=7, polyorder=2, deriv=0)]}},
{'model': Ridge(alpha=1.0)}]
Expected result: two branch-specific model results per fold. Predictions remain separate. For a stack, put models inside each path and use the prediction merge explained in merge: decide what to bring back together.
Give each modality its own preprocessing
For a dataset with spectra first and laboratory markers second, default source
names are source_0 and source_1. Use your declared names if you assigned them.
{
"pipeline": [
{
"split": {
"class": "sklearn.model_selection.KFold",
"params": {
"n_splits": 3,
"shuffle": true,
"random_state": 17
}
}
},
{
"branch": {
"by_source": true,
"steps": {
"source_0": [
{
"class": "nirs4all.operators.transforms.StandardNormalVariate"
}
],
"source_1": [
{
"class": "sklearn.preprocessing.StandardScaler"
}
]
}
}
},
{
"merge": {
"sources": "concat"
}
},
{
"model": {
"class": "sklearn.cross_decomposition.PLSRegression",
"params": {
"n_components": 3
}
}
}
]
}
pipeline:
- split:
class: sklearn.model_selection.KFold
params:
n_splits: 3
shuffle: true
random_state: 17
- branch:
by_source: true
steps:
source_0:
- class: nirs4all.operators.transforms.StandardNormalVariate
source_1:
- class: sklearn.preprocessing.StandardScaler
- merge:
sources: concat
- model:
class: sklearn.cross_decomposition.PLSRegression
params:
n_components: 3
from nirs4all.operators.transforms import StandardNormalVariate as SNV
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
pipeline = [{'split': KFold(n_splits=3, shuffle=True, random_state=17)},
{'branch': {'by_source': True,
'steps': {'source_0': [SNV()], 'source_1': [StandardScaler()]}}},
{'merge': {'sources': 'concat'}},
{'model': PLSRegression(n_components=3)}]
Expected result: 31 spectral + three marker columns make a 34-column matrix. Physical-sample IDs must align across sources. This fragment needs that two-source dataset; the D08 multisource example provides complete data construction and prediction replay.
Route observations using metadata
With a site metadata column containing site_a and site_b, each cohort can
receive different preprocessing. The following fragment assumes equal output
widths and uses the Python legacy lane for metadata separation.
{
"pipeline": [
{
"branch": {
"by_metadata": "site",
"steps": {
"site_a": [
{
"class": "nirs4all.operators.transforms.StandardNormalVariate"
}
],
"site_b": [
{
"class": "sklearn.preprocessing.StandardScaler"
}
]
}
}
},
{
"merge": "concat"
},
{
"split": {
"class": "sklearn.model_selection.KFold",
"params": {
"n_splits": 3,
"shuffle": true,
"random_state": 17
}
}
},
{
"model": {
"class": "sklearn.cross_decomposition.PLSRegression",
"params": {
"n_components": 3
}
}
}
]
}
pipeline:
- branch:
by_metadata: site
steps:
site_a:
- class: nirs4all.operators.transforms.StandardNormalVariate
site_b:
- class: sklearn.preprocessing.StandardScaler
- merge: concat
- split:
class: sklearn.model_selection.KFold
params:
n_splits: 3
shuffle: true
random_state: 17
- model:
class: sklearn.cross_decomposition.PLSRegression
params:
n_components: 3
from nirs4all.operators.transforms import StandardNormalVariate as SNV
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
pipeline = [{'branch': {'by_metadata': 'site',
'steps': {'site_a': [SNV()], 'site_b': [StandardScaler()]}}},
{'merge': 'concat'},
{'split': KFold(n_splits=3, shuffle=True, random_state=17)},
{'model': PLSRegression(n_components=3)}]
Expected result: observation rows are reassembled by identity. Site A’s columns are not joined side by side with site B’s columns.
Route using an existing quality tag
A tag records a flag without removing rows. This example flags spectra whose variance is below 0.8 in the synthetic data’s arbitrary units, then gives them a different transform. The threshold is a demonstration, not a universal spectral quality criterion. Use this fragment with the synthetic dataset above and the Python legacy lane.
{
"pipeline": [
{
"tag": {
"class": "nirs4all.operators.filters.SpectralQualityFilter",
"params": {
"min_variance": 0.8,
"tag_name": "low_variance"
}
}
},
{
"branch": {
"by_tag": "low_variance",
"values": {
"ordinary": false,
"flagged": true
},
"steps": {
"ordinary": [
{
"class": "nirs4all.operators.transforms.StandardNormalVariate"
}
],
"flagged": [
{
"class": "sklearn.preprocessing.StandardScaler"
}
]
}
}
},
{
"merge": "concat"
},
{
"split": {
"class": "sklearn.model_selection.KFold",
"params": {
"n_splits": 3,
"shuffle": true,
"random_state": 17
}
}
},
{
"model": {
"class": "sklearn.cross_decomposition.PLSRegression",
"params": {
"n_components": 3
}
}
}
]
}
pipeline:
- tag:
class: nirs4all.operators.filters.SpectralQualityFilter
params:
min_variance: 0.8
tag_name: low_variance
- branch:
by_tag: low_variance
values:
ordinary: false
flagged: true
steps:
ordinary:
- class: nirs4all.operators.transforms.StandardNormalVariate
flagged:
- class: sklearn.preprocessing.StandardScaler
- merge: concat
- split:
class: sklearn.model_selection.KFold
params:
n_splits: 3
shuffle: true
random_state: 17
- model:
class: sklearn.cross_decomposition.PLSRegression
params:
n_components: 3
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.cross_decomposition import PLSRegression
from nirs4all.operators.transforms import SNV
from nirs4all.operators.filters import SpectralQualityFilter
pipeline = [
{"tag": SpectralQualityFilter(min_variance=0.8, tag_name="low_variance")},
{"branch": {
"by_tag": "low_variance",
"values": {"ordinary": False, "flagged": True},
"steps": {"ordinary": [SNV()], "flagged": [StandardScaler()]},
}},
{"merge": "concat"},
KFold(n_splits=3, shuffle=True, random_state=17),
{"model": PLSRegression(n_components=3)},
]
Expected result: each observation enters exactly one path, then row reassembly restores the original 48-row order and 31-column width. Here the tag uses true = failed the quality check, whereas the filter’s keep mask uses true = passed. No rows are deleted. A variance-based decision can be computed from new spectra without knowing their concentration.
Route directly from a filter’s pass/fail result
Use by_filter when you do not need a stored tag. It applies the same criterion
and routes to default branch names passing and failing.
{
"pipeline": [
{
"branch": {
"by_filter": {
"class": "nirs4all.operators.filters.SpectralQualityFilter",
"params": {
"min_variance": 0.8,
"tag_name": "low_variance"
}
},
"steps": {
"passing": [
{
"class": "nirs4all.operators.transforms.StandardNormalVariate"
}
],
"failing": [
{
"class": "sklearn.preprocessing.StandardScaler"
}
]
}
}
},
{
"merge": "concat"
},
{
"split": {
"class": "sklearn.model_selection.KFold",
"params": {
"n_splits": 3,
"shuffle": true,
"random_state": 17
}
}
},
{
"model": {
"class": "sklearn.cross_decomposition.PLSRegression",
"params": {
"n_components": 3
}
}
}
]
}
pipeline:
- branch:
by_filter:
class: nirs4all.operators.filters.SpectralQualityFilter
params:
min_variance: 0.8
tag_name: low_variance
steps:
passing:
- class: nirs4all.operators.transforms.StandardNormalVariate
failing:
- class: sklearn.preprocessing.StandardScaler
- merge: concat
- split:
class: sklearn.model_selection.KFold
params:
n_splits: 3
shuffle: true
random_state: 17
- model:
class: sklearn.cross_decomposition.PLSRegression
params:
n_components: 3
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.cross_decomposition import PLSRegression
from nirs4all.operators.transforms import SNV
from nirs4all.operators.filters import SpectralQualityFilter
pipeline = [
{"branch": {
"by_filter": SpectralQualityFilter(min_variance=0.8),
"steps": {"passing": [SNV()], "failing": [StandardScaler()]},
}},
{"merge": "concat"},
KFold(n_splits=3, shuffle=True, random_state=17),
{"model": PLSRegression(n_components=3)},
]
Expected result: two complementary observation subsets are reassembled into
48 rows, each still with 31 features. This is routing, not exclusion. The fixed
variance filter needs no learned threshold; learned outlier filters have fitted
state and require a deliberate validation/deployment protocol. Optional names
changes the two pass/fail branch labels.
Enumerate all routing forms
Form |
Required input |
Check before deployment |
|---|---|---|
Named mapping |
Lists of branch steps |
Use meaningful report names |
List of lists |
Anonymous branch steps |
Names help when the recipe grows |
|
Metadata column |
New rows need known routable values |
|
Earlier tag: flag a sample and keep it |
The tag must be computable for unlabeled rows |
|
Filter definitions |
Check overlaps and unmatched rows |
|
Multiple declared sources |
Match |
A target-outlier tag can diagnose training data but usually cannot route a new sample whose target is unknown. Tiny separated cohorts may lack sufficient rows for the requested CV folds or PLS components.
Avoid surprises
_or_compares recipes;branchkeeps paths in one workflow.Nested branches multiply complexity. Confirm a simple branch/merge first.
New metadata values require a routing decision; they are not automatically interchangeable with training sites.
Runnable sources: D01 branching basics, D06 separation branches. Full schema: Pipeline Keywords Reference.