merge: decide what to bring back together

After branching, choose what to combine. Features, predictions and separated observations have different meanings. The argument determines the operation.

Choose the result you need

Read the output axis: a merge can widen a matrix, build stacking inputs or restore observation rows.

You have	You want	Write	Example output
Two feature branches, each 48 × 31	One learner using both	merge: features	48 × 62
Two single-target base regressors	A combiner of their predictions	merge: predictions	48 × 2
Two 31-feature branches, each with a regressor	Both features and predictions	merge: all	48 × 64
Different observation cohorts	Original observation order	merge: concat	48 rows, original width
31 spectral + three marker features	Matrix for a classical learner	merge: {sources: concat}	48 × 34
Equal-width sources	A source axis	merge: {sources: stack}	Source-axis representation
Sources for a source-aware model	Named blocks retained	merge: {sources: dict}	Named source blocks

Rows match by observation identity. Do not independently sort sources and then assume that matching row positions represent the same specimens.

Worked example: combine predictions safely

SNV + PLS and standardization + Ridge may make complementary errors. A final Ridge learns to combine their predictions. This is stacking: its features are model outputs, not wavelengths.

Note

These tabs express the same Python SDK workflow. JSON and YAML are recipe files; Python can also use sklearn/nirs4all objects. R, Octave and WASM native pipeline facades do not execute this host-controller node directly. For a recipe that runs in those languages, use 10. Run the same recipe in your language.

JSON

{
  "pipeline": [
    {
      "split": {
        "class": "sklearn.model_selection.KFold",
        "params": {
          "n_splits": 3,
          "shuffle": true,
          "random_state": 17
        }
      }
    },
    {
      "branch": {
        "pls": [
          {
            "class": "nirs4all.operators.transforms.StandardNormalVariate"
          },
          {
            "model": {
              "class": "sklearn.cross_decomposition.PLSRegression",
              "params": {
                "n_components": 3
              }
            }
          }
        ],
        "ridge": [
          {
            "class": "sklearn.preprocessing.StandardScaler"
          },
          {
            "model": {
              "class": "sklearn.linear_model.Ridge",
              "params": {
                "alpha": 1.0
              }
            }
          }
        ]
      }
    },
    {
      "merge": "predictions"
    },
    {
      "model": {
        "class": "sklearn.linear_model.Ridge",
        "params": {
          "alpha": 0.1
        }
      }
    }
  ]
}

YAML

pipeline:
- split:
    class: sklearn.model_selection.KFold
    params:
      n_splits: 3
      shuffle: true
      random_state: 17
- branch:
    pls:
    - class: nirs4all.operators.transforms.StandardNormalVariate
    - model:
        class: sklearn.cross_decomposition.PLSRegression
        params:
          n_components: 3
    ridge:
    - class: sklearn.preprocessing.StandardScaler
    - model:
        class: sklearn.linear_model.Ridge
        params:
          alpha: 1.0
- merge: predictions
- model:
    class: sklearn.linear_model.Ridge
    params:
      alpha: 0.1

Python

import tempfile
import numpy as np
import nirs4all
from nirs4all.data.dataset import SpectroDataset
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.cross_decomposition import PLSRegression
from sklearn.linear_model import Ridge
from nirs4all.operators.transforms import SNV

rng = np.random.default_rng(17)
X = rng.normal(size=(48, 31))
y = 2 * X[:, 10] - X[:, 20] + rng.normal(scale=0.1, size=48)
dataset = SpectroDataset("worked_node")
dataset.add_samples(X[:40], {"partition": "train"},
                    headers=[str(i) for i in range(31)])
dataset.add_samples(X[40:], {"partition": "test"})
dataset.add_targets(y.reshape(-1, 1))


pipeline = [
    KFold(n_splits=3, shuffle=True, random_state=17),
    {"branch": {
        "pls": [SNV(), {"model": PLSRegression(n_components=3)}],
        "ridge": [StandardScaler(), {"model": Ridge(alpha=1.0)}],
    }},
    {"merge": "predictions"},
    {"model": Ridge(alpha=0.1)},
]
with tempfile.TemporaryDirectory() as workspace:
    result = nirs4all.run(pipeline, dataset, engine="dag-ml",
                         workspace_path=workspace, save_charts=False, verbose=0)
    print("CV validation score:", result.cv_best_score)
    print("Selected model test RMSE:", result.best_rmse)
    result.close()

What to expect: each base model provides one prediction column, giving the combiner two columns for the same observation IDs. For training rows, the values come from models that excluded the corresponding row from fitting: out-of-fold (OOF) predictions. New observations pass through the saved base models before the final Ridge predicts.

OOF prevents teaching the combiner with overly optimistic predictions on rows a base model already learned. Choosing the architecture still consumes validation information. Keep an outer holdout for an independent final claim. The synthetic example explains the mechanism without promising a better score.

Combine features instead

These branches contain transforms only. The learner comes after the merge.

JSON

{
  "pipeline": [
    {
      "split": {
        "class": "sklearn.model_selection.KFold",
        "params": {
          "n_splits": 3,
          "shuffle": true,
          "random_state": 17
        }
      }
    },
    {
      "branch": {
        "normalized": [
          {
            "class": "nirs4all.operators.transforms.StandardNormalVariate"
          }
        ],
        "smoothed": [
          {
            "class": "nirs4all.operators.transforms.SavitzkyGolay",
            "params": {
              "window_length": 7,
              "polyorder": 2,
              "deriv": 0
            }
          }
        ]
      }
    },
    {
      "merge": "features"
    },
    {
      "model": {
        "class": "sklearn.linear_model.Ridge",
        "params": {
          "alpha": 1.0
        }
      }
    }
  ]
}

YAML

pipeline:
- split:
    class: sklearn.model_selection.KFold
    params:
      n_splits: 3
      shuffle: true
      random_state: 17
- branch:
    normalized:
    - class: nirs4all.operators.transforms.StandardNormalVariate
    smoothed:
    - class: nirs4all.operators.transforms.SavitzkyGolay
      params:
        window_length: 7
        polyorder: 2
        deriv: 0
- merge: features
- model:
    class: sklearn.linear_model.Ridge
    params:
      alpha: 1.0

Python

from nirs4all.operators.transforms import SavitzkyGolay
from nirs4all.operators.transforms import StandardNormalVariate as SNV
from sklearn.linear_model import Ridge
from sklearn.model_selection import KFold

pipeline = [{'split': KFold(n_splits=3, shuffle=True, random_state=17)},
 {'branch': {'normalized': [SNV()],
             'smoothed': [SavitzkyGolay(window_length=7, polyorder=2, deriv=0)]}},
 {'merge': 'features'},
 {'model': Ridge(alpha=1.0)}]

Expected result: 62 columns: all normalized columns followed by all smoothed columns. One Ridge learns from that matrix. Compare against each single view before concluding that extra columns help.

Select particular blocks

This fragment selects branches 0 and 1 and keeps pre-branch features. For two 31-column branches and 31 originals, the result has 93 columns.

JSON

{
  "merge": {
    "features": {
      "branches": [
        0,
        1
      ]
    },
    "include_original": true,
    "output_as": "features",
    "on_missing": "error",
    "on_shape_mismatch": "error"
  }
}

YAML

merge:
  features:
    branches:
    - 0
    - 1
  include_original: true
  output_as: features
  on_missing: error
  on_shape_mismatch: error

Python

merge_step = {'merge': {'features': {'branches': [0, 1]},
           'include_original': True,
           'output_as': 'features',
           'on_missing': 'error',
           'on_shape_mismatch': 'error'}}

You can specify features and predictions in the same mapping. Retaining original inputs changes model width, scaling balance and required capacity.

Select and name the source fusion

For a two-source dataset whose declared names are source_0 (31 spectral features) and source_1 (three laboratory markers), this fragment chooses those two blocks explicitly. It operates on sources rather than duplication branches.

JSON

{
  "merge": {
    "sources": {
      "strategy": "concat",
      "sources": [
        "source_0",
        "source_1"
      ],
      "on_incompatible": "error",
      "output_name": "spectra_and_markers",
      "preserve_source_info": true
    }
  }
}

YAML

merge:
  sources:
    strategy: concat
    sources:
    - source_0
    - source_1
    on_incompatible: error
    output_name: spectra_and_markers
    preserve_source_info: true

Python

merge_step = {'merge': {'sources': {'strategy': 'concat',
                       'sources': ['source_0', 'source_1'],
                       'on_incompatible': 'error',
                       'output_name': 'spectra_and_markers',
                       'preserve_source_info': True}}}

Expected result: 34 columns for each shared observation, named spectra_and_markers. Source names identify provenance; they are not inferred from the scientific meaning of column values.

Source-merge field	Choices / purpose
strategy	concat, stack, dict
sources	all, source indices, or declared names
on_incompatible	error, flatten, pad, truncate for incompatible stack widths
output_name	Label of merged source
preserve_source_info	Retain input-source provenance metadata

flatten falls back to column concatenation; pad introduces zero-filled positions; truncate discards features to match the shorter width. Prefer error until you have chosen and documented the scientific meaning of a fallback.

Enumerate merge options

Key	Controls	Starting choice
features	All branch features, indices, or {branches: [...]}	all or explicit indices
predictions	Base predictions, indices, per-branch selection	all for a small stack
sources	concat, stack, dict, or source mapping	concat for matrix models
concat	Reassemble separated rows when true	merge: concat shorthand
include_original	Retain inputs from before branching	False unless needed
on_missing	error, warn, skip	error during development
on_shape_mismatch	Incompatible output shapes	error
output_as	Output representation	features here
unsafe	Disable OOF reconstruction	Leave false
source_names	Labels for retained source outputs	Meaningful modality names

merge_sources and merge_predictions are aliases handled by the same controller. Advanced per-branch prediction selection includes best, top_k, metrics and aggregation. See Pipeline Keywords Reference for exact mappings.

Common mistakes

• concat restores separated rows; features joins branch columns.

• stack needs compatible source widths and a consumer that accepts that axis.

• unsafe: true does not repair missing OOF predictions. Fix the fold plan or missing base model outputs before trusting evaluation.

• Silently skipping missing branches changes the meaning of model columns.

Executable fusion and stacking companion: D08. Start with branch: let the same samples follow different paths if merge inputs remain unclear.