Developer Guide

Documentation for developers who want to understand NIRS4ALL internals or contribute to the project.

Overview

This section is for developers who want to:

  • Understand the internal architecture of NIRS4ALL

  • Extend the library with custom controllers and operators

  • Contribute to the project

🏗️ Architecture

High-level overview of the pipeline execution engine, data flow, and component interactions.

System Design

Architecture Overview
🔄 Pipeline Architecture

Detailed pipeline execution flow and internal mechanics.

Internals

Pipeline Architecture Overview
🎮 Controllers

The controller registry system that dispatches pipeline steps to appropriate handlers.

Extension Point

Controller System
📦 Artifacts & Storage

User guide for the artifact storage system.

Storage

Artifacts & Storage
Caching & Memory

Block-based storage, copy-on-write branches, step caching, and memory observability.

Performance

Caching and Memory Optimization

Additional Developer Topics

🔧 Artifacts Internals

Internal implementation details and extension points for the artifacts system.

Deep Dive

Artifacts Developer Guide
📝 Metadata

Pipeline and dataset metadata handling.

Data

Metadata Usage Guide
📤 Outputs vs Artifacts

Understanding the difference between outputs and artifacts.

Concepts

Outputs vs Artifacts: Serialization Architecture
🧪 Synthetic Data

Synthetic data generator internals and extension.

New in 0.6

Synthetic Data Generator
🧪 Testing Guide

Running tests, markers, fixtures, and writing new tests.

Quality

Testing Guide

Quick Start: Custom Controller

Here’s a minimal example of creating a custom controller:

from nirs4all.controllers import register_controller, OperatorController

@register_controller
class MyController(OperatorController):
    priority = 50  # Lower = higher priority

    @classmethod
    def matches(cls, step, operator, keyword) -> bool:
        return keyword == "my_custom_keyword"

    @classmethod
    def use_multi_source(cls) -> bool:
        return False

    def execute(self, step_info, dataset, context, runtime_context, **kwargs):
        # Your implementation here
        return context, output

See Controller System for the complete guide.

See Also