Workspace Introduction

The workspace is the durable memory of your experiments. Without it, results remain process-local and fragile. With it, you get run tracking, artifact reuse, chain replay, export workflows, and lifecycle cleanup.

Why Workspace Management Is Central

Imagine running 50 pipeline variants, finding the best one, and then losing all the metadata when your Python process exits. Or trying to remember which preprocessing steps were used for a model you trained last month.

The workspace solves this by:

Tracking all runs with metadata and timestamps
Storing predictions with fold/partition/branch context
Persisting artifacts as content-addressed binaries
Building chains for replay and export
Enabling queries like “show me the top 10 models by RMSE”

Active Workspace Resolution

Workspace path resolution follows a priority order:

Explicitly set path — via workspace= parameter
Environment variable — NIRS4ALL_WORKSPACE
Default local workspace — ./workspace in current directory

Tip

Set NIRS4ALL_WORKSPACE in your environment to use a shared workspace across projects:

export NIRS4ALL_WORKSPACE="/path/to/shared/workspace"

Workspace Layout

A workspace is initialized with this structure:

workspace/
├── store.sqlite          # SQLite database (runs, chains, predictions metadata)
├── arrays/               # Prediction arrays (Parquet sidecar files per dataset)
│   ├── wheat.parquet
│   └── corn.parquet
├── artifacts/            # Content-addressed binary files (hash-sharded)
│   ├── ab/
│   │   └── abc123...joblib
│   ├── cd/
│   │   └── cde456...joblib
│   └── ...
├── exports/              # User-triggered exports (.n4a bundles, etc.)
├── library/              # Reusable pipeline templates
├── figures/              # Chart outputs (optional)
├── outputs/              # Step outputs (optional)
└── logs/                 # Log files (optional)

Key principle: Structured metadata lives in SQLite, dense arrays in Parquet sidecar files, binary artifacts in content-addressed files.

Store-First Architecture

The workspace uses a store-first architecture: structured runtime data lives in SQLite WAL (store.sqlite).

Why SQLite WAL?

Zero-dependency embedded database (part of Python stdlib)
WAL mode enables concurrent readers with a single writer
Robust and battle-tested across billions of deployments
SQL access for custom queries
ACID transactions with crash-safe journaling

The DB is the source of truth for runs and predictions, avoiding scattered manifest files and simplifying query/ranking workflows.

Core Store Tables

The SQLite database contains these core tables:

Table	Purpose
`runs`	High-level experiment sessions (run_id, timestamp, config)
`pipelines`	Per-variant execution metadata (pipeline_id, run_id, dataset_name)
`chains`	Replayable step-to-artifact mappings (chain_id, steps, artifact refs)
`predictions`	Scalar metrics and identifiers (rmse, r2, accuracy, fold_id, partition)
`artifacts`	Binary metadata and content hash references (artifact_id, hash, type)
`logs`	Structured step events (timestamp, level, message)
`projects`	Project grouping for runs

Dense prediction arrays (y_true, y_pred, y_proba, sample_indices, weights) are stored in per-dataset Parquet sidecar files under arrays/, managed by ArrayStore.

Together, these capture full runtime provenance and power analysis and replay.

Artifact Storage Model

Artifacts (fitted models, transformers) are content-addressed by hash, with git-style sharding:

artifacts/{hash[:2]}/{hash}.{ext}

Example:

artifacts/ab/abc123def456...789.joblib
artifacts/cd/cde456fgh789...012.joblib

Benefits:

Natural deduplication — identical binaries share the same hash
Stable lookup — artifact_id → hash → file path
Predictable cleanup — reference counting determines orphaned artifacts

Content Addressing

If two models have identical fitted parameters (rare but possible with small datasets or deterministic seeds), they share the same artifact file. This saves disk space and simplifies caching.

Chain as Deployment Unit

A chain represents:

Ordered preprocessing and model sequence
All needed artifact references (hash + metadata)
Branch/source context
Fold context (for CV chains) or refit context (for deployment chains)

This makes it the key replay unit and the key export unit.

Chain structure:

{
    "chain_id": "abc123",
    "pipeline_id": "def456",
    "fold_id": "fold_0",  # or "refit" for deployment chains
    "steps": [
        {"step_index": 0, "operator": "MinMaxScaler", "artifact_id": "art_001"},
        {"step_index": 1, "operator": "PLSRegression", "artifact_id": "art_002"},
    ],
    "branch_path": None,
    "source_index": 0,
}

Export Paths

NIRS4ALL supports two primary export formats:

.n4a Bundle (ZIP Archive)

Full bundle with artifacts and metadata:

result.export("model.n4a")

Contains:

manifest.json (metadata, pipeline config, dataset info)
artifacts/ (all fitted models and transformers)
chain.json (step sequence with artifact references)

Use case: Production deployment, sharing models with collaborators.

.n4a.py (Portable Python Script)

Self-contained Python script with embedded artifacts:

result.export("model.n4a.py")

Contains:

Python script with predict() function
Base64-encoded artifacts embedded in the script

Use case: Quick sharing, minimal dependencies, executable without NIRS4ALL installation.

Export Philosophy

Training does not eagerly create export files. Export is an explicit operation triggered via result.export(). This keeps workspace clean and avoids wasting disk space on unwanted exports.

Replay Paths

Replay uses chain metadata and artifact lookup to:

Apply transforms in stored sequence
Load fitted artifacts
Execute model prediction logic

Replay supports both refit and fold-based variants:

# Replay from chain in workspace
result = nirs4all.predict(chain_id="abc123", data=X_new)

# Replay from exported bundle
result = nirs4all.predict(model="model.n4a", data=X_new)

Library Templates

The workspace library stores reusable pipeline templates:

workspace/library/
├── templates/
│   ├── baseline_pls.json
│   ├── advanced_cnn.json
│   └── custom_preprocessing.json
└── metadata/
    └── templates.db

Templates include:

Config — full pipeline definition
Metadata — name, description, author, tags, categories
Provenance — source run_id (optional)

Benefits:

Knowledge reuse across projects
Team-level pattern sharing
Standardized workflows

Example

# Save a pipeline as a template
result.save_template(
    name="Baseline PLS",
    description="SNV + SG derivative + PLS regression",
    tags=["baseline", "regression"],
    category="preprocessing",
)

# Load a template
pipeline = nirs4all.load_template("Baseline PLS")

Query and Ranking

The workspace supports operational queries:

# Query top predictions
top_preds = workspace.query_best(metric="rmse", limit=10)

# List runs
runs = workspace.list_runs(dataset="my_dataset")

# List pipelines in a run
pipelines = workspace.list_pipelines(run_id="run_001")

# Filter predictions
preds = workspace.filter_predictions(
    dataset="my_dataset",
    model_class="PLSRegression",
    metric="rmse",
    threshold=0.5,
)

This enables operational reporting without custom SQL.

Cleanup Workflows

Workspace cleanup is essential for long-running experimentation programs:

Deleting Runs

nirs4all workspace delete-run --run-id run_001

Deletes run metadata, pipelines, chains, and predictions. Artifacts are not deleted immediately (reference counting).

Garbage Collecting Artifacts

nirs4all artifacts cleanup

Deletes orphaned artifacts (no longer referenced by any chain).

Database Vacuum

nirs4all workspace vacuum

Reclaims disk space from deleted records.

Cleanup Strategy

Recommended cleanup schedule:

Delete obsolete runs monthly
Garbage collect artifacts weekly
Vacuum database quarterly

This keeps workspace performant and disk usage manageable.

Workspace Lifecycle Diagram

Here’s the full lifecycle from run to cleanup:

        flowchart LR
    A[Initialize Workspace] --> B[Begin Run]
    B --> C[Begin Pipelines]
    C --> D[Store Predictions and Chains]
    D --> E[Complete Run]
    E --> F[Query and Analyze]
    F --> G[Export]
    F --> H[Replay]
    F --> I[Cleanup and GC]

Workspace Governance

As workspace usage grows, governance becomes important:

Versioning

Dataset versions — track dataset hash and version history
Pipeline versions — track pipeline config changes
Chain versions — link chains to specific artifact versions

Provenance

Run metadata — timestamp, user, config, git commit (if available)
Chain provenance — link chains to source runs
Artifact provenance — track which steps generated which artifacts

Access Control

Workspace isolation — separate workspaces for different projects
Read-only access — for auditing and analysis
Write access — for running experiments and cleanup

Advanced Topics

Workspace Merging

Merge multiple workspaces into one:

nirs4all workspace merge --source workspace_a/ --target workspace_b/

Useful for consolidating experiments from different environments.

Workspace Backup

Backup workspace database, arrays, and artifacts:

# Database + arrays only (fast)
cp workspace/store.sqlite workspace_backup/store.sqlite
cp -r workspace/arrays/ workspace_backup/arrays/

# Full backup (slow)
tar -czf workspace_backup.tar.gz workspace/

For production, use incremental artifact backups (copy only new artifacts by modification time).

Custom Queries

Access workspace database directly with SQLite:

import sqlite3

conn = sqlite3.connect("workspace/store.sqlite")
result = conn.execute("""
    SELECT pipeline_name, AVG(rmse) as avg_rmse
    FROM predictions
    WHERE dataset_name = 'my_dataset'
    GROUP BY pipeline_name
    ORDER BY avg_rmse
    LIMIT 10
""").fetchall()

Useful for custom analytics and reporting.

Next Steps

Now that you understand workspace persistence, explore:

Pipeline Workflow — See how execution generates workspace data
Controllers Intro — Learn how controllers emit artifacts