Immutable Artifacts and Data Lineage Graphs

Immutable artifacts and content addressable storage form the foundation of reproducible Machine Learning (ML) systems. Every model, dataset, and feature definition is stored as an immutable blob identified by a cryptographic hash (Secure Hash Algorithm 256 or SHA 256). A manifest file lists all dependencies with their hashes and is itself signed to detect tampering. For example, a training run produces a manifest containing the model binary (sha256:a1b2c3), the dataset snapshot (sha256:d4e5f6), the feature schema (sha256:g7h8i9), the code commit (git:123abc), library versions (requirements.txt hash), and the container image digest. This manifest becomes the artifact of record. Any change to inputs produces a different hash, making drift immediately visible. Data lineage graphs extend this immutability across the entire pipeline. Each node represents a versioned artifact (raw data source, transformation script, feature table, training dataset, model). Edges represent processes that transform one artifact into another, annotated with configuration fingerprints and execution metadata. For instance, raw transactions flow through a fraud signal extractor (config version 2.3) to produce feature table v47, which feeds training job j1829 to produce model m92. When a data source is subject to a General Data Protection Regulation (GDPR) deletion request, the lineage graph identifies every downstream artifact that transitively depends on that data. The system can then trigger retraining with the affected records excluded, or if the impact is minimal, assess whether the model can remain in production under differential privacy guarantees. Reproducibility requires determinism. Pin library versions using lock files (pip freeze, Pipenv.lock, Poetry.lock). Use container images with fixed digests, not mutable tags like latest. Fix random seeds for splitting and initialization. Record hardware fingerprints (Central Processing Unit or CPU architecture, Graphics Processing Unit or GPU type) because some operations are nondeterministic across platforms. Snapshot training data as immutable manifests rather than live database queries that return different rows over time. Where nondeterminism is unavoidable (distributed training with asynchronous updates), define an acceptable tolerance (model outputs differ by less than 0.01 Area Under the Curve or AUC) and maintain calibration datasets to validate behavior equivalence. Meta and Google emphasize lineage for impact analysis. If a bug is found in a feature computation that ran for 90 days, lineage identifies which models need retraining and which predictions may be invalid. Microsoft uses signed manifests in its Responsible AI workflows to ensure review boards inspect the exact artifacts that will be deployed, not a later mutation. Amazon's separation of duties model requires that deployment approvals reference immutable artifact hashes, preventing bait and switch where a reviewed model is swapped before launch.

Training manifest: {"model": "sha256:a1b2c3", "dataset": "sha256:d4e5f6", "features": "sha256:g7h8i9", "code": "git:123abc", "container": "docker@sha256:xyz789", "random_seed": 42, "libs": "sha256:requirements_lock"} signed with private key, any tampering breaks signature verification

Lineage query: MATCH (d:DataSource {id:'transactions_q1'}) to (m:Model) RETURN path shows transactions_q1 → fraud_signal_v2.3 → features_v47 → training_j1829 → model_m92, enabling targeted retraining when source is affected by deletion or corruption

Reproducibility test: Rerun training job with same manifest on different hardware (Tesla V100 vs A100 GPUs), verify model outputs differ by less than 0.5% on 10K held out examples, accept as equivalent despite floating point variance