Bronze Layer: The Raw Data Foundation

The Ingestion Challenge: At scale, your Bronze layer handles tens of thousands of events per second from hundreds of heterogeneous sources. A single microservice might emit clickstream events, a database might send Change Data Capture (CDC) records, and SaaS tools push API dumps. Each has different formats, schemas, and reliability characteristics. Bronze must ingest everything without becoming a bottleneck. The design philosophy is simple: capture first, ask questions later. Bronze prioritizes throughput and durability over schema validation or usability. If a source sends malformed JSON, you store it anyway with metadata flagging the issue. If records arrive out of order or duplicated, you keep all copies. This seems wasteful, but it solves a critical problem: once you drop or transform data at ingestion, you can never recover it. Storage Strategy: Bronze typically uses append only, partitioned storage in a data lake or lakehouse format like Delta Lake or Apache Iceberg. Partitions are usually by ingestion date or event timestamp, sometimes with a secondary partition by source system. For example, /bronze/clickstream/date=2024-01-15/hour=14/ contains all clickstream events ingested during that hour. Each record includes critical metadata: source_system_id, ingestion_timestamp, schema_version, and sometimes a validation_status flag. This metadata becomes invaluable when debugging data quality issues months later. You can query: "Show me all records from source X that failed validation between midnight and 1am on January 15th." Why This Matters: Regulatory and audit requirements often demand complete historical records. If your financial reporting has an error, auditors may require proof of what the source system actually said six months ago. Without Bronze, you have no ground truth. Additionally, when you discover a bug in Silver or Gold transformations, you can replay from Bronze with corrected logic rather than asking source systems to resend historical data (which they often cannot do). The Storage Cost Trade Off: At petabyte scale, storing everything raw is expensive. If you ingest 100 TB daily and retain Bronze for 2 years, that's 73 PB of storage. At cloud storage rates of roughly $20 per TB per month, you're paying $1.5 million annually just for Bronze. This is why the medallion model requires organizational commitment: you trade cost for governance, lineage, and replay capability. Cheaper alternatives like keeping only Silver exist, but they sacrifice these benefits.