Silver Layer: Building Trusted Analytical Datasets

The Transformation Workflow: Silver jobs consume Bronze incrementally, often through streaming or micro batch processing. They apply a specific set of transformations: schema validation, deduplication, type casting, referential integrity checks, and naming standardization. The goal is to create clean, conformed datasets that analysts and data scientists can trust without needing to understand source system quirks. Consider a common scenario: three different source systems store customer information, each with its own schema. System A uses customer_id (integer), System B uses custId (string), and System C uses user_id (UUID). Silver transformations map all three to a standardized customer_uuid column with consistent type and format. Performance and Freshness: Databricks customers commonly achieve end to end Silver latency under 15 minutes for streaming refreshed tables at p99. This means that 99% of events that landed in Bronze are cleaned and available in Silver within 15 minutes. For batch oriented Silver tables updated hourly or daily, this latency extends but remains predictable. Ownership and Stability: Central data platform teams typically own most Silver tables because they represent source aligned, business agnostic transformations. This centralization is intentional: it prevents different teams from applying conflicting cleaning rules to the same source data. If Marketing and Finance both need customer data, they both consume the same Silver customer table rather than creating two different cleaned versions. The risk here is Silver becoming a dumping ground. If teams start encoding business specific logic in Silver ("only include customers who made a purchase in the last 90 days"), then changes break other consumers. Silver should answer "what is the clean version of this source entity" not "what subset does my team need right now." Business specific filters and joins belong in downstream Platinum or Gold layers.