Organizational Patterns: Centralized vs Decentralized Lakes

The Organizational Challenge: Technical architecture is only half the story. At scale, the harder problem is organizational: who owns what, who can access what, and how teams coordinate without blocking each other. Data lake architecture patterns must solve social problems, not just technical ones. There are two fundamental approaches, each solving different problems at different company scales. Centralized Lake Pattern: One central team owns the lake infrastructure, storage accounts, and governance. All data flows through central pipelines into a shared namespace. The central team maintains the catalog, enforces data quality standards, and manages access controls. This works beautifully at 100 to 500 TB scale with a few dozen teams. You get consistent tooling, unified governance, and clear ownership. Cost is controlled because the central team sees all usage. Data discovery is easy because everything is in one place. The failure mode appears around 10x scale or 100 plus teams. The central team becomes a bottleneck. Domain teams wait weeks for schema approvals or new dataset onboarding. Innovation slows because every change requires central coordination. Teams start building shadow data infrastructure to move faster, undermining the whole model. Decentralized Lake Pattern (Data Mesh): Each domain or business unit owns its own lake zone, sometimes even separate storage accounts or cloud projects. Domains control their own ingestion pipelines, schemas, and Service Level Agreements (SLAs). A central platform team provides shared tooling, templates, and global discovery services, but domains make their own decisions. This scales organizationally. A payments team can iterate on fraud models without waiting for approval from a central analytics team. Ownership is clear: if payment data quality degrades, the payments domain is accountable. The trade offs are real. Governance becomes distributed, making it harder to enforce consistent data quality or prevent duplicate work. Cost control is weaker because no single team sees the full picture. You might discover that three domains are independently ingesting the same third party dataset, tripling cost. Cross domain queries become harder because data contracts and schemas vary. Hybrid Reality: Most large companies end up with a hybrid model. A central platform team owns core infrastructure (storage accounts, networking, identity management) and provides self service tools. Domains own their data products and publish them to a global catalog with metadata and Service Level Objectives (SLOs). Central governance sets baseline policies (like data retention rules and privacy controls) but domains implement them. AWS typically recommends a central data lake account for shared datasets, with each business unit having its own account for domain specific data. Cross account access is managed through Lake Formation or similar resource sharing. Databricks promotes a lakehouse model with Unity Catalog providing centralized governance while domains maintain autonomy over their tables.