Centralized vs Federated Metadata: The Trade-Off

The Core Decision: One of the most critical architectural choices in metadata management is whether to centralize all metadata in a single catalog or federate it across domain owned systems. This trade-off appears frequently in system design interviews because it reveals how you think about organizational scalability versus technical consistency. Centralized Catalogs: The Case for Consistency Centralized systems like AWS Glue Data Catalog or Databricks Unity Catalog give you consistency and global enforcement. You get a single search surface where analysts can find any dataset regardless of domain. Governance policies such as "any PII column must be masked for non-privileged users" apply uniformly across batch pipelines, streaming systems, and ML workflows. Lineage views span the entire organization, showing how a change in one team's table affects another team's dashboard. This is crucial for regulated environments. A financial services company with compliance requirements cannot tolerate gaps in lineage or inconsistent PII tagging across 50 teams. The centralized catalog becomes the source of truth for auditors. However, centralization can slow teams down. If onboarding a new data source requires central approval, schema reviews, and complex workflows, teams may wait days or weeks. At organizations with hundreds of data engineers, this becomes a bottleneck. Additionally, the central team must support every storage system, format, and integration, which scales poorly. Federated Catalogs: The Data Mesh Approach Federated or data mesh architectures flip this. Each domain (for example, payments, recommendations, fraud detection) owns its portion of the metadata, possibly with its own tooling and standards. A lightweight central layer aggregates only high-level metadata: dataset names, owners, and descriptions. Detailed schemas, lineage, and policies remain domain owned. This improves local autonomy and speed. The payments team can iterate on their schema evolution strategy without coordinating with central platform teams. They can choose tools that fit their specific needs. The trade-off is inconsistency. One domain might tag PII as sensitive_data, another as pii_flag, and a third might not tag it at all. Cross domain lineage becomes best effort: if the payments domain and recommendations domain use different lineage tools, tracing a flow from one to the other requires manual reconciliation. Discovery suffers too: analysts must search multiple catalogs or rely on tribal knowledge about which domain owns what. When to Choose Each: Choose centralized when you have strong compliance needs (PCI, HIPAA, GDPR), relatively mature data teams that can agree on standards, and a platform team with capacity to support centralization. Organizations under 500 data practitioners often succeed with centralization. Choose federated when you have hundreds of autonomous teams, rapid iteration is critical, and you can tolerate some inconsistency in exchange for speed. Organizations with 1000+ data engineers and strong domain ownership culture (like data mesh practitioners) lean federated. Many organizations end up hybrid: centralized for critical metadata like access control and PII tagging, federated for domain-specific details like business glossaries and custom lineage.