What is a Data Catalog System?

The Core Problem: As companies grow beyond a few data stores, a specific problem emerges. You have dozens of databases, hundreds of data pipelines, thousands of tables, and maybe tens of thousands of analysts and engineers. When someone needs customer email data or monthly revenue metrics, they face three questions that become impossible to answer through tribal knowledge or spreadsheets. First, what data actually exists? Searching through wikis or asking on Slack doesn't scale past 50 tables. Second, can you trust it? That "users" table might be deprecated or have known quality issues. Third, what breaks if you change this dataset? Drop a column and you might accidentally break 15 downstream dashboards. What a Catalog Actually Stores: A data catalog stores metadata, not the data itself. Think of it as a card catalog in a library: it tells you about the books but doesn't contain the actual content. For each data asset (tables, views, streams, dashboards, machine learning features), the catalog tracks schema information like column names and types, ownership details showing who's responsible, lineage relationships showing which jobs produce this data and what consumes it, usage statistics like query counts and user access patterns, data quality signals such as freshness and completeness, and access policies defining who can view or modify the asset. Where It Fits: The catalog sits alongside your data platform components (warehouses, lakes, streaming systems), not in front of them. When someone runs a query, it goes directly to the warehouse. The catalog doesn't intercept queries or move data. Instead, it listens to events from these systems, builds a unified view of all metadata, and exposes it through search APIs and a user interface. This architecture means the catalog can be eventually consistent. If a table gets updated, the catalog might learn about it 30 to 120 seconds later. That's acceptable because discovery and documentation don't need real time precision. The trade off is simple: you get automated, always on metadata coverage across every system without adding latency to actual data queries.