Search, Lineage, and Graph Queries at Scale

The Search Problem: When an analyst types "monthly active users" into a catalog search box, they expect results in under 300 ms. Behind the scenes, the system queries an index that might contain billions of metadata documents covering millions of tables, columns, dashboards, and metrics. The search index is built from denormalized metadata. For a table entity, the index includes not just the table name and description, but also column names, owner usernames, tag keywords, and frequently co occurring search terms. Indexing pipelines compute derived signals: popularity score based on query counts over the last 30 days, freshness indicator showing when data was last updated, trust level from certification status and data quality checks. Ranking matters critically. A search for "users" might match 5,000 tables. Without good ranking, analysts see noise and revert to asking colleagues. Production catalogs use signals like how many people have queried this table in the last week, how recently the data was updated, whether it's certified by a data platform team, and how often this result gets clicked for this search term. This transforms raw matches into a ranked list where the top 10 results are genuinely useful. The Lineage Graph Challenge: Lineage queries answer questions like "if I change this table, what breaks downstream?" or "where does this dashboard's data originally come from?" This requires traversing a directed graph where nodes are data assets and edges represent data flow. At scale, naive graph traversal becomes a performance problem. A single highly connected table might feed 500 downstream jobs. Each of those might feed 10 more tables. A recursive query can explode exponentially. At companies like LinkedIn with hundreds of millions of edges, an unbounded traversal could take minutes. Production systems apply several techniques. First, depth limits: show downstream dependencies up to 3 or 4 hops, then stop. Second, breadth limits: if a node has over 100 outgoing edges, sample the most important ones based on usage. Third, caching: heavily queried paths (like lineage for production dashboards) are precomputed and cached. Fourth, async expansion: show immediate dependencies in under 200 ms, then let users click to load deeper levels. With these optimizations, a catalog can keep lineage queries under 500 ms p99 even at LinkedIn scale, while providing clear UI warnings when results are partial. Policy and Governance: The policy engine decides what metadata each user can see and what actions they can perform. When someone searches for tables, the catalog filters results based on their permissions. When they try to mark a table as certified, the system checks if they're in the data platform owner group. Many catalogs don't directly enforce access to the underlying data. The warehouse or lake handles that. Instead, the catalog controls catalog visibility: whether you can see that a sensitive table exists, read its schema and lineage, or modify its metadata. This separation lets you adopt a catalog without refactoring your entire security model.