Metadata Catalog Failure Modes and Edge Cases

The Subtle Failures: When a metadata system fails, the impact is often subtle and long lasting, making these failures particularly dangerous in production. Interviewers care deeply about this because it reveals whether you understand operational reality beyond the happy path. Failure Mode 1: Stale or Incomplete Metadata The most common failure is stale metadata. If crawlers or ingestion jobs fall behind due to backpressure or bugs, the catalog might show a table as "updated hourly" when the upstream pipeline has been failing for 12 hours. Dashboards then silently use outdated data, leading to wrong business decisions. To mitigate this, robust systems track freshness separately. They record the latest successful job run time as a separate field and surface SLA breaches explicitly in the UI. For example, if a table's SLA is "update every hour" and the last update was 3 hours ago, the catalog marks it with a warning icon. Teams set up alerts when critical tables breach freshness SLAs. Failure Mode 2: Schema Drift Not Captured Schema drift happens when a source system changes a column from integer to string, or starts sending a new nested field, but the catalog does not capture this change promptly. Consumers that rely on cataloged schemas see runtime errors or silent data truncation. The root cause is often eventual consistency in schema discovery. A connector might poll for schema changes every 15 minutes. If a breaking change happens immediately after a poll, consumers have a 15 minute window where they see stale schemas. Robust systems treat the schema registry and catalog as part of the ingestion contract. They enforce fail fast behaviors: if a connector detects a schema incompatibility (for example, a new required field), it stops ingestion and alerts immediately rather than silently dropping data. Some systems use schema evolution tools like Confluent Schema Registry to enforce compatibility rules (backward, forward, full) at write time. Failure Mode 3: Catalog Becomes a Single Point of Failure If the authorization layer depends on catalog metadata to make policy decisions, a catalog outage can block all queries. This is catastrophic: a 5 minute catalog outage means 5 minutes of zero data access across the organization. To avoid this, teams often cache critical metadata in query engines. For example, Spark might cache table schemas and basic access rules locally with a 5 minute time to live (TTL). If the catalog is unreachable, Spark serves from cache with slightly stale data. This degrades gracefully: queries continue but with outdated policies. Catalogs are designed for high availability: 99.9 percent or better, with multi-region redundancy and read replicas that serve slightly stale data if needed. Some implementations use consensus systems like Raft or Paxos to ensure the catalog itself does not lose data during node failures. Failure Mode 4: Lineage Cycles and Incorrect Dependencies In complex directed acyclic graphs (DAGs) with branching and backfills, naive lineage reconstruction from logs can produce cycles or incorrect dependency chains. For example, if a backfill job runs out of order, log based lineage might show table B depending on table A, when actually table A depends on table B. This breaks impact analysis. If you change table A's schema, the catalog might incorrectly report that table B is affected, causing unnecessary alerts and blocking deployments. Systems like Delta Lake help by providing transaction logs that capture operations deterministically with timestamps and version numbers. Lineage engines must handle late arriving events, deduplication, and versioning. Some implementations use checksums or cryptographic hashes of data to verify lineage correctness: if table B claims to depend on table A version 5, the engine verifies that table B's data actually contains hashes matching table A version 5. The Interview Insight: When discussing metadata systems in interviews, mentioning these failure modes and mitigations shows deep operational experience. The key pattern is: measure freshness separately from existence, enforce fail fast on schema changes, cache critical metadata for graceful degradation, and treat lineage as a versioned, verifiable artifact rather than a best effort log.