Reverse ETL at Scale: Production Architecture

Multi Tenant Control Plane: At scale, Reverse ETL platforms like Census and Hightouch serve thousands of customers, each with dozens of syncs running at different frequencies. The control plane is a centralized scheduler and orchestrator that manages this complexity. It maintains a job queue where each job represents a sync batch for a specific customer, warehouse model, and destination. For a single large customer, this might mean 50 different syncs: 20 to Salesforce covering different objects, 15 to Marketo for various audience segments, 10 to Zendesk for support context, and 5 to messaging platforms. Some syncs run every 5 minutes for critical real time segments. Others run hourly for bulk updates or nightly for low priority enrichment data. Distributed Worker Architecture: Workers pull jobs from the queue and execute the three phase pipeline. To prevent one slow customer from blocking others, work is partitioned by destination and customer. A misconfigured integration that throttles heavily or a customer warehouse query that takes 10 minutes to run will not degrade performance for other tenants. Workers implement circuit breakers. If a destination API returns 429 rate limit errors on 5 consecutive attempts, the circuit opens and that destination is paused for exponential backoff periods (first 1 minute, then 2, then 4, up to 30 minutes). This prevents wasting resources on failing syncs and gives the destination time to recover. Dead letter queues capture permanently failing records. For example, if 2,500 records are synced but 50 fail validation because email addresses are malformed, those 50 go to a dead letter queue where they can be inspected, fixed in the warehouse, and retried manually. Freshness and Monitoring: Production Reverse ETL tracks several key metrics. First, records processed per second across all syncs gives overall throughput. Second, p50 and p95 freshness lag measures time from when a record is updated in the warehouse to when it appears in the destination. For most operational analytics, p50 of 2 to 5 minutes and p95 under 10 minutes is acceptable. Real time personalization needs sub second latency and requires different architecture like streaming CDPs. Error rates per connector show which destinations are most fragile. If Salesforce syncs fail 0.1% of the time but a custom internal API fails 5% of the time, that signals where to invest in better error handling or API improvements. Per field validation failure counts help identify data quality issues in the warehouse upstream. State Management: The platform maintains state in a distributed database. For each sync, it stores the last successful watermark, the mapping between warehouse IDs and destination IDs, and metadata like last run time and error counts. This state must be consistent and highly available since it drives incremental extraction. If the watermark is lost, the system might resync millions of records unnecessarily or skip changed data entirely.