API Ingestion at Scale: Production Reality

The difference between a prototype and production API ingestion is handling scale, rate limits, and downstream SLAs across dozens or hundreds of sources. Rate Limits Are Your Hard Constraint: Every API has limits. Salesforce might allow 200 requests per minute per tenant. Shopify enforces bucket based rate limiting with burst allowances. If you scale horizontally by adding more workers without coordination, you just hit limits faster and increase error rates. At 200 requests per minute with 1000 records per page, you need 1000 requests to fetch a million records. That is 5 minutes minimum, assuming perfect efficiency. In practice, add retry overhead and you are looking at hours for large backfills. The Multi Tenant Challenge: Fivetran and Airbyte serve thousands of customers, each with their own API tokens and rate limits. A naive approach where each customer gets dedicated workers fails at scale. Instead, they use a shared scheduler that tracks per connector rate limit state globally. When customer A is nearing their Shopify limit, the scheduler throttles their syncs and allocates capacity to customer B. This centralized coordination is why scaling to 10x more customers does not mean 10x more infrastructure. It means smarter scheduling and backoff algorithms. Two Phase Ingestion Pattern: Consider a real flow from a commerce platform. A product catalog lives in a headless CMS with an API similar to Bloomreach. The data team needs this in both a search index (for customer facing search) and a warehouse (for analytics). Phase one: An Airflow job submits product updates through the ingestion API. The API returns a job identifier. Airflow polls the status endpoint every 10 seconds. Job completes in 30 to 300 seconds for tens of thousands of products. Phase two: Once ingestion succeeds, a separate API call triggers index rebuilding. But indexing can only run once per hour to protect search cluster SLAs and avoid overwhelming background workers. This decoupling means multiple teams can ingest concurrently without impacting customer facing search performance. Observability Essentials: Track lag between source updates and destination visibility. Measure 95 percent of updates applied within 5 minutes for near real time pipelines. Log HTTP status code distributions per endpoint. Monitor payload size distributions to catch schema inflation. Structured logging with request identifiers and job identifiers is critical for debugging partial failures in production.