Failure Modes and Production Edge Cases

When RDDs Collapse at Scale The most common RDD failure mode is performance collapse as data volume grows 10x. A job that processes 100 GB smoothly suddenly fails when pointed at 1 TB of data. The root cause is object overhead and garbage collection. Each RDD record lives as a full JVM object in memory with headers, pointers, and type metadata. At small scale this overhead is manageable. At 1 billion records with average 200 bytes per object, you need 200 GB of heap just for the data, plus another 50 to 80 GB for object overhead. This triggers major garbage collection cycles. On a cluster with 300 executors, GC pauses of 20 to 45 seconds become frequent. Executors miss heartbeats and get marked dead by the driver. Tasks get retried multiple times. What should be a 2 hour job stretches to 8 hours with frequent failures. Some teams respond by adding more memory, which delays but doesn't solve the underlying problem. Schema Drift in DataFrames DataFrame pipelines face a different beast: schema evolution. Your job reads JSON events from S3 where an upstream team controls the schema. One day they add a new field or change user_age from integer to string. Your DataFrame code that expects an integer suddenly throws an analysis exception. In long running streaming jobs processing 100,000 events per second, this manifests as sudden spikes in null values or failed micro batches. The worst case is silent data corruption: Spark casts the string to null instead of failing, and your aggregations become subtly wrong without obvious errors. Robust production pipelines need schema validation at read time. Check expected columns exist, verify data types match expectations, and fail fast with clear error messages rather than propagating corrupt data downstream. The User Defined Function Trap A subtle failure mode hits when teams embed complex logic in UDFs. Suppose you need to parse and validate email addresses, so you write a Python UDF that applies regex and domain lookups. This UDF gets called on a DataFrame with 500 million rows. The problem: Spark cannot see inside the UDF, so it cannot push filters, prune partitions, or estimate selectivity. If your query filters on event_date after calling the UDF, Spark must still process all 500 million rows through Python serialization before applying the filter. Worse, Python UDFs serialize data between JVM and Python processes for every row. At 100,000 rows per second per executor, this serialization overhead can add 50 to 100 milliseconds per micro batch in streaming jobs, violating latency SLAs. The fix: rewrite UDF logic as native Spark SQL expressions when possible, or at least apply cheap filters before expensive UDFs to reduce data volume. Dataset Encoder Hell Datasets introduce their own complexity: encoders must handle nested structures, nullability, and versioning. If you evolve a case class in Scala by adding a field or changing a type, older serialized data may fail to decode. This shows up in streaming checkpoints. Your job has been running for months, checkpointing state using Dataset encoders. You deploy a code change that adds a field to your domain class. Suddenly the job cannot read old checkpoint data and fails on startup. Recovery requires either rolling back the change or manually migrating checkpoint state, both risky during incidents. Data Skew Kills Performance This affects all three APIs but is particularly insidious. Suppose you join user events with a user dimension table on user_id. If 20 percent of events belong to a few power users, those keys hash to the same partition. You end up with one task processing 2 TB while 299 other tasks finish in 5 minutes. That single straggler task runs for 6 hours, and your job SLA is blown. This shows up as massive variance in stage completion times: median 3 minutes, p99 of 4 hours. The solution involves salting skewed keys or using specialized joins, but requires recognizing the pattern first through metrics on shuffle read sizes per task. Memory Pressure from Broadcast Joins DataFrame optimizer loves broadcast joins for small tables. If the dimension table is under 200 MB, Spark sends a copy to every executor. But if someone misconfigures the broadcast threshold or the table grows unexpectedly to 2 GB, every executor tries to materialize 2 GB in memory. On a cluster with 8 GB executor memory, this triggers immediate OutOfMemory errors across the board, failing the entire job. The mistake is often invisible until the dimension table crosses a size threshold.