Object Storage & Blob Storage • Multipart Uploads & Resumable TransfersHard⏱️ ~2 min
Failure Modes: Orphaned Uploads, Part Limits, and Integrity Drift
Production multipart and resumable upload systems must defend against several critical failure modes that can silently accumulate costs, violate correctness, or cause user visible failures.
Orphaned or incomplete uploads are a major cost sink. When a multipart upload session is initiated but never finalized or aborted, the uploaded parts remain in storage indefinitely and accrue charges. A single 10 GB incomplete upload with 100 parts at $0.023 per GB-month costs $0.23 monthly until cleaned up. At scale, thousands of orphaned sessions can represent significant waste. Mitigation requires lifecycle policies that automatically abort incomplete uploads after a configured number of days (commonly 7 to 14 days), client watchdogs that abort sessions on application crashes, and operational dashboards to monitor orphan counts.
Exceeding part limits is a common developer mistake. Uploading a 1 TB object with 64 MB parts requires 16,384 parts, exceeding Amazon S3's 10,000 part maximum. The upload will fail at finalization with a cryptic error. The fix is to ensure part size is at least ceiling(total size ÷ 10,000); for 1 TB, use 100 MB or larger parts. Dynamic chunk sizing must recalculate as the object grows to avoid hitting limits late in the upload.
Data integrity drift can occur if per-part checksums are not validated. Silent bit flips or corrupted transmission can result in a successfully finalized object that contains incorrect data. Clients must verify per-part checksums or ETags returned by the server and retry mismatches. At finalization, compute and verify an overall object checksum where supported (S3 supports MD5 or SHA256 checksums for complete objects). Another edge case is resume offset ambiguity: if the server reports "last committed byte N" but the client misinterprets this as starting at byte N instead of N+1, overlapping or missing bytes corrupt the final object. Always trust the server reported committed range and resume at the next byte after the last confirmed.
💡 Key Takeaways
•Orphaned uploads: Incomplete multipart sessions persist indefinitely and accrue storage charges; 10 GB incomplete upload costs $0.23 per month at $0.023 per GB-month
•Part limit violation: 1 TB object with 64 MB parts = 16,384 parts exceeds S3 10,000 max; must use >= 100 MB parts; validate part size = ceiling(total size ÷ max parts)
•Data integrity: Verify per-part checksums or ETags on each upload; compute and validate overall object checksum at finalization to detect silent corruption
•Resume offset ambiguity: Server reports "last committed: byte 524,288,000"; client must resume at byte 524,288,001, not 524,288,000, to avoid overlap or gaps
•Mitigation: Lifecycle policies to auto-abort incomplete uploads after 7 to 14 days, client watchdogs on crashes, per-part checksum validation, final object integrity check
📌 Examples
Amazon S3 lifecycle rule: Automatically abort incomplete multipart uploads after 7 days to prevent orphaned part accumulation and reduce storage costs
Client retry logic: Upload part 42 of 10 GB object, server returns ETag; client verifies ETag matches local checksum, retries on mismatch, marks part complete only after verification
Google Cloud Storage resumable: After network drop, client queries status, receives "last committed: 524288000", resumes upload starting at byte 524288001 to avoid duplication