Implementation Patterns: Privacy by Design

The Architecture Components: GDPR compliance at scale requires systematic architectural patterns, not ad hoc solutions. These patterns embed privacy controls into your data platform from the ground up, making compliance automatic rather than manual. Pattern 1: PII Isolation with surrogate keys Use surrogate keys for internal references and keep mapping from real world identifiers (email, phone, government identifier) in small, heavily protected identity store. Downstream data lakes and warehouses operate on surrogate keys like surrogate_id=abc123, not raw email addresses. This sharply limits where direct PII exists, reducing your blast radius for breaches and simplifying deletion. Pattern 2: Data catalog with automated classification Maintain central catalog that knows every dataset, its schema, and classification of each column as PII, quasi identifier, or non sensitive. Integrate automated scanners that detect likely PII patterns using regular expressions and machine learning, quarantine new datasets until reviewed. This becomes source of truth when processing deletion requests or compliance audits. At scale, manual classification is impossible; automation is mandatory. Pattern 3: Consent and purpose service Separate service stores user consents and allowed purposes for each data category. All data production and consumption code queries this service, either inline or via replicated policy caches, to decide which events to drop, tag, or route. For example, recommendation engine checks if user consented to personalization before reading their browsing history. Service must handle 100,000 to 500,000 requests per second with p99 latency under 10 milliseconds to avoid becoming bottleneck. Pattern 4: Field level protection with tokenization Apply tokenization, hashing, or encryption at field level for PII. Tokenization systems provide stable tokens for linkage: each email maps to consistent token like token_8f3a2b, allowing analytics and joins while keeping raw PII in minimal key vault. Challenge is this system must scale with total write traffic, requiring distributed token generation with 100,000 to 500,000 requests per second throughput and sub 10 millisecond p99 latency. Pattern 5: Tiered storage with retention Implement scheduled jobs that enforce retention policies, dropping or aggregating events older than 13 months. Use tiered storage where older data is more heavily aggregated and stripped of PII. For example, raw logs for 30 days, pseudonymized session level data for 1 year, only coarse aggregates beyond that. This automatically reduces compliance scope as data ages. Pattern 6: Access control with audit logging Use role based and attribute based access controls for analytical stores. Log all access to PII or sensitive datasets with user identity, purpose, query text, and context. Retain these audit logs securely to demonstrate compliance. For high risk access patterns (querying millions of user records), require additional approval workflows and automated anomaly detection. Handling 10x Scale Growth: As you grow from 100 terabytes to multiple petabytes, these patterns must scale horizontally. Tokenization service becomes distributed with consistent hashing for token stability. Data catalog shards by dataset namespace. Deletion orchestrator uses partitioned queues to parallelize work. The key insight is that privacy infrastructure must scale like any other critical path system, with capacity planning, load testing, and redundancy built in from day one. Companies that treat compliance as afterthought find themselves unable to meet SLAs when data volume explodes.