Production Implementation: Multi-Tier PII Detection Pipeline

At scale, a single PII detection strategy cannot meet both latency and coverage requirements. Consider a product telemetry pipeline ingesting 200,000 events per second at peak with p95 latency budget of 250 milliseconds end to end. Events contain structured fields like user IDs and device metadata, plus unstructured text in feedback forms and error messages. The system must support real time dashboards while feeding offline training datasets. The solution is a three tier architecture. Tier one runs schema validation and fixed transforms at ingestion in under 1 millisecond per event. A centrally managed schema contract bans direct identifiers unless explicitly justified. Ingestion filters drop fields like full email, GPS coordinates finer than 3 decimal places, and complete IP addresses, keeping only truncated prefixes aligned to k-anonymity granularity. This deterministic layer provides baseline protection with negligible latency. Tier two applies fast pattern based detectors in the hot path. Maintain regex validators for credit card formats (Luhn algorithm check), Social Security Numbers, national IDs, IPv4 strings, and email patterns. These run in 2 to 5 milliseconds per event on structured fields. When a match is found, the system either suppresses the field, replaces it with a type label like CREDIT_CARD, or routes the event to a quarantine queue for manual review. This tier catches 80% to 90% of structured PII with minimal latency impact. For identifiers required for linkage, like user ID, the hot path calls a token vault. A well designed vault with HSM backed key management and horizontal scaling handles 50,000 tokenizations per second at 3 to 8 millisecond p95 latency per request. Tier three performs deep Named Entity Recognition (NER) on unstructured text asynchronously. Open source frameworks like spaCy report 0.8 seconds per medium sized document on a single CPU core. Cloud services like Google Cloud Data Loss Prevention (DLP) API and AWS Comprehend return entities in 0.5 to 0.7 seconds per document. These latencies are prohibitive for 200,000 events per second, so the system splits flow. The hot path publishes redacted events immediately. A side stream samples high risk fields or events flagged by tier two heuristics, applies NER in parallel batch jobs, then retroactively patches the data warehouse or deletes items that fail policy. Use conservative confidence thresholds: NER recall is imperfect and adversarial formats evade detection, so route low confidence matches to human review. Track detection precision and recall on labeled test sets and retune models quarterly as attack patterns evolve.

Microsoft Azure Event Hubs ingests telemetry at 300,000 events per second, applies schema drops in 0.8ms, pattern matching in 4ms, and samples 5% of text fields for async NER with Google DLP API

Apple's token vault generates deterministic HMAC SHA256 tokens per app bundle with 90 day key rotation, handling 80,000 requests per second at 5ms p95 by sharding across 12 HSM backed nodes

Meta's PII pipeline uses tier two regex to redact credit card numbers in support tickets with 95% precision and 88% recall, routing false positives to a review queue that processes 2,000 items daily