Implementation Patterns: From Prototyping to Production Governance

Operationalizing interpretability requires treating the explainer as a versioned, monitored service with clear interfaces and governance hooks. The explainer consumes the exact same feature vectors as the model to avoid divergence. It never re-extracts features. This tight coupling is enforced through a shared feature service or feature store that serves both model and explainer from a single serialized feature vector, typically in a columnar format like Arrow or Parquet for efficient batch processing. For SHAP, select a background dataset that captures the operational distribution. A standard pattern is to sample 500 to 2,000 rows stratified by key segments (geography, product tier, risk band). Recompute this monthly or when data drift exceeds a threshold (5 percent population stability index shift). For tree ensembles with 100 trees of depth 6 and 50 to 100 features, expect 2 to 5 milliseconds per instance on a modern CPU core. If throughput needs are 10,000 requests per second and explanations are required for 10 percent of traffic (1,000 rps), provision 20 to 50 vCPU for the explainer tier plus 30 percent headroom for spikes. For deep models, prefer gradient based attribution to reduce cost. On a T4 class GPU, a single explanation completes in 10 to 30 milliseconds depending on input size and model depth. Governance patterns include storing per decision explanations with model version, feature manifest, data timestamp, background sample identifier, and explainer configuration. Add quality checks: stability metrics across runs (top 5 features should not change more than 20 percent on repeated calls with same input), sign consistency checks against domain constraints (higher income should not contribute negatively to credit score), and coverage checks for protected attributes (ensure explanations do not surface prohibited proxies in user facing contexts). For user facing text, map raw attributions into human readable reasons using templates approved by legal and UX teams (income below threshold contributed negatively becomes your income of $45K is below the preferred range). Reliability includes caching frequently requested explanations at the edge for 24 hours, rate limiting user triggered explanation requests (10 per user per hour to prevent abuse), and circuit breakers that fall back to precomputed top K reasons if the explainer exceeds p95 latency budget. Monitor attribution drift by tracking the distribution of top features over rolling 7 day windows. A shift from payment history appearing in 40 percent of explanations to 60 percent may signal data drift, feature pipeline bugs, or model degradation. Finally, evaluate alternatives where interpretability requirements are strict: enforce monotonicity constraints (higher income always improves score) or use generalized additive models, which provide inherent interpretability and reduce reliance on post hoc explanations.

A credit platform uses a shared feature store (Tecton) that serializes features as Arrow and serves both XGBoost model and SHAP explainer, eliminating a 3 month debugging effort caused by training serving skew.

Microsoft Azure ML explanation store ties each attribution vector to a model checkpoint SHA, feature manifest version, and background sample timestamp, enabling audit queries that reproduce exact explanations from 2 years ago.

Google Cloud Vertex AI automatically refreshes SHAP background samples when weekly batch analysis detects population stability index drift exceeding 5 percent, triggering recomputation of baseline predictions.

A fraud detection system monitors attribution drift and detects that transaction amount shifted from 30 to 55 percent of top attributions over 7 days, revealing a feature pipeline bug that duplicated the amount field, triggering rollback.