Cost Optimization Trade-offs: Caching vs Model Routing

The Decision Framework: Caching and model routing are complementary but solve different problems. Caching eliminates redundant compute for repeated work. Model routing sends different requests to different cost/capability tiers. Understanding when to use each is critical for interview discussions. When Caching Dominates: Use aggressive caching when your workload has natural repetition and correctness is paramount. Enterprise knowledge bases, customer support FAQs, and documentation search all see 30 to 50 percent of queries asking the same small set of questions. A 40 percent cache hit rate with exact matching gives you 40 percent cost reduction with zero quality risk. For regulated domains like healthcare or finance, where approximate answers can cause compliance issues, exact caching is often the only safe optimization. Semantic caching introduces risk of returning contextually wrong answers that could violate regulations. Caching also wins when latency matters more than cost. Serving from cache takes under 5 milliseconds versus 500 to 2000 milliseconds for model inference. For user facing chat where every 100 milliseconds hurts engagement, that 100x to 400x speedup is worth more than marginal cost savings from routing. When Model Routing Dominates: Model routing shines when requests have variable complexity and you can tolerate some quality degradation. Consider a coding assistant: simple syntax questions can go to a smaller, faster model (200 milliseconds, $0.001 per request), while complex algorithm design needs the big model (800 milliseconds, $0.02 per request). A classifier model (often a fine tuned smaller model at 50 milliseconds, $0.0001 per classification) routes each request. If 70 percent of queries are simple, you route them to the cheap model. This saves roughly 65 percent of costs: 0.7 times $0.001 plus 0.3 times $0.02 equals $0.0067 per request on average versus $0.02 for always using the expensive model. The trade-off is quality variance. That cheap model might have 92 percent accuracy versus 97 percent for the expensive model. For simple queries where both are highly accurate, this is fine. But complex queries routed to the cheap model by mistake suffer significant quality drops. Combining Both Strategies: Production systems often use both. A request flow might be: check response cache first (5 milliseconds if hit), on miss, classify query complexity (50 milliseconds), route to appropriate model tier (200 to 800 milliseconds), cache the response for future hits. This stacks savings. If 30 percent hit the cache and 50 percent of the remaining 70 percent route to the cheap model, your effective cost is: 0.30 times $0 plus 0.35 times $0.001 plus 0.35 times $0.02, which equals $0.0074 per request versus $0.02 baseline. That is 63 percent cost reduction. The Hidden Costs: Model routing adds complexity that you must defend with unit economics. The classifier itself costs money and adds latency. If classification takes 80 milliseconds and costs $0.0002 per request, and your routing only saves $0.005 per routed request, you need over 50 percent cheap model routing just to break even on the classifier cost. Aggressive routing also risks quality degradation in ways that are hard to detect. Routing 80 percent to cheap models might preserve average accuracy metrics, but critical edge cases could deteriorate sharply. You need per segment quality monitoring and rollback mechanisms when optimizations hurt specific user cohorts. Semantic caching has similar hidden costs. Vector search for similarity takes 10 to 30 milliseconds and requires maintaining an embedding index. If your baseline hit rate is only 15 percent, the overhead might not justify the savings.