//The problem
The facility-intelligence platform answered questions in natural language over a 300-table Postgres warehouse. Every question became a Claude call to generate SQL — and the bill scaled linearly with traffic. But the traffic wasn't random: operations staff asked the same questions, phrased a dozen ways, all day long.
"How many open work orders in Block B?" and "count of unresolved tickets, block B" are the same query. Paying a frontier model to re-derive identical SQL thousands of times a day is pure waste — the trick is recognizing "the same" when the words differ.
//What it is
A normal cache keys on an exact string. A semantic cache keys on meaning: embed the question into a vector, and on the next request, look for a stored vector that's close enough to reuse its answer. Close in embedding space ≈ same intent.
The catch: "close enough" is doing a lot of work. A hit must be a guarantee that the cached SQL is identical to what the model would have produced — not merely related. That single constraint shapes every decision below.
A cache hit isn't "close enough." It's a promise the answer is identical. Tune for precision, not hit-rate.
//Architecture
Questions are embedded with a small sentence-transformers model, then matched against a Redis vector index. A hit returns the stored SQL instantly; a miss falls through to Claude and writes the result back. The whole lookup is a few milliseconds.
# semantic cache lookup emb = embed(question) # sentence-transformers hit = redis.vector_search(emb, k=1, radius=0.93) if hit: # near-duplicate → skip the LLM return hit.sql sql = llm.generate(prompt(question, schema)) redis.add(emb, sql, ttl=86400) return sql
Embeddings are cheap and local; the expensive call only happens on a genuine miss. Cached entries carry a TTL so schema or data changes can't serve stale SQL forever.
//Threshold
The similarity radius is the entire ballgame. Too loose and you serve confidently wrong SQL; too tight and the hit-rate collapses. I swept the threshold against a labelled set of paraphrase pairs and optimized for a near-zero false-hit rate first, hit-rate second.
~/cache $ python eval_threshold.py --grid 0.85:0.97 threshold=0.90 hit-rate=0.68 false-hit=2.1% threshold=0.93 hit-rate=0.61 false-hit=0.4% ← chosen threshold=0.96 hit-rate=0.38 false-hit=0.0%
0.93 kept false hits to 0.4% while still catching six in ten questions — the sweet spot where savings are large and trust stays intact.
//Guardrails
Some questions must never be served from cache: anything with a relative time window (today, this week), per-user scoping, or freshly-changed schema. Those are detected before lookup and routed straight to the model. A cache that's wrong once costs more trust than it ever saved in dollars.
//Results
After rollout, the cache absorbed the long tail of repeated questions and the inference bill dropped immediately — with latency on hits effectively free compared to a full generation round-trip.
The lesson generalizes past SQL: when your model keeps answering the same thing, the cheapest token is the one you never generate.