Home / Case Studies / Semantic Caching Tradeoff

Semantic Caching for LLM Cost & Latency

Matching questions by meaning cuts cost sharply, and starts answering questions nobody asked.

LLMOps Semantic Cache Embeddings FAISS Cost Optimization Latency

TL;DR

The Decision That Triggered This Build

In a high-volume question-answering product, a large share of traffic re-asks something already answered, worded differently. Every one of those is a paid, slow model call producing an answer that already exists.

The product question: how much of that is reclaimable, and what is the failure mode of reclaiming it? An exact cache is safe and weak. A semantic cache is powerful and introduces a risk the exact cache never had.

Why Exact Matching Fails

Users do not repeat themselves verbatim

The same handful of meanings arrives under many spellings, punctuation, capitalisation, "hey", "please". An exact cache pays full price for each, and here it topped out at a 0.691 hit rate.

Meaning-level matching collapses that

Many spellings, one cached answer, lifting the hit rate to 0.724 while cutting model calls by the same fraction.

And it introduces a new way to be wrong

Matching on similarity means sometimes matching something that is merely similar, the false hit the whole project exists to measure.

Technical Approach

Flowchart. An incoming question is embedded and matched against cached question vectors by nearest-neighbour search. If similarity meets the threshold the cached answer returns in about ten milliseconds at no cost; otherwise the model is called at roughly nine hundred milliseconds and billed, and the new question and answer join the cache. A note marks that a cache hit may be a true match or a false one, distinguishable only with human duplicate labels.

Every request is a threshold decision. The threshold is the entire design, at 0.80 it bought a 3.5× speedup for a 1.35% false-hit rate.

Stream, do not load

The dump is one XML document of a few hundred megabytes (293 MB of Posts.xml); parsing it whole exhausts a 16GB machine. iterparse with elem.clear() keeps memory flat regardless of file size. That one line is the difference between a script that runs and one that swaps.

Duplicate links become meaning-groups

Links are pairwise, but A↔B and B↔C means all three are the same question. The groups are the connected components of that graph, union-find, a few lines. 1,945 duplicate pairs resolved into the ground-truth meaning-groups everything else is scored against.

Cosine on an interpretable scale

Vectors (BGE) are L2-normalised into an inner-product index, so the threshold and its sweep read directly off [-1, 1].

Design choice: the model call stays simulated, deliberately

Every metric here, hit rate, latency, calls avoided, false-hit rate, depends on which question matched and never on the answer's text. Paying for real generations would change what the notebook costs to run and none of its results. This project needs no API key at all.

Evaluation

Semantic hit rate
0.724
exact-match baseline 0.691
False-hit rate
0.0135
budget 0.02
Model calls avoided
72.4%
at chosen threshold 0.80
Latency improvement
3.5×
5,000-request stream

The threshold sweep, a representative slice of the full 10-point sweep:

Threshold Hit rate False-hit rate
0.70 0.792 0.154
0.75 0.752 0.037
0.80 0.724 0.0135
0.85 0.706 0.0068
0.90 0.698 0.0009
0.92 0.695 0.000
Line chart plotting cache hit rate and false-hit rate against the similarity threshold from 0.70 to 0.98. Hit rate falls from 0.79 to 0.69 as the threshold tightens; the false-hit rate falls from 0.15 to zero. A dotted horizontal line marks the exact-match baseline hit rate of 0.69, and a vertical marker identifies the chosen operating point at 0.80 where the false-hit rate is 1.35%.

Hit rate and false-hit rate move together. The loosest threshold has the best hit rate and a 15% false-hit rate, choosing a threshold is choosing how much wrongness a saving is worth.

Main weakness: the headline metric is blind to the failure

This is the project's sharpest result, and it is in the data above: the loosest threshold (0.70) produces the best hit rate (0.792) and the lowest latency, and a 15.4% false-hit rate. Judged on every metric a normal dashboard shows, the worst configuration looks like the best one. The failure is silent: a fluent, confident, cached answer, with nothing logged. This is why the false-hit rate had to be built before the threshold could be chosen, and why the chosen point is 0.80, not 0.70.

Recommendation

Adopt semantic caching, threshold chosen from the curve

Pick the operating point against an explicit false-hit budget. Where a wrong answer is cheap, a game-help bot, move down the curve and take the savings. Where it is expensive, move up and pay for the calls. The measured curve makes that a decision with numbers on it, not a guess.

Next Iteration Priorities