I measured my semantic cache. Then I turned it off.

Someone asks your assistant "What is Redis?" and gets back a confident, well written explanation of Memcached.

Nothing crashed. No error was logged. The latency looked wonderful, because the answer never went near a model. It came out of a cache that decided the two questions were close enough to be the same question.

That failure is the reason I spent a weekend measuring a feature I had already built, and the reason I ended up switching it off.

What a semantic cache is trying to do

Ordinary caching is exact. The key is the text of the prompt. Ask the identical question twice and the second answer is free.

The trouble is that people almost never ask the identical question twice. "What is Docker?" and "Tell me about Docker containerization" want the same answer, but as strings they share very little. An exact cache sees two different keys and pays for the model twice.

A semantic cache tries to close that gap. Instead of matching text, it converts each prompt into an embedding, which is a list of numbers meant to capture what the sentence is about. When a new prompt arrives, it looks for a stored prompt whose embedding is close by. If the closest one is nearer than some threshold, it serves that stored answer.

Everything rests on that threshold. Below it, two questions are treated as the same question.

The appeal is obvious. A cache hit costs nothing and returns in milliseconds. If a meaningful share of traffic is repeat questions in different words, you have removed that share of your bill.

I built this twice inside my LLM inference gateway, once on Redis vector search and once on Postgres with pgvector, so I could compare them. Then, before wiring it into anything real, I tried to answer a question I had written into the project spec at the start: what is the false hit rate, measured rather than assumed?

The test everybody runs

The natural way to test a semantic cache is to collect pairs of questions that mean the same thing and check how many the cache catches.

I wrote 120 of them. Real rephrasings, the kind people actually type:

Run the sweep against only these, and the result is genuinely encouraging. At a threshold of 0.80 the cache catches 76.7 percent of them. Three out of four rephrased questions answered for free.

If I had stopped there, I would have shipped it. The number was good, the feature worked, and the demo was convincing.

The test that changed my mind

The paraphrase set can only tell you how often the cache fires. It cannot tell you how often it fires when it should not, because every pair in it is supposed to be a hit. To learn anything about the failure, I needed pairs that look similar and mean different things.

So I wrote 120 more. Same shape, same vocabulary, same topic, opposite meaning:

Every one of these is a wrong answer waiting to happen. If the cache treats "What is a microservice?" and "What is a monolith?" as the same question, someone gets an answer about the opposite of what they asked, with no indication anything went wrong.

Then I swept the threshold from 0.80 to 0.99 and measured both sets at each step. The hit rate is how many paraphrase pairs the cache correctly catches. The false hit rate is how many near miss pairs it wrongly catches.

The numbers

Read the top row again. At the loosest setting, the cache catches 76.7 percent of the questions it should, and also 60.8 percent of the questions it must not. Nearly two thirds of the near misses come back as confident wrong answers.

The instinct is to tighten the threshold. Tightening does reduce wrong answers, but it destroys the correct ones faster.

By 0.88 the two rates cross. Past that point the cache is wrong more often than it is right, which is a strange thing for a cache to be.

At 0.95 it catches 5 percent of real paraphrases and 20 percent of near misses. It has stopped being a cache and become a random wrong answer generator with excellent latency.

The last row is the one that ended the feature. At 0.99, the cache catches zero paraphrases. Not one out of 120. And it still produces false hits on 8.3 percent of the near misses. Every possible benefit is gone and some of the harm remains.

There is no threshold on this chart I would put in front of users. That is the finding.

Why this happens

I expected a tradeoff. I expected to find a threshold that gave up some hits to buy safety. What I did not expect was that no such point exists, and the reason is worth sitting with.

An embedding model is trained to place text that appears in similar contexts near other text that appears in similar contexts. It learns topic, register, and shape.

Nothing in that training teaches it that two sentences make opposite claims.

"What is a microservice?" and "What is a monolith?" appear in the same articles, the same documentation, the same conversations. They are maximally related. A model trained on relatedness is doing exactly its job when it places them close together, and that correctness is the problem: relatedness is not the property a cache needs.

A cache needs to know whether two questions have the same answer. That is a much stricter relationship. "Compiler" and "interpreter" are near neighbours in meaning and complete opposites in what they should return.

Worse, the words that flip a meaning tend to be small and common. Async and synchronous. Encryption and compression. Microservice and monolith. These carry the entire semantic load of the sentence while barely moving the embedding, because the rest of the sentence, which is almost all of it, is identical.

So near misses do not sit safely far away with a clean gap before the paraphrases. They sit right on top of them. Adjusting a threshold cannot separate two things that overlap.

What I would build instead

Giving up on similarity does not mean giving up on caching. It means being honest about which parts are safe.

Three ideas in that diagram, in order of how much I trust them.

Normalise, then match exactly. Lowercase the text, trim the whitespace, strip the trailing question mark. This catches a real share of repeat traffic and it can never produce a wrong answer, because the questions genuinely are the same. It is unglamorous and it is free.

Verify before serving. Treat similarity as a shortlist rather than a decision. When a near neighbour turns up, ask a small cheap model whether the two questions have the same answer. You are still avoiding the expensive call, and you have replaced a distance measurement with something that actually reasons about meaning.

Narrow the scope. Every number above comes from general questions on many topics. A cache over one narrow intent, in one product surface, with a known set of things users ask, is a different problem with a different answer. The wide open case is the one that fails.

What I would not do is pick a threshold, watch the hit rate climb on a dashboard, and call it a win. Hit rate is the metric that made this feature look good for as long as I only measured hit rate.

What I actually shipped

The gateway still contains both semantic cache implementations, the Redis one and the pgvector one. They work. The sweep is committed next to them, along with all 240 pairs, and the README says plainly that the feature is off by default and why.

I thought about deleting the whole thing. I kept it because the measurement is more useful than the feature would have been, and because a cache that is off with a written reason is a better artifact than a cache that is on with a nice hit rate.

The tradeoff I went looking for turned out not to exist. Finding that out cost a weekend, which is considerably less than finding it out from a user who was told that a monolith is a microservice.

If you are running a semantic cache in production right now, I would gently suggest writing thirty near miss pairs for your own traffic and measuring your false hit rate before you finish reading anything else today. You may get a better answer than I did. Your users are already living with whatever the answer is.

The code, the pair set, and the full sweep are in llm-inference-gateway. If your pairs disagree with mine, I would genuinely like to see the numbers.