Update: layering a lexical prefilter, and where it earns its place

The best response to the post above came from a reader who did not stop at agreeing with it:

Have you thought of using multi-layered cache processing, first a SimHash or MinHash for near duplicate detection, then a semantic match for approximate guesses? Might reduce the false-hit rate but of course not 100% accurate as a full hash.

What struck me is that we had arrived at the same shape from opposite directions. The last section of that post is a two-stage pipeline: normalise and match exactly first, treat similarity as a shortlist second. His proposal is also a two-stage pipeline. We had different candidates for stage one, and neither of us had measured which one belongs there.

That is a testable disagreement, so I tested it. What follows is his idea and my existing design measured against the same pair set, and the combined answer is better than either of us had on our own.

The shared instinct

Cheap filter first, expensive filter second. Only pay for the expensive one on candidates that survive. It is how most retrieval systems are laid out, and it is why both of us landed there.

Where we differed is what stage one is for. I was reaching for a stage that could not be wrong. He was reaching for a stage that was fast. Those turn out to be very different requirements, and the measurement is what separates them.

What I measured

The evaluation set from the post above is 121 paraphrase pairs that should be cache hits and 121 adversarial near-miss pairs that must not be. I computed three lexical measures over both: character 3-gram Jaccard, word-set Jaccard, and a 64-bit SimHash.

One deliberate choice. I computed exact Jaccard rather than building a MinHash sketch, because MinHash is an estimator of Jaccard. Measuring Jaccard directly gives the ceiling of what any MinHash configuration could reach, so the result speaks to the whole family rather than to one tuning I happened to pick.

For the prefilter to help, paraphrases need to score high and near misses need to score low.

Every separation is negative.

On word-set Jaccard the pairs that must never be merged are close to four times more lexically similar than the pairs that should be. The filter does not fail to discriminate. It discriminates confidently, in the wrong direction.

Why it comes out backwards

This is not noise, and it was sitting in the two tables above the whole time.

The near misses are minimal edits:

The paraphrases are near-total rewrites:

Lexical similarity is a direct measure of edit distance. Flipping the meaning of a question is, by construction, a tiny edit — usually one noun. That is the entire mechanism. The prefilter fires hardest on exactly the pairs that are dangerous and stays quiet on exactly the pairs that are safe.

Following it through as a gate

Means are suggestive; what matters is the behaviour when it is actually wired in as stage one.

In an AND gate a candidate must clear the lexical threshold before the embedding stage runs at all. That makes the paraphrase pass rate a hard ceiling on end-to-end recall, and the near-miss rejection rate the most the gate can possibly buy in safety.

At a word-set Jaccard cutoff of 0.30:

The gate caps recall under 6% while removing 8% of the false hits — the trade running the wrong way, paid for with an extra stage. Loosening the cutoff lets more paraphrases through but proportionally more near misses with them. Tightening it removes false hits, but by then it has already removed every true hit.

The OR version, where either stage firing is enough to serve, can only add false hits.

The principle this gave me

I did not have a clean way to say why my two-stage design should work and this one should not. Now I do, and it came out of testing his version rather than defending mine.

Layered filters only help when the stages fail independently.

SimHash and embeddings do not fail independently. They fail on the same pairs, for a related reason: two strings that are nearly identical in characters are also nearly identical in embedding space, because the embedding is computed from those characters. Asking a cheaper method to check the same property is not a second opinion — it is the first opinion with less information.

That is most of why the embeddings failed in the first place, and it is why a cheaper measure of the same thing cannot patch it.

It also explains why the verifier stage does work. It is not a better distance measurement. It answers a different question — do these two questions have the same answer? — which is the property a cache actually needs, and the one nothing in the embedding pipeline was ever measuring.

Where the prefilter earns its place

Here is the part I did not expect, and the reason this update improves the design rather than just closing a question.

The lexical stage is a bad safety gate and a genuinely good retrieval stage. Those are different jobs, and once they are separated both ideas fit in the same pipeline:

Two places his suggestion lands, both real:

• Candidate retrieval. MinHash LSH narrows the search space before the exact comparison runs. That is a latency win, and a good one, on a stage where being approximate costs nothing because the verifier still has the final say.
• The normalise-then-match-exactly tier. This is the shingling and hashing machinery doing what it is actually reliable at. The guarantee there comes from exactness, not from the sketch.

Speed, both times. Never safety. That distinction is the thing I took from this exchange, and it is now the rule the design is built on: similarity is allowed to propose, and only a verifier is allowed to decide.

The other two questions

The same comment asked two more things worth answering.

What is Anthropic doing differently? Less than it looks, and the difference is not the one people expect. Their prompt caching is not semantic at all — a hit requires an exact prefix match, identical tokens up to a marked breakpoint.

But the structural difference is not exact versus fuzzy. It is what gets cached. A prompt cache stores the encoded prefix state so those tokens do not have to be reprocessed. The model still runs. It still generates a fresh answer to the question that was actually asked. Mine skipped the model entirely and replayed a stored answer.

That is the whole risk surface. A prefix cache's worst case is a miss, and you pay full price. An answer cache's worst case is that it is confidently wrong. One is a compute cache and the other is an answer cache, and only one of them can be incorrect.

Isn't the whole point of AI to have some randomness? I would separate two things this merges: variance in how a question is answered, and variance in which question was answered.

Temperature explores the distribution of good responses conditioned on an input. A cache false hit changes the input — it answers a question the user did not ask. No sampling temperature has ever turned a prompt about Redis into an essay about Memcached.

There is an irony worth landing gently, too. The cache is the least random component in the whole stack: frozen, deterministic replay of a response generated once, at some temperature, on some past day. If you genuinely valued output diversity, the semantic cache is the first thing you would switch off.

Rerun it

The test is committed as eval/lexical_sweep.py, with results in eval/results/lexical_sweep.json.

python -m eval.lexical_sweep

That hashing exists because of this exercise. A result you cannot tie to its inputs is a screenshot, not a measurement.

If you write your own pairs and the separation comes out positive, I would like to see them. The interesting version of this update is the one where somebody's traffic has near misses that look nothing alike, and the gate works exactly as proposed. Thanks to the reader who pushed on it — the pipeline is better for it than the one I published.