Semantic Caching for LLMs: Cut Repeat Query Costs

Every production LLM app hits the same wall: users ask the same thing in different words, and each rephrasing triggers a fresh, full-price model call. "How do I reset my password?" and "I forgot my password, what now?" are one question to a human but two distinct strings to a naive backend, each costing tokens, latency, and money. Semantic caching closes that gap by matching queries on meaning instead of exact text, so one cached answer can serve every phrasing of the same intent.

This is a support piece in our broader LLM cost optimization guide. Use it as the deep dive on caching by meaning.

What is semantic caching?

Semantic caching stores past query and response pairs and retrieves them by how similar a new query is in meaning, not in text. When a request arrives, the system embeds it into a vector, searches a vector store for the closest previous query, and if the nearest match is close enough, returns the cached response instead of calling the model again.

A classic key-value cache keys on the literal string, so changing one word misses the lookup and you pay for another call. Exact-match caching (sometimes called language model caching in its simplest form) captures almost none of the variety in real traffic, where users reorder words, add filler, and paraphrase. Keying on meaning is why a semantic caching llm layer can lift cache hit rates from a few percent to twenty, forty, or more, cutting both cost and tail latency at once.

How semantic caching works

An llm semantic caching layer sits in front of your model as a lookup gate with four steps. First, embed the incoming query. Second, compare that vector against stored query vectors using a distance metric such as cosine similarity. Third, find the nearest neighbor and its score. Fourth, apply a threshold: above it, return the cached response; below it, fall through to the LLM and write the new pair back to the cache.

Here is the core logic in Python:

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

class SemanticCache:
    def __init__(self, embed_fn, threshold=0.92):
        self.embed_fn = embed_fn
        self.threshold = threshold
        self.entries = []

    def lookup(self, query):
        q_vec = self.embed_fn(query)
        best_score, best_response = 0.0, None
        for vec, _, response in self.entries:
            score = cosine_similarity(q_vec, vec)
            if score > best_score:
                best_score, best_response = score, response
        return best_response if best_score >= self.threshold else None

    def store(self, query, response):
        self.entries.append((self.embed_fn(query), query, response))

In production you would not loop in Python. You push vectors into a purpose-built vector cache such as Redis with vector search, Milvus, Qdrant, or pgvector, and let it do approximate nearest neighbor search at scale.

Tuning the similarity threshold

The threshold is the most important knob. Set it too high and you reject valid matches, so the hit rate collapses toward exact-match levels. Set it too low and you serve answers to questions the user did not ask, which is worse than a miss because it is silently wrong. Most teams start conservative, around 0.9 to 0.95 cosine similarity, and loosen only after measuring false positives on real traffic.

Semantic caching versus prompt caching

These are often confused, but they work at different layers. Prompt caching, offered natively by Anthropic and OpenAI, caches the processed state of a long repeated prefix (a system prompt, a shared document, a tool schema) so the model does not reprocess those tokens. It still runs the model, just cheaper. See Anthropic's prompt caching docs and our guide on prompt caching to cut inference costs. Semantic caching can skip the model call entirely when a semantically equivalent query was already answered. The two are complementary: prompt caching makes the calls you make cheaper, semantic caching avoids duplicate calls. A mature cost stack runs both.

Building a semantic cache for your LLM app

You need three components: an embedding model, a vector store, and a policy layer that decides when to trust a hit. The embedding model determines match quality. A small model keeps lookups cheap and low-latency, which matters because the cache is on the hot path, but it must still distinguish "cancel my subscription" from "pause my subscription" or it will produce dangerous false positives. Since embedding costs far less than a full generation, a slightly larger embedding model is usually worth it.

Open-source projects such as GPTCache productionize this pattern, adding pluggable embedding backends, vector stores, and eviction policies, so starting from a library is usually faster than rolling your own.

Measuring the cost savings

The value of a semantic cache is a direct function of your hit rate and the cost of the calls it avoids. If the cache serves a fraction H of requests, model spend drops by roughly H, minus the small cost of embedding and vector search. As an illustration, an app making one million calls a month at two cents each spends twenty thousand dollars; a thirty percent hit rate avoids three hundred thousand calls, saving thousands per month even after embedding and vector-store costs, while improving tail latency on every hit. Treat these as a model to plug your own numbers into, not a promise. Instrument from day one: track hit rate, the similarity-score distribution on hits and misses, and any wrong answers that trace back to a cache hit.

Pitfalls and when not to use semantic caching

The biggest danger is the false-positive match: two queries close in embedding space that need different answers. "Refund policy for physical goods" and "refund policy for digital goods" embed closely, yet the answers differ, and a too-low threshold serves the wrong one. Personalized or stateful responses are also a poor fit; if the answer depends on the user's account or the current time, a shared cached response is simply incorrect, so cache only genuinely shared content or key on user context.

Freshness matters too. When prices, docs, or policies change, cached answers go stale, which is a cache invalidation problem you must solve before production.

Production considerations

Give entries a time-to-live so nothing lives forever, and add an explicit purge path so a changed source document evicts every answer derived from it; tying entries to the source data version makes targeted invalidation possible instead of full flushes. Add an eviction policy (least-recently-used is a sensible default) so a bounded cache drops the right entries when it fills. Watch for embedding drift: if you upgrade the embedding model, old and new vectors live in different spaces, so re-embed the cache or version it by model and never compare vectors from different models.

Where semantic caching fits in your cost stack

Semantic caching is one lever among several. It pairs with prompt caching for the calls you cannot avoid, and with LLM model routing, which sends easy queries to cheaper models. Caching removes duplicate work, routing right-sizes the rest, and prompt caching discounts shared context. Used together they cut inference bills sharply without degrading quality. See our AI cost optimization guide and the Levelop blog for the full picture.

Conclusion

Semantic caching is one of the highest-leverage LLM cost optimizations because it attacks the most wasteful thing your system does: paying full price to answer the same question twice. Get the embedding model, the threshold, and invalidation right and you cut your inference bill while making the product faster. The team at Levelop writes regularly about patterns that make LLM apps cheaper and more reliable. Pair this with prompt caching and model routing, and let your traffic tell you where the savings are.