Vector Embedding Models: Generation, Versioning, and Drift

Most teams building on top of large language models spend weeks tuning their retrieval prompts and almost no time on the system that produces the vectors those prompts depend on. That system is the embedding pipeline, the machinery that turns content into vector embeddings and keeps them fresh. Your choice of vector embedding models, and how you operate them, quietly decides whether your semantic search returns the right document or a plausible looking wrong one. When the pipeline lags, users see stale answers. When it silently drifts, relevance rots over months and nobody notices until a customer complains.

This post breaks down the production embedding pipeline as a system design problem. We cover how to generate embeddings in batch and in real time, how to version an embedding model so a change does not corrupt your index, how to detect drift before it degrades quality, and how to apply incremental updates without re-embedding your entire corpus every night. If you have already read our guide on the retrieval stack for RAG, think of this as the data plane that keeps that retrieval layer honest.

Why the Embedding Pipeline Is a Separate System

An embedding is a fixed-length vector that captures the meaning of a chunk of text, an image, or a user event. The retrieval layer treats these vectors as ground truth. It does not know how they were made or how old they are. That separation of concerns is useful, but it also means the pipeline has to guarantee three properties on its own: freshness, consistency, and reproducibility.

Freshness means a document that changed five minutes ago should be searchable with its new meaning, not its old one. Consistency means every vector in the index was produced by the same model with the same preprocessing, so distances between them are comparable. Reproducibility means you can rebuild any vector from its source and get the same result, which matters for debugging and audits.

Treating embedding generation as a throwaway script inside your ingestion job breaks all three. The moment you have two code paths that call the model with slightly different chunking, your index holds vectors that are not directly comparable, and cosine similarity starts lying to you.

Batch Generation: The Backbone

Batch generation is how you build and rebuild the bulk of your index. You take a corpus, split it into chunks, run each chunk through the embedding model, and write the vectors to your store. The interesting engineering is not the loop, it is everything around it: throughput, idempotency, and cost control.

Throughput and batching

Embedding models are far more efficient when you feed them many inputs at once. A single request per chunk wastes most of the GPU. The standard pattern is to accumulate chunks into micro-batches sized to the model's optimal input, run them concurrently, and backpressure when the downstream vector store cannot keep up.

# embed_batch.py
import asyncio
from itertools import islice

async def embed_corpus(chunks, model, store, batch_size=256, concurrency=8):
    sem = asyncio.Semaphore(concurrency)
    it = iter(chunks)

    async def run(batch):
        async with sem:
            vectors = await model.embed([c.text for c in batch])
            await store.upsert(
                [(c.id, v, {"doc_id": c.doc_id, "model_version": model.version})
                 for c, v in zip(batch, vectors)]
            )

    tasks = []
    while True:
        batch = list(islice(it, batch_size))
        if not batch:
            break
        tasks.append(asyncio.create_task(run(batch)))
    await asyncio.gather(*tasks)

Notice that every vector is written with a model_version in its metadata. That single field is what makes versioning and drift detection possible later. Do not skip it.

Idempotency and cost

Batch jobs fail halfway. If a rerun re-embeds everything, you pay twice and you risk a partial index where some vectors are new and some are old. Make the unit of work idempotent by keying each vector on a content hash of its chunk plus the model version. If the hash and version already exist in the store, skip it. This turns a full rebuild into a cheap no-op when nothing changed, and it turns a crash recovery into a resume rather than a restart.

Cost control matters at scale. Embedding ten million chunks with a hosted API can cost more than the rest of your infrastructure combined. Cache aggressively on content hash, deduplicate identical chunks before you call the model, and reserve the largest models for content where retrieval quality actually moves a business metric.

Real-Time Generation: The Freshness Path

Batch keeps the index correct in bulk. Real-time keeps it fresh. When a user edits a document, uploads a file, or triggers an event that must be searchable immediately, you cannot wait for the nightly job. You need a streaming path that embeds a single item and upserts it within seconds.

The clean design is an event-driven one. A change data capture stream or a message queue carries a "document changed" event. A consumer picks it up, fetches the current text, chunks it, embeds the chunks, and upserts them, deleting any stale chunks that no longer exist in the new version. This is the same fan-out-on-write idea we discussed in the context of feeds, applied to vectors.

# realtime_consumer.py
async def handle_change(event, loader, chunker, model, store):
    doc = await loader.fetch(event.doc_id)
    chunks = chunker.split(doc.text)

    vectors = await model.embed([c.text for c in chunks])
    records = [(f"{doc.id}:{i}", v,
                {"doc_id": doc.id, "model_version": model.version,
                 "content_hash": c.hash})
               for i, (c, v) in enumerate(zip(chunks, vectors))]

    await store.upsert(records)
    # remove chunks that existed before but are gone now
    await store.delete_where(doc_id=doc.id, keep_ids={r[0] for r in records})

The two paths together form a lambda-style architecture: batch for correctness and completeness, streaming for latency. Reconciliation runs periodically to catch anything the stream missed, comparing source timestamps against vector metadata and re-embedding the gaps.

Versioning: Changing the Model Without Breaking the Index

Sooner or later you will want to upgrade your embedding model. Teams rarely commit to one of the available vector embedding models forever. A newer model captures meaning better, or a cheaper one saves money, or you fine-tune one on your own domain. Here is the trap: vectors from two different models live in different spaces. You cannot compare a query embedded with model B against documents embedded with model A. The distances are meaningless.

This means an embedding model change is a full reindex, and it must be a safe one. The pattern that works is a versioned, dual-write migration.

First, tag every vector with its model_version, which you are already doing. Second, when you introduce model B, embed the entire corpus with B into a separate namespace or index while A keeps serving live traffic. Third, embed incoming queries with both models during the transition and route reads to A. Fourth, once B's index is complete and validated, flip query embedding to B and reads to B's index atomically. Finally, keep A around long enough to roll back, then delete it.

# version_router.py
class EmbeddingRouter:
    def __init__(self, active="A", shadow=None):
        self.active = active
        self.shadow = shadow  # e.g. "B" during a migration

    def embed_query(self, text, models):
        # always embed with active; optionally shadow-embed for eval
        primary = models[self.active].embed_one(text)
        if self.shadow:
            _ = models[self.shadow].embed_one(text)  # logged for offline compare
        return self.active, primary

The validation step before you flip is where teams get lazy. Do not trust that model B is better because the benchmark says so. Build an offline evaluation set of real queries with known-good results, embed it against both indexes, and compare recall and mean reciprocal rank. Only flip when B wins on your data, not on a public leaderboard. Our guide on evaluating AI systems with a second model covers how to build that harness.

Drift Detection: Catching Rot Before Users Do

Drift is the slow enemy. Nothing crashes, no alert fires, but relevance degrades because the distribution of your data has moved away from what the model was good at. A support knowledge base accumulates new product lines. A marketplace adds categories the model has never seen. The vectors are still produced correctly, but they cluster worse, and retrieval quality slips a few points a month until it is bad.

There are two distinct things to monitor. The first is embedding distribution drift: are the vectors your pipeline produces this week statistically different from last month's? You can track this with the mean and variance of embedding norms, the distribution of pairwise distances in a sample, or a population stability index over clusters. A sudden shift usually means your input data changed or your model client changed. A slow shift means the world moved.

The second, and the one that actually matters to users, is retrieval quality drift. Sample real queries, record which results the system returned, and measure how often the top results were clicked or thumbs-upped. Falling engagement on the top result is the truest drift signal you have, because it measures the outcome rather than a proxy.

# drift_monitor.py
import numpy as np

def population_stability_index(baseline, current, bins=10):
    edges = np.quantile(baseline, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    b = np.histogram(baseline, edges)[0] / len(baseline)
    c = np.histogram(current, edges)[0] / len(current)
    b, c = np.clip(b, 1e-6, None), np.clip(c, 1e-6, None)
    return float(np.sum((c - b) * np.log(c / b)))

# psi > 0.2 is a common threshold for "investigate"

Wire drift metrics into the same observability stack you use for the rest of your AI system rather than a one-off dashboard. Our post on LLM observability in production explains how to trace retrieval quality alongside model latency and cost so drift shows up next to everything else you already watch.

When drift crosses a threshold, the response is usually a targeted re-embedding of the affected segment, a chunking adjustment, or a model upgrade through the versioned migration above. Drift detection tells you where to spend the re-embedding budget instead of blindly rebuilding everything.

Incremental Updates: Re-Embedding Only What Changed

The naive pipeline re-embeds the whole corpus on a schedule. That is simple and wrong at scale. If one paragraph of a thousand-page manual changes, re-embedding the entire manual wastes compute and delays freshness for everything else.

Incremental updates hinge on content-addressed chunks. Hash each chunk. On update, diff the new chunk hashes against what is stored for that document. Embed only the chunks whose hash is new, upsert them, and delete the chunks whose hash disappeared. Unchanged chunks keep their existing vectors untouched.

This makes update cost proportional to the size of the change, not the size of the document. It also composes cleanly with versioning: an incremental update always writes the current model version, so a document that has not changed since a model upgrade still gets migrated the next time it is touched, and reconciliation catches the rest.

The subtle part is chunk boundary stability. If your chunker splits differently after an edit, every downstream chunk gets a new hash and you re-embed the whole document anyway. Use a chunking strategy with stable boundaries, such as splitting on semantic units or headings rather than a fixed sliding window that shifts when text is inserted near the top. This one design choice is the difference between an update that costs one chunk and one that costs a thousand.

Putting the Pieces Together

A production embedding pipeline is really four cooperating subsystems. Batch generation builds and rebuilds the index efficiently and idempotently. Real-time generation keeps recently changed content fresh through an event-driven path that shares code with batch. Versioning lets you change models through a dual-write migration validated on your own data. Drift detection and incremental updates keep the whole thing healthy over months without paying to rebuild from scratch.

If you are designing this for an interview or for a real product, the thing to articulate is the interplay: how freshness, consistency, and reproducibility trade off, and how each subsystem defends one of them. That framing is exactly what separates a senior answer from a script. For more on how these AI-era system design questions show up in interviews, the rest of our System Design writing on levelop.dev walks through the neighboring topics, from model serving to agent architecture.

Building AI-native products and preparing for system design interviews that increasingly test this material? Levelop helps engineers master modern system design with structured, interactive practice. Explore more on the Levelop blog.