Agent RAG Architecture: The Retrieval Stack for 2026
Every AI product that answers questions about your data is running the same five layers underneath, and most teams only think hard about one of them. They pick a vector database, ship it, and then spend the next three months confused about why the assistant confidently cites a paragraph that does not answer the question. Retrieval quality decides the product.
The retrieval stack is where AI products actually succeed or fail. The model is mostly a commodity now. What separates a product that people trust from one they abandon is whether the right 2,000 tokens land in the context window at the right moment. That is a retrieval problem, not a model problem.
This is a working guide to agent RAG architecture: chunking, embeddings, vector databases, hybrid search, and re-ranking, plus the agentic loop that ties them together. It is written for engineers who have shipped a naive RAG prototype and watched it fall apart on real data.
What agent RAG architecture actually means
Classic retrieval augmented generation is a straight line. A user asks something, you embed the query, you pull the top k nearest chunks from a vector store, you paste them into a prompt, and the model answers. One retrieval, one generation, done.
Agent RAG architecture breaks that line into a loop. The model decides whether to retrieve at all, decides what to search for, reads what came back, judges whether it is sufficient, and searches again with a better query if it is not. Retrieval becomes a tool the agent calls rather than a preprocessing step that happens to it.
The difference matters because real questions are rarely answerable in one hop. "Why did our error rate spike last Tuesday and did we ship anything related?" needs an incident lookup, then a deploy log lookup scoped by the timestamp the first lookup returned. A single-shot pipeline cannot do that. An agentic rag architecture can, because the second query depends on the first result.
Everything below still applies in both cases. The agentic loop makes retrieval quality more important, not less, because a bad first retrieval sends the agent down a wrong path that it will then spend three more calls defending.
Layer 1: chunking, where retrieval quality is won or lost
Chunking is the least glamorous part of a rag pipeline and by far the highest leverage. You are deciding, ahead of time and without knowing the questions, what unit of text will be retrievable. Get it wrong and no amount of embedding model upgrading will save you.
Fixed-size chunking and why it disappoints
The default everyone starts with is 512 or 1,000 tokens with a 10 to 20 percent overlap. It is fast, it is predictable, and it slices sentences in half. Worse, it strips context: a chunk that reads "this reduced latency by 40 percent" is useless when the paragraph naming the system is 200 tokens upstream.
Fixed-size chunking is a reasonable baseline for uniform prose. It is a bad default for documentation, code, transcripts, and anything with structure, which is most of what companies actually want to search.
Structural and semantic chunking
Structural chunking respects the document's own boundaries: split on markdown headings, on function definitions, on speaker turns, on table rows. The document already tells you where its ideas start and stop, so use that instead of guessing with a token counter.
Semantic chunking goes further and splits where the embedding of consecutive sentences diverges past a threshold, cutting at topic shifts rather than at arbitrary offsets. It costs more to build because you embed at the sentence level first, and it is worth it on long unstructured text where headings do not exist.
Contextual chunk headers, the cheapest win available
The single highest return change most teams can make is prepending context to every chunk before embedding it. Anthropic's contextual retrieval work reported that adding a short generated summary of where a chunk sits in its parent document cut failed retrievals substantially, and it costs one cheap model call per chunk at index time.
def contextualize(chunk: str, doc_title: str, section: str, summary: str) -> str:
"""Prepend locating context so the embedding carries document identity."""
header = f"Document: {doc_title}\nSection: {section}\nContext: {summary}"
return f"{header}\n\n{chunk}"
def build_index_records(doc, chunks, summarizer):
records = []
for i, chunk in enumerate(chunks):
summary = summarizer(doc.text, chunk) # one cheap LLM call per chunk
records.append({
"id": f"{doc.id}::{i}",
"text_for_embedding": contextualize(
chunk, doc.title, doc.section_of(i), summary
),
"text_for_display": chunk,
"metadata": {
"doc_id": doc.id,
"url": doc.url,
"updated_at": doc.updated_at,
},
})
return recordsNote the split between text_for_embedding and text_for_display. What you embed and what you show the model do not have to be the same string, and treating them separately unlocks a lot.
Layer 2: embeddings and the vector database
An embedding model turns text into a vector so that semantically similar text lands nearby in high-dimensional space. A vector database stores those vectors and answers nearest-neighbour queries fast enough to sit in a request path.
Choosing an embedding model
Three properties matter more than the leaderboard position. Dimensionality drives your storage and query cost, and models supporting Matryoshka truncation let you trade a little recall for a lot of index size. Maximum input length has to comfortably exceed your chunk size plus the contextual header. Domain fit beats general benchmark scores: a model trained on web text will underperform on dense legal or medical corpora regardless of how it ranks on MTEB.
The practical rule is to pick one, measure retrieval recall on a labelled set of fifty real questions from your own product, and only then consider swapping. Teams routinely upgrade embedding models and see nothing, because their actual bottleneck was chunking.
What a vector database actually buys you
A vector database gives you approximate nearest neighbour indexing (usually HNSW or IVF), metadata filtering that runs alongside the vector search rather than after it, and the operational parts nobody wants to build: incremental upserts, replication, and index rebuilds without downtime.
What it does not give you is relevance. Vector search retrieves what is semantically close to the query, which is not the same as what answers the query. That gap is the reason the next two layers exist.
Filtering deserves particular attention. Pre-filtering on metadata before the vector search is usually what you want for permissions and freshness, and getting this wrong is how RAG systems leak documents across tenant boundaries. Treat retrieval permissions as an access control problem, not a ranking problem.
Layer 3: hybrid search, because embeddings cannot spell
Dense vectors are bad at exact tokens. Search for an error code, a product SKU, a function name, or a person's surname, and semantic similarity will cheerfully return the conceptually adjacent thing instead of the exact match. Keyword search, meanwhile, is excellent at exact tokens and useless when the user phrases a question in words the document never used.
Hybrid search runs both and fuses the results. The standard fusion method is Reciprocal Rank Fusion, which combines rankings rather than scores and therefore needs no score normalisation between two systems whose numbers mean completely different things.
from collections import defaultdict
def reciprocal_rank_fusion(result_lists, k=60, weights=None):
"""Fuse ranked ID lists from dense and sparse retrievers."""
weights = weights or [1.0] * len(result_lists)
scores = defaultdict(float)
for weight, results in zip(weights, result_lists):
for rank, doc_id in enumerate(results, start=1):
scores[doc_id] += weight * (1.0 / (k + rank))
return sorted(scores, key=scores.get, reverse=True)
dense = vector_index.search(query_embedding, top_k=50)
sparse = bm25_index.search(query_text, top_k=50)
candidates = reciprocal_rank_fusion([dense, sparse], weights=[1.0, 0.7])[:50]Retrieve wide here. Pulling 50 candidates per retriever and narrowing later is far better than pulling 5 and hoping. Recall at this stage is what caps the quality of everything downstream, and you cannot re-rank a document you never retrieved.
Layer 4: re-ranking, the step most teams skip
Retrieval optimises for speed across millions of documents, so it uses a bi-encoder: query and document are embedded separately and compared by cosine distance. That is cheap and approximate. A cross-encoder re-ranker reads the query and the document together and scores actual relevance, which is far more accurate and far too slow to run over a whole corpus.
The pattern is to retrieve 50 candidates cheaply, re-rank them with a cross-encoder, and pass the top 5 to the model. You are spending one extra model call to convert a broad, noisy candidate set into a small, precise one.
This matters more than it sounds. Context windows got large enough that teams started stuffing 30 chunks into the prompt and calling it done, but models attend unevenly across long contexts and irrelevant chunks actively degrade the answer. Five well-ranked chunks reliably beat thirty mediocre ones, and they cost less.
Layer 5: the agentic loop
Now assemble it. What makes an agent rag architecture agentic is that the model controls the loop rather than sitting at the end of it.
A working loop has four moves. Query planning rewrites the user's question into one or more search queries, decomposing multi-part questions and resolving pronouns against conversation history. Retrieval executes the hybrid search and re-ranking described above. Sufficiency judgement asks the model whether what came back actually answers the question. Iteration reformulates and searches again when it does not, with a hard cap on rounds.
That cap matters. Without a budget, agents loop on unanswerable questions until they time out or hallucinate something to escape. Two or three retrieval rounds, then answer with what you have and say plainly what you could not find.
Query rewriting is the highest-value piece. Users ask "does it support SSO?" with no subject, and embedding that string retrieves noise. A rewrite step that turns it into "does Acme Enterprise plan support SAML single sign-on" changes the retrieval result completely. This is the same discipline as broader context engineering: what reaches the model matters more than which model it is.
Failure modes worth instrumenting
Retrieval systems fail quietly, which is why observability is not optional. Four failures cover most incidents.
The chunk was never retrieved, meaning the answer existed but recall missed it, and the fix is chunking or hybrid search rather than prompting. The chunk was retrieved but ranked below the cutoff, which is a re-ranking problem. The chunk was retrieved and the model ignored it, which is usually context dilution from injecting too much. The chunk was stale, which is an indexing freshness problem that no retrieval tuning can fix.
Distinguishing these requires logging the retrieved candidate set, the post-re-ranking order, and what was actually injected, for every request. Teams that only log the final answer cannot tell these four apart and end up guessing. The same tracing discipline that applies to AI agent evaluation applies here: measure the retrieval step separately from the generation step, or you will optimise the wrong one. Pair it with LLM observability so every stage is traceable in production.
Build a small labelled evaluation set early. Fifty real questions with the document that should answer each one is enough to measure recall at 10 and to know within minutes whether a change helped. Without it, every tuning decision is guesswork.
A reference architecture
Putting the layers in order, an agent rag architecture that holds up in production looks like this.
- At index time: parse documents, chunk structurally, generate contextual headers, embed, and write to a vector database alongside a keyword index and metadata for filtering and permissions.
- At query time: rewrite the query, run dense and sparse retrieval in parallel with pre-filters applied, fuse with RRF, re-rank the fused candidates with a cross-encoder, and inject the top handful with source attribution.
- Around all of it: trace every stage, evaluate retrieval separately from generation, and re-index on document change rather than on a nightly cron if freshness is part of your product promise.
None of these layers is individually difficult. The failure mode is skipping four of them, shipping a vector database and a prompt, and concluding that RAG does not work.
Frequently asked questions
What is agent RAG architecture?
Agent RAG architecture is a retrieval augmented generation design where the model controls retrieval as a tool rather than receiving a fixed set of chunks before it runs. The agent plans queries, retrieves, judges whether the results are sufficient, and searches again if they are not. The underlying stack is the same as classic RAG (chunking, embeddings, a vector database, hybrid search, and re-ranking), but the control flow is a bounded loop instead of a single pass.
How is agentic RAG different from regular RAG?
Regular RAG does one retrieval per user turn and the pipeline decides what to fetch. Agentic RAG lets the model decide whether to retrieve, what to search for, and when to stop, which makes multi-hop questions answerable. The cost is latency and unpredictability, so production systems cap the number of retrieval rounds and fall back to answering with what is available.
Do I still need a vector database if models have long context windows?
Yes, for anything beyond a small corpus. Long context lets you inject more, but attention degrades across long inputs and cost scales with every token you send on every request. A vector database plus re-ranking keeps injected context small and precise, which is both cheaper and measurably more accurate than dumping everything in.
What chunk size should I use for RAG?
Start at roughly 500 to 800 tokens with contextual headers and structural boundaries, then measure rather than guess. The right size depends on your documents: dense reference material wants smaller chunks, narrative or procedural content wants larger ones. Build a fifty-question evaluation set and tune chunk size against retrieval recall, because the correct answer differs by corpus.
Is hybrid search worth the extra infrastructure?
Almost always, if your content contains identifiers, error codes, product names, or any exact tokens users will type. Dense retrieval alone reliably misses exact-match queries, and Reciprocal Rank Fusion makes combining the two straightforward without score normalisation. If your corpus is pure narrative prose and users never search for specific strings, dense-only is defensible.
Where should re-ranking sit in the rag pipeline?
Immediately after fusion and before context injection. Retrieve 30 to 50 candidates from hybrid search, re-rank them with a cross-encoder, and pass only the top 3 to 8 into the prompt. Re-ranking is the cheapest single quality improvement available to most teams and it is also the step most commonly missing.
Further reading
The original retrieval augmented generation paper by Lewis and colleagues (2020) is still the clearest statement of the core idea. For practical guidance on approximate nearest neighbour indexes, the HNSW paper is the reference implementations converged on. More engineering guides are on the Levelop blog, and you can see what we are building at Levelop.