Ask a documentation assistant "how does authorization work in MCP?" for the 2024-11-05 release of the Model Context Protocol, and a straightforward RAG pipeline will give you a confident, well-cited answer about OAuth 2.1 flows, protected resource metadata and bearer tokens. It is wrong. The 2024-11-05 specification has no authorization section at all; it arrived in 2025-03-26. Every citation was real, every quote accurate, and the answer still described a protocol that did not exist yet.
That failure is the reason this project exists. I had already built a hand-built RAG pipeline over the Laravel docs, and I wanted to see what LangChain and LangGraph would add on a corpus where the frameworks should genuinely fit: a versioned specification, questions that need routing, and answers that need to say "this release does not cover that". The result is an agentic RAG assistant over the MCP spec, and, more usefully, a set of eval numbers that did not say what I expected.
Versions make RAG harder than it looks
The MCP spec ships as dated releases, from 2024-11-05 to 2026-07-28, and they genuinely disagree. JSON-RPC batching was added in 2025-03-26 and removed again in 2025-06-18. The 2026-07-28 release removed the initialize handshake entirely. An answer that blends releases is worse than no answer.
So every chunk is tagged with its spec_version at ingest, and every search is scoped to the selected release plus the version-independent pages. The scope is a small, boring function, and the pgvector filter it produces is the whole trick:
def version_scope(version: str, versions: SpecVersions) -> Scope:
"""`all` searches everything; `latest` or a dated version also includes unversioned pages."""
if version == ALL:
return None
return (versions.resolve(version), UNVERSIONED)
def scope_filter(scope: Scope) -> dict[str, Any] | None:
"""PGVector metadata filter for a scope; None means no filter."""
if scope is None:
return None
return {"spec_version": {"$in": list(scope)}}That filter is why the opening example surprised me: the wrong answer came from chunks that really were tagged 2024-11-05. The docs site keeps a copy of its guides for every release, but they are near-copies of the current guides with only the links rewritten. The 2024-11-05 "Understanding Authorization in MCP" tutorial describes OAuth flows that the 2024-11-05 specification never defined.
The fix was a label, not a model
Each source passed to the answer model is now labelled Specification, Guide or Extension, taken from the docs site's own navigation tabs, and the prompt says only Specification sources define what a release supports. The same question now opens with "the specification sources for version 2024-11-05 do not mention authorization; these flows appear only in Guide sources". No bigger model, no extra call: just metadata the corpus already had.
The graph
The question flow is a LangGraph state graph with six nodes. A route node uses structured output to decide whether a question needs the docs, is small talk, or is off topic. Retrieve searches the selected release. Grade checks the retrieved chunks, and when too few are relevant a rewrite node rephrases the query and the graph searches again, up to twice, before it declines and lists the closest sections it found. Answer writes from the relevant chunks only, with numbered citations.
The interesting edge is the one after grading. It is a plain function that LangGraph calls to pick the next node:
def make_after_grade(
relevance_threshold: int, max_rewrites: int
) -> Callable[[GraphState], GradeOutcome]:
"""Enough relevant chunks: answer. Too few: rewrite, up to the cap; then decline."""
def after_grade(state: GraphState) -> GradeOutcome:
if sum(state.get("grades", [])) >= relevance_threshold:
return "answer"
if state.get("rewrites", 0) < max_rewrites:
return "rewrite"
return "decline"
return after_grade
graph.add_conditional_edges(
"grade",
make_after_grade(relevance_threshold, max_rewrites),
["answer", "rewrite", "decline"],
)
graph.add_edge("rewrite", "retrieve")This is where LangGraph earned its place. Conditional edges and a capped loop map one to one onto the design, and its streaming API gives you an event as each node starts and finishes, which is what drives the live progress in the web UI ("Searching the 2026-07-28 docs...", "Checking which of the 6 chunks are relevant...") without any extra plumbing.
Free tiers shaped the design
Everything ran on free tiers: Gemini 3.1 Flash-Lite for routing, grading and answers, and Voyage for embeddings and reranking (used because they're free; larger models will return better results). Some Gemini Flash models stopped at 20 requests a day, so grading every chunk with its own call was never an option. Grading is one structured call that returns the numbers of the relevant chunks and a note on what is missing, which also feeds the rewrite prompt. A typical question costs three model calls: route, grade, answer.
Where plain code beat the framework
Retrieval is hybrid: BM25 keyword search and pgvector semantic search, merged with reciprocal rank fusion, then reranked by Voyage down to the six chunks the answer sees. LangChain has a retriever for each of those steps, and three of them did not survive contact with a versioned corpus.
EnsembleRetrievermerges results by their text. The same section appears word for word in several releases, so under the "all versions" scope it collapsed chunks from different releases into one and lost the versionBM25Retrieverlives in the sunsetlangchain-communitypackage, and the underlyingBM25Okapiscoring gives words found in half the chunks almost no weight, so common words like "server" stopped countingVoyageAIRerankrebuilds each document without its id, which the citations need
The fusion I replaced EnsembleRetriever with is twenty lines, keyed on the stored chunk id rather than the text, and it records each retriever's rank for the trace panel:
def reciprocal_rank_fusion(
rankings: Mapping[str, Sequence[Document]], c: int = RRF_K
) -> list[Document]:
scores: dict[str, float] = {}
ranks: dict[str, dict[str, int]] = {}
first: dict[str, Document] = {}
for name, ranking in rankings.items():
for rank, doc in enumerate(ranking, start=1):
key = doc.id or doc.page_content
scores[key] = scores.get(key, 0.0) + 1 / (c + rank)
ranks.setdefault(key, {})[name] = rank
first.setdefault(key, doc)
ordered = sorted(scores, key=scores.__getitem__, reverse=True)
return [
first[key].model_copy(update={"metadata": _annotated(first[key], ranks[key], scores[key])})
for key in ordered
]
The parts that did fit were the unglamorous ones. The LangChain indexing API with a SQL record manager gave me incremental ingestion for free, skipping unchanged chunks and deleting pages removed upstream. with_structured_output(..., include_raw=True) gave typed routing and grading decisions plus the token counts for the trace. And init_chat_model made every model a provider:model setting, so Gemini and Voyage are examples rather than dependencies.
What the evals showed
I wrote 46 questions and checked every expected fact against the docs repo myself: version-specific questions for every release, cross-version ones ("when was batching added, and when was it removed?"), questions about features a release does not have, off-topic questions that should be declined, and greetings. A runner scores retrieval, LLM-judged key facts, citations and correct declines (off-topic questions, and features a release lacks, handled without a normal answer), then I ran four configurations:
$ make evals-compare
run config pass retrieval facts citations ok-declines false dec.
vector-only vector 89% 83% 83% 81% 88% 2
hybrid hybrid 87% 86% 82% 86% 75% 2
hybrid-rerank hybrid + rerank 94% 89% 83% 89% 100% 2
baseline hybrid + rerank + grading 89% 89% 82% 89% 88% 3I expected the full agentic pipeline to win. It did not.
- Reranking was the change that helped. It fixed a cross-version question and every "this release does not have that feature" question, and hybrid plus rerank was the best configuration at 43 of 46
- Grading and the rewrite loop did not earn their extra call. They declined "How long can tool names be?" because only one chunk was relevant and the threshold is two, and fixed nothing the reranked search had missed
- Three failures appeared in every configuration, so no search change could fix them: the router declined "How does ping work?" and a question about how tasks changed between releases as off topic, and "Is ping still part of MCP?" never retrieved the changelog that says ping was removed
The honest caveat: with 46 questions, one question is about two percentage points, and most gaps here are one to three questions. The judge was also the answering model grading its own work. Treat the ordering as indicative, not settled. It still changed what I would build next: a better router and a lower threshold before any more agentic machinery.
Interestingly, my Laravel project found the opposite about hybrid search: there it lost to vector-only. Two small question sets pointing in different directions is the best argument I know for measuring on your own data rather than trusting the usual advice.
Summary
LangGraph is a good fit when the flow really is a graph: routing, a capped retry loop and per-step streaming came almost for free. LangChain's indexing API and structured output saved real work. The retrieval building blocks were where plain code won, because a versioned corpus breaks assumptions about text being unique. And the most effective fixes were cheap ones: labelling sources by type, and adding a reranker. The grading loop that makes the diagram look agentic did not move the numbers.
The code, the eval questions and the results are on GitHub, and the README covers where the frameworks helped piece by piece. If you are building RAG over documentation that has versions, permissions or a lot of near-duplicate pages and want it measured rather than guessed at, get in touch via the contact section.