Home / Case Studies / Hybrid Retrieval & Eval Harness

Hybrid RAG Retrieval + Automated Evaluation Harness

Retrieval that survives error codes and paraphrase alike, and an evaluation suite generated from the bot's own traffic.

RAG Hybrid Search BM25 Reciprocal Rank Fusion Reranking LLM-as-Judge Evaluation FAISS

TL;DR

A support bot for appliance manuals that answers both vaguely-worded symptoms and exact error codes, then cites and checks every sentence against its source. A second system turns the bot's own logs into a regression test suite. Reranking lifted recall@10 from 0.78 to 0.94, and the generated suite caught a 14% failure rate hiding in real traffic.

Contents

  1. Introduction and Problem
  2. Technical Approach
  3. Results

Introduction and Problem

A support bot that answers from official appliance manuals has to serve two incompatible kinds of question. One user types "my fridge won't get cold", a fuzzy symptom in everyday words that never appears in the manual, which phrases the same fault as "not maintaining temperature". Another types "error CF", an exact code printed verbatim in exactly one passage. The first question needs semantic matching to bridge the vocabulary gap; the second needs the literal string and nothing else.

A single retriever is good at one of these and bad at the other. To an embedding model, ten error-code passages that differ only in a two-letter code all read the same, so the one token that matters is the one it weights least; keyword search has the mirror-image blind spot on paraphrase. Both failures are silent, surfacing only as a fluent, confident, wrong answer that no exception or dashboard catches.

This project asked whether one embedding model is enough or exact matching has to run alongside it, and, because a bot that invents a repair step is a liability, how to keep the answers grounded over time rather than only at launch.

Technical Approach

The retrieval pipeline

Two retrievers with complementary weaknesses run in parallel and are merged by rank, not by score. Dense vectors handle paraphrase; BM25 handles exact strings. The fused list is reranked by a cross-encoder that reads query and candidate together, and only the top chunks reach generation, where every sentence is cited and then checked against the chunk it cites.

Pipeline diagram. A user question splits into two parallel retrieval paths, dense vector search via BGE embeddings and FAISS, and BM25 keyword search, each returning fifty candidates. The two ranked lists merge through reciprocal rank fusion, pass through a cross-encoder reranker down to eight chunks, and feed a model call producing a cited answer, which a cheaper model call then checks for faithfulness. A dashed branch shows every generation being written to JSONL trace logs.

Two retrievers with complementary weaknesses, fused by rank and reranked before generation. Reranking lifted recall@10 from 0.78 to 0.94. Every answer is logged.

  1. Ingest

    Extract PDF text, strip the repeating headers and footers (left in, they become the most common text in the corpus and pollute every keyword score), unwrap line breaks, and chunk at ~300 words with 50 overlapping so a repair procedure is never severed mid-step. 13 manuals became 1,475 chunks.

  2. Build two indexes

    BGE embeddings (BAAI/bge-large-en-v1.5) in a FAISS inner-product index over L2-normalised vectors, so every score is a cosine similarity in [-1, 1]; rank_bm25 over the identical chunks.

  3. Fuse by rank

    Reciprocal rank fusion, 1/(60 + rank) from each list. A chunk both retrievers rank highly collects from both and rises.

  4. Rerank

    A cross-encoder (Cohere rerank-english-v3.0) scores the top 50 down to 8. Too slow for the whole corpus, which is why fast retrieval earns the candidate list first.

Design choice: fuse by rank, not by score

Cosine similarity lands around 0.4 to 0.8; BM25 is unbounded and routinely exceeds 3. Summing them lets BM25 shout down the vector signal for no principled reason, and any weighting has to be re-tuned the moment the embedding model changes. RRF discards magnitudes and keeps only positions, so it needs no normalisation and no weights.

Choices made with data, not assertion

Three parameters were measured, not assumed. Chunk size was swept at 150 / 300 / 600 words on the same gold set: 150-word chunks scored recall 0.806 against 0.722 at 300 and 0.667 at 600, contradicting the common 300-word default, because a tighter passage is less diluted by its neighbours. Abstention keys off cosine rather than the fused score, since cosine is the only signal on a fixed interpretable interval; the honest limit is that BGE cosine compresses on a small corpus, so an off-topic "who won the World Cup" scored 0.50, above the 0.45 cutoff, and the threshold needs tuning per corpus. Cost and latency come from the traces, read over logs already written: about $0.0016 per answer, roughly $1.60 per 1,000 at paid-tier rates.

An evaluation harness that writes itself

A second system treats the retrieval logs as raw material, because the cases a system actually struggles with beat any you would invent. To give the generator something to find, the bot answered its clean gold set plus a slice of realistic traffic (ambiguous, out-of-scope, and uncovered questions); on the clean set the judge found nothing, and every failure lived in the traffic, exactly as with real users.

Circular flow diagram. JSONL traces from the retrieval system are normalised, judged by a model into verdicts with failure reasons, embedded and clustered into named failure modes, turned into structured evaluation cases, and validated against a schema, with invalid cases discarded, producing a regression suite that tests the retrieval system, closing the loop. A side branch shows the suite itself being validated by seeding known bugs.

The suite is generated from the system's own traces, it found a 14% failure rate, almost all one mode, then validated by seeding known bugs and confirming it catches them.

  1. Mine

    Normalise JSONL traces into question / retrieved chunks / answer. Logging is therefore a designed interface rather than debug output: a trace missing its chunk ids is useless here.

  2. Judge

    A cheap model rules each answer good or bad and names the failure. Rules can check form, is every sentence cited, does the citation exist, but not whether a passage about a washing machine answers a question about a refrigerator. That is the failure class rules miss entirely.

  3. Cluster

    Embed the free-text failure reasons and cluster them, so many individual failures become a handful of named modes with counts. The 8 failures resolved into 5 modes, and 6 of the 8 were one family: the question named no appliance and the answer assumed one.

  4. Generate

    Emit Pydantic-validated EvalCase records whose target_failure_mode must be one the clustering actually discovered. An invented mode is rejected rather than silently corrupting the suite. 8 cases were emitted, 0 rejected.

Results

On a 36-question gold set, reranking is what earns the jump: hybrid with reranking reached recall@10 0.944 and nDCG@10 0.754, against a vector-only baseline of 0.778 and 0.547, with 94% of answer sentences grounded (123 of 131). Every system ran on the same set, because a hybrid number alone proves nothing.

System recall@10 nDCG@10
vector-only (baseline) 0.778 0.547
BM25-only 0.500 0.364
hybrid (RRF) 0.722 0.460
hybrid + rerank 0.944 0.754

Two honest notes. First, the reranker is what earns the jump: RRF alone actually scored below vector-only here (0.722 vs 0.778), because the gold set is symptom-dominated and dense retrieval already handles those, the cross-encoder is what closes the gap and then some. Second, the verified gold set is 35 symptom questions and 1 error-code question, so the error-code column is a single data point and is reported as such, not as a rate; the genuine error-code-vs-symptom comparison is named as the top next-iteration priority.

The self-generated suite surfaced a 14% failure rate (8 of 57 traces, 6 of them the unspecified-appliance mode), caught 100% of seeded bugs, and its judge agreed with a human on 86.7% of 30 hand-labelled traces.

Main weakness: the suite can only catch failures it has seen

The suite is validated by seeding bugs of rising subtlety. Stripping every citation, and replying "I do not know" to everything, are caught at the floor. The subtle case is an answer that is fluent, fully cited, and says nothing. Mining logs gets you the failures you have, never the ones you have not had yet, and on this small corpus the generated assertions also over-trigger on healthy answers, so the 100% seeded-bug catch comes with imperfect discrimination that a larger trace corpus would sharpen.

The judge's accuracy gates every other number, so it was measured

A judge is perfectly self-consistent when it is wrong; it will never tell you. So 30 traces were hand-labelled and agreement measured: 87%. The interesting part is the direction of the 4 disagreements, every one was the judge flagging an answer the human accepted (the ambiguous-appliance answers a person reads as "good enough for a small model"). The judge is stricter than the human, which for a regression detector is the safer bias, but it is a bias, and naming it is the point. A subtler trap also had to be fixed during the build: with a soft prompt the judge rubber-stamped everything as good, and only an explicit rubric with a worked example made it discriminate.

Next iteration priorities