Home / Case Studies / Hybrid Retrieval System

Hybrid Retrieval System for Earnings Call Interpretation

Combining vector retrieval, knowledge graphs, and agentic planning to improve financial due-diligence workflows.

Graph RAG LangGraph LLM-as-a-Judge ChromaDB Knowledge Graph Agentic AI Financial NLP

TL;DR

A hybrid Graph RAG system for long, mixed-format earnings-call transcripts, built on a LangGraph-style agentic loop that routes each query across vector search, financial tables, and a graph layer. An LLM-as-a-judge critic scores every answer against a grounding rubric and regenerates failures until they pass. The critic loop raised answers clearing the grounding bar from 62% to 91%, at about 88% agreement with human reviewers.

Contents

  1. Introduction and Problem
  2. Technical Approach
  3. Results

Introduction and Problem

Earnings calls are core inputs for institutional due diligence, but each transcript can run 50 to 300 pages of executive narrative, analyst Q&A, and financial tables. An analyst might ask "how did gross margin move quarter over quarter, and why", a question that spans prose (the "why") and exact numbers (the "how much") across several filings at once.

The system had to serve three shapes of that question, single-company Q&A, quarter-over-quarter narrative, and peer benchmarking, all grounded in evidence. Plain retrieval struggles with each: top chunks capture local semantics but miss cross-document and cross-quarter structure, values drift without table retrieval, and a single-pass fetch cannot adapt to whether a question needs metrics, risks, or transcript tone.

The question was whether retrieval could move beyond chunk relevance into reliable, evidence-grounded financial reasoning, and whether an automated critic could hold that grounding without a human checking every answer.

Technical Approach

The three evidence layers

Three sources are integrated so one pipeline can do both semantic retrieval and symbolic reasoning. A vector store (ChromaDB): transcript chunks embedded with all-MiniLM-L6-v2, filtered by ticker, filing type, and quarter, with a quarter-relaxed fallback when an exact match is missing. Structured Parquet tables: metrics, segment entries, and risk indicators under a shared quarter normalisation, giving deterministic filters that anchor numeric outputs. A graph layer: a company-quarter-metric entity schema in NetworkX with a Neo4j-compatible design, the foundation for richer temporal and peer reasoning.

The agentic loop

The pipeline runs as a LangGraph-style agentic loop. A planner model selects tools per query and returns a JSON policy (use_vector_search, use_metrics, use_segments, use_risks, plus vector_query). Retrieved evidence is fused into one JSON blob and passed to the response model with explicit grounding instructions, then handed to an LLM-as-a-judge critic before anything reaches the analyst.

Graph RAG pipeline diagram. A query passes to a planner that emits a JSON tool-routing policy, forking across three evidence layers: a ChromaDB vector store, structured Parquet tables, and a NetworkX graph layer. The three converge into evidence fusion, then a writer produces a grounded draft, which an LLM-as-a-judge critic scores against a grounding rubric and regenerates on failure, before a human-in-the-loop checkpoint releases the answer.

The agentic loop: a research agent gathers grounded context from document and live-data sources, a writer drafts, and an LLM-as-a-judge critic critiques and regenerates until the draft clears the rubric, with a human-in-the-loop checkpoint before anything ships.

Planner and tool routing: intent-aware retrieval selection, with a safe fallback to all-tools mode when the planner output is invalid. Evidence fusion: transcript snippets, metrics, segment data, and risk rows are combined into a single context package. The critic: an LLM-as-a-judge scores each answer against a grounding, relevance, and completeness rubric and regenerates any draft that fails until it clears the bar. Hallucination controls: low temperature and explicit prompts to avoid guessing numbers when the data is absent.

Design choice: enforce the rubric at generation time

The same critic-and-regenerate philosophy as the agentic compliance work: a judge model enforces the rubric while the answer is being written rather than trusting a single pass, so weak or ungrounded answers are caught and rewritten before an analyst ever sees them.

Results

Each answer is scored by the LLM-as-a-judge critic on the rubric below. The critic held about 88% agreement with human reviewers on pass/fail, and its regenerate-on-fail loop raised the share of answers clearing the grounding bar from 62% on the first pass to 91%, cutting analyst verification by roughly 40% on assisted queries. Tone and guidance synthesis were strong; strict numeric and temporal fact alignment remained the hardest case.

A sample of the critic's per-answer rubric scores:

Case Overall Relevance Factual Grounded Complete Clear
Tone and guidance 1.0 1.0 1.0 1.0 1.0 1.0
Risk relevance 0.1 1.0 0.0 0.0 0.2 1.0
Main weakness

Numeric grounding and quarter attribution remain the dominant error source in difficult queries.

Next iteration priorities