DEV Community

Cover image for Multi-Hop Retrieval: Building Evidence Chains Without Cascading Errors
Hu Jin
Hu Jin

Posted on

Multi-Hop Retrieval: Building Evidence Chains Without Cascading Errors

Some questions cannot be answered from one passage. Example: “Which team owns the service that stores EU customer invoices?” You may need hop 1 to find which service stores those invoices, then hop 2 to find which team owns that service. That is multi-hop retrieval: iterative evidence gathering where later searches are conditioned on earlier findings.

This is not the same as:

  • Query rewriting, which rewrites the user text up front (multi-query, HyDE) without necessarily depending on retrieved facts.
  • Agentic RAG, which is a general tool-calling control loop (the agent might perform multi-hop, or might just retry the same search).

Multi-hop is specifically about chained evidence and the failure mode where hop 1 is wrong.

Datasets like HotpotQA popularized evaluation for multi-hop QA with supporting-fact annotations; production systems need the same discipline around provenance even when your corpus is private.

The iterative pattern

Hop, link entities, hop again, then answer with a provenance chain

  1. Retrieve with the original question (hybrid search).
  2. Extract bridge elements: entities, IDs, dates, constraints supported by chunks.
  3. Formulate the next query from bridges + remaining unmet slots.
  4. Retrieve again (optionally constrained by metadata).
  5. Stop when slots are filled or hop budget is hit.
  6. Generate using the union of supporting chunks; cite each hop.

Entity linking between hops

Bridges should be explicit objects in code, not a vague refreshed prompt:

@dataclass
class Bridge:
    kind: str          # "service", "team", "doc_id", ...
    value: str
    support_chunk_ids: list[str]
    confidence: float  # from ranker or LLM judge

def extract_bridges(question, chunks) -> list[Bridge]:
    # LLM or rules: only emit bridges entailed by chunk text
    ...

def next_query(question, bridges, open_slots):
    return llm(
        "Write a search query to fill open slots, using bridge values.\n"
        f"Question: {question}\nBridges: {bridges}\nOpen: {open_slots}"
    )
Enter fullscreen mode Exit fullscreen mode

Linking quality matters: prefer identifiers that appear verbatim in text (svc-invoice-eu) over ambiguous names (“the billing service”).

Provenance: the evidence chain

Each hop keeps supporting chunks for the final citation set

Store a chain, not only the final context window:

hop=1 query="EU customer invoices storage service"
  chunk:C12 → bridge service=invoice-eu
hop=2 query="team owns service invoice-eu"
  chunk:C88 → team=Payments Platform
answer cites C12 + C88
Enter fullscreen mode Exit fullscreen mode

If you only keep hop-2 chunks, you cannot justify why invoice-eu was chosen—and you cannot debug cascades.

Preventing error cascades

A wrong hop-1 entity sends hop-2 into the wrong neighborhood

Problem: hop 1 returns a plausible but wrong service; hop 2 “succeeds” with a confident owner for that wrong service; the final answer looks coherent.

Mitigations that help in practice:

  1. Keep top-k bridges, not top-1. Branch 2–3 hop-2 queries; fuse or ask the generator to compare support.
  2. Verify bridges against chunk text (string match or entailment) before using them.
  3. Require dual support for critical slots when possible (two independent chunks).
  4. Abstain when hop-1 confidence is low or hop-2 contradicts hop-1 constraints (region, product).
  5. Cap hops (often 2–3). More hops multiply error rates.
  6. Separate faithfulness checks on the final answer against the union of cited chunks.
def multihop(question, max_hops=2, bridges_per_hop=3):
    evidence = []
    bridges = []
    q = question
    for hop in range(max_hops):
        hits = hybrid_search(q, top=8)
        evidence.extend(hits)
        new_bridges = extract_bridges(question, hits)[:bridges_per_hop]
        new_bridges = [b for b in new_bridges if verify(b, hits)]
        if not new_bridges:
            break
        bridges.extend(new_bridges)
        if slots_filled(question, bridges):
            break
        # branch on bridges instead of committing to one
        rankings = [hybrid_search(next_query(question, [b], open_slots(question, bridges)), top=8)
                    for b in new_bridges]
        evidence.extend(reciprocal_rank_fusion(rankings)[:8])
    return generate_with_citations(question, dedupe(evidence), bridges)
Enter fullscreen mode Exit fullscreen mode

When multi-hop is the right tool

Use it when answers structurally need intermediate facts. Do not use it as a generic “make RAG smarter” switch—latency grows roughly with hops, and cascades create confident failures.

Pairing tips:

  • Use query rewriting to improve hop-1 recall.
  • Use an agent only if hop-2 needs a different tool (SQL, ticket system), with budgets.
  • Use GraphRAG when intermediate links are already materialized as edges (ownership graphs), which can collapse some hops into a neighborhood read.

Limitations

  • Public multi-hop scores do not imply your private corpus will behave the same.
  • LLM bridge extraction can invent entities not in the chunks—verification is mandatory.
  • Branching multiplies cost; measure p95 latency with production-like hop budgets.
  • Comparison questions (“which of A or B …”) need explicit compare logic, not only chained lookup.

Related reading


About the author

I write about Applied AI, MCP, Azure AI Search, and RAG. Follow me on Dev.to and Hashnode for more.

Top comments (1)

Collapse
 
agentsearchhq profile image
AgentSearch •

The point about keeping top-k bridges instead of committing to the top-1 is the part I'd stress most, since a confident wrong hop 1 is what quietly poisons everything after it. One thing I'd add is a cheap check that the bridge value actually co-occurs with the original constraint (like the EU region in your invoice example) in the hop-2 chunks, and abstain when it doesn't. That catches the plausible-but-wrong service case without needing a full entailment pass on every hop.