The trace starts with retrieval. Generation appears underneath it. The ordering test passes.
But generation began while retrieval was still running.
“A appears before B” can describe the order in which events were encountered. “A's result was available before B began” describes an execution dependency. “B actually consumed A's result” is a stronger data-flow claim. They are not three ways of saying the same thing.
I maintain AgentInspect. This article tests those distinctions with controlled promises and [email protected]. The fixture does not invoke a model, search engine, or vector database.
Make the race deterministic
Avoid a demonstration that depends on a slow network. Hold retrieval open until the test explicitly releases it:
import { inspectRun, step } from "agent-inspect";
await inspectRun("overlapping-retrieval", async () => {
let release;
const ready = new Promise(resolve => { release = resolve; });
const lookup = step.tool("retrieve_policy", () => ready);
await step("generate_answer", async () => "synthetic answer", {
type: "llm",
});
release({ policyId: "fixture-policy" });
await lookup;
}, { traceDir: ".agent-inspect", silent: true });
Here type: "llm" labels an instrumented boundary. It does not turn the callback into a model call or capture hidden reasoning. The callback returns a fixed string.
Retrieval is invoked before generation, but the result cannot become available until after generation completes. The fixture encodes that relationship without guessing a millisecond delay.
Start order is insufficient when the second operation depends on the first result.
Use a typed relation across tool and model steps
After opening the generated file with openTraceFile from agent-inspect/readers, evaluate the relation:
import {
defineTraceContract,
evaluateTraceContractRead,
} from "agent-inspect/checks";
const contract = defineTraceContract({
run: { requireCompleted: true },
steps: {
orderRelations: [{
before: { kind: "TOOL", name: "retrieve_policy" },
after: { kind: "LLM", name: "generate_answer" },
mode: "happens-before",
requireEndpoints: true,
}],
},
});
const result = evaluateTraceContractRead(trace, contract);
Use the typed relation because this crosses a TOOL-to-LLM boundary. tools.requiredOrder is a tool-only surface, not a generic list of arbitrary workflow nodes.
TraceContract is beta, and steps.orderRelations is explicitly experimental in this release. Keep the version pin and test the exact semantics you intend to rely on.
The measured results for the overlap fixture are:
| Mode | Result | Question being tested |
|---|---|---|
first-occurrence |
Pass | Was the first matching retrieval encountered first? |
happens-before |
Fail | Did the first matching retrieval finish before generation started? |
all-occurrences |
Fail | Did all matching retrieval work finish before any matching generation started? |
Overlap can produce diagnostic information even when first-occurrence ordering passes. Read findings and warnings, not only the top-level label.
Stronger is not always more correct
all-occurrences is a global boundary, not an automatic pairing algorithm.
Consider a legitimate two-turn interaction: retrieve, generate, retrieve again, generate again. A rule requiring all retrieval to precede all generation can reject that workflow even if each individual answer waited for its own retrieval.
That is not a reason to weaken the rule until it passes. It is a reason to model the intended scope. Use separate runs or explicit application-level links that associate each answer with its retrieval result. Don’t assume that matching names imply pairwise dependency.
Similarly, requireEndpoints: true matters. An ordering statement should not quietly satisfy your product requirement when one of the necessary endpoints is missing.
Fix the application dependency first
The corresponding positive control awaits retrieval before invoking generation:
const policy = await step.tool("retrieve_policy", async () => ({
policyId: "fixture-policy",
}));
await step("generate_answer", async () => policy.policyId, {
type: "llm",
});
Inside a captured run, this fixture passes the tested all-occurrences relation. The change is the dependency in the application, not a delay inserted into the test.
For a real model call, completion still does not establish consumption. Record a bounded result reference at retrieval and the reference actually supplied to generation. An application assertion can compare those references without storing the retrieved document or prompt.
Even matching references prove wiring, not factual grounding. Whether the answer uses the evidence correctly requires a separate domain or semantic check.
Keep incomplete execution visible
The current implementation has a narrower boundary worth documenting. In the companion fixture, one generate_answer starts early and never finishes; another matching generation completes later. The isolated all-occurrences ordering rule passes, while the full contract with completion checking fails through run.status.
This was reproduced against 6.31.35. It is not a claim that the ordering rule completely diagnoses every unfinished occurrence. Do not ship an ordering-only gate or describe that diagnostic gap as fixed.
The test deliberately creates an unawaited pending operation to expose the boundary. That is a negative control, not an application pattern to copy. Keep a valid fixture, an overlap fixture, a repeated-turn fixture, and an incomplete fixture in your contract tests.
Ask for the smallest defensible conclusion
A trace can establish that a recorded step preceded another. With adequate timestamps and boundaries, it can establish that one finished before the other began. With application-owned references, it can help establish which result was supplied downstream.
None of those facts alone proves that an answer is correct.
The practical improvement is to name the obligation precisely. If generation needs retrieval output, test result availability. If it must use a particular retrieval result, test the data-flow link. If the answer must be grounded, test the answer against that evidence.
One green ordering check should not have to impersonate all three.
Implementation references
- Ordering implementation, including completed and running occurrences
- TraceContract surface and limitations
- Companion:
examples/probes.test.mjs, bothD03tests and the awaited positive control.

Top comments (7)
The three-way split — "A appears before B" vs "A's result was available before B began" vs "B actually consumed A's result" — is the distinction I wish more trace tooling forced you to make. The failure I keep seeing in RAG agents is the first one quietly standing in for the third: the ordering test is green, retrieval technically started first, and generation ran against an empty or stale result the whole time. Nobody notices until an answer is confidently wrong and the trace "proves" everything was in order.
Your happens-before contract nails the availability claim cleanly. The one I still find hard is the consumption claim — proving generation read retrieval's output, not just that it was free to. In practice that seems to need data provenance (tagging the value and checking it actually reached the consumer's inputs), not just timestamps, since a step can complete after another with zero data dependency between them. Does agent-inspect have a way to assert consumption, or is happens-before the strongest relation it models today?
@max_quimby You have the boundary right. The
orderRelationscheck used here in agent-inspect establishes availability, not consumption; this fixture does not provide a built-in provenance assertion. I would tag the retrieved result with a bounded reference and compare it with the reference actually passed into generation. A missing or stale reference should fail even ifhappens-beforepasses. That proves wiring, while grounding the answer still needs a separate check.The repeated-turn case also makes me want a stale-result negative control: hold retrieval A, start a newer request B, then release A after B completes. Even if both steps finish, the application should reject A when it belongs to the wrong request or an invalidated generation. A per-request correlation ID plus an explicit cancellation or freshness rule would make that failure observable without logging the retrieved content.
@innokentyb That is a strong negative control. I would let A and B both complete so a completion or ordering check can still be green, then assert that generation for B accepts only a retrieval reference carrying B’s request ID and current freshness epoch. Releasing A afterward should be recorded as stale and ignored, whether or not cancellation reached the underlying retrieval. That tests the application decision without storing retrieved content.
Your three-way split — appeared first, finished first, actually consumed — is the bit I'll steal, because in our planner work the middle one is where things quietly go wrong. A planner acting on a half-ready retrieval feels like the same failure shape as acting on a step whose result is still UNKNOWN: the trace looks ordered, so nothing complains, but the decision was made on air. Do you check the bounded result reference at the point of consumption, or is that a check you'd put at the contract level instead?
@aheadkit I would check the bounded reference at the consumption boundary: record the retrieval result ID and the ID actually supplied to generation, then compare them in an application assertion. The current
orderRelationscontract checks timing; it cannot infer data flow from step names or timestamps. If those explicit links become part of a contract later, the same comparison could move there without claiming that ordering proves consumption.That's the check the contract is missing — the IDs, not the timestamps. Ordering can tell you the steps ran in sequence, it can't tell you generation actually ate what retrieval found. Going to try that assertion on a real pipeline and see what it catches that the timing check walks past. What would you do with a mismatch once it fires — fail the run, or just log it?