DEV Community

Cover image for Count Attempts, Not JSONL Rows

Count Attempts, Not JSONL Rows

Dashboards lie when rows mean attempts

A trace contains six records with the same tool name. How many times did the agent try the operation?

Six is a tempting answer. It might also be three calls with start and finish records, two calls with duplicated telemetry, or six unrelated operations that happen to use the same tool.

Counting rows is easy. Counting attempts requires an identity model.

I maintain AgentInspect. The examples here were tested against 6.31.35 using a synthetic read operation. This is about interpreting execution evidence, not implementing live retries or promising exactly-once effects.

Give each identifier one job

The following vocabulary works even if your tracing library uses different field names:

Identity Meaning Changes on a retry?
Run One captured execution Usually not within that execution
Operation One logical request being attempted No
Attempt One execution attempt of that operation Yes
Event or delivery One observation or its transport occurrence Potentially
Idempotency key A downstream service's effect-deduplication scope Defined by that service's contract

An attempt can emit multiple events. One event can be delivered more than once. One operation can span several attempts. These are relationships, not interchangeable counters.

One logical operation contains two attempts; each attempt has lifecycle records, and a repeated delivery does not create another attempt.

Transport duplication and repeated application execution must remain distinguishable.

Record identity where the operation is invoked

Install the pinned release:

npm install --save-exact [email protected]
Enter fullscreen mode Exit fullscreen mode

Use the metadata-bearing step() API. In this version, the step.tool() shortcut accepts the name and function, not a third metadata argument.

import { inspectRun, step } from "agent-inspect";

function attempt(attemptId, attemptNumber, fn, retryOf) {
  return step("lookup_policy", fn, {
    type: "tool",
    metadata: {
      toolName: "lookup_policy",
      operationId: "fixture-op-1",
      attemptId,
      attemptNumber,
      ...(retryOf ? { retryOf } : {}),
    },
  });
}

await inspectRun("two-read-attempts", async () => {
  await attempt("a1", 1, async () => {
    throw new Error("Synthetic temporary failure");
  }).catch(() => {});

  return attempt("a2", 2, async () => ({ found: true }), "a1");
}, { traceDir: ".agent-inspect", silent: true });
Enter fullscreen mode Exit fullscreen mode

The swallowed first error is deliberate fixture behavior. It is not a general error-handling recommendation. The fixture represents a bounded recovery path whose first attempt fails and second succeeds.

The IDs are static because the test runs in an isolated fixture. Real application identities need suitable uniqueness and scope. Don’t reuse fixture-op-1 across unrelated operations and expect the evaluator to infer the difference.

Ask the rule the question you mean

After opening the generated trace with openTraceFile, evaluate two contracts:

import {
  defineTraceContract,
  evaluateTraceContractRead,
} from "agent-inspect/checks";

const oneAttempt = defineTraceContract({ retry: { maxAttempts: 1 } });
const twoAttempts = defineTraceContract({ retry: { maxAttempts: 2 } });

const strict = evaluateTraceContractRead(trace, oneAttempt);
const bounded = evaluateTraceContractRead(trace, twoAttempts);
Enter fullscreen mode Exit fullscreen mode

For the source fixture, the first fails with contract.retry.max-attempts; the second passes. The result refers to attempts associated with the recorded logical operation, not the total number of JSONL rows.

The current implementation groups explicit retry identity within a run. It uses distinct attempt IDs when available and supports other documented evidence forms, including validated attempt numbering. Missing or conflicting identity is not permission to invent a retry relationship from a repeated name.

The important engineering decision comes before the rule: where does your application define the logical operation? An adapter cannot reliably reconstruct a business boundary you never recorded.

Redelivery is a useful negative control

The companion test copies the second attempt's exact terminal record and appends it to the source trace. It then evaluates the same contracts again.

Source variation Maximum one attempt Maximum two attempts
Two recorded attempts Fail Pass
Same trace plus exact terminal redelivery Fail Pass

This is a deliberately narrow claim. An identical redelivered record should not manufacture an extra attempt. It does not follow that every repeated-looking record can be discarded.

If the same declared identity appears with different arguments, status, or ownership, you have a conflict to investigate. Blindly applying new Map(events.map(e => [e.id, e])) can erase contradictory evidence by keeping whichever row arrived last.

A useful test set therefore includes both exact duplication and conflicting reuse. Keep their expected outcomes separate.

Don’t promote attempt counts into effect counts

Two attempts do not prove two writes. One timed-out attempt does not prove zero writes. A successful response does not necessarily prove the effect was observed by a recipient.

For a read operation, that distinction may only affect latency analysis. For payments, notifications, or reservations, it changes the recovery protocol.

An idempotencyKey recorded in metadata is evidence that a key was recorded. To claim that the destination deduplicated the effect, you need the destination's behavior and appropriately scoped receipts. AgentInspect's trace analysis is not that destination.

This is why I would keep retry accounting, effect verification, and authorization as separate checks. They may use related identifiers, but they answer different questions.

Check the identity after conversion, too

If you export traces, rerun the attempt-count check after reading the exported representation. A structurally valid span tree can lose operation metadata while retaining every tool name.

The included OTLP experiment preserves the retry verdict for this fixture when run status is evaluated separately. It also exposes a run-status discrepancy after recovery, documented in the companion interoperability article. Passing one invariant is not proof of complete semantic equivalence.

Start with one operation, two attempts, and one duplicate delivery. That tiny fixture is often more informative than a large trace full of plausible-looking counters.

Implementation references

Top comments (5)

Collapse
 
mike-johnstone profile image
Mike Johnstone •

One extra fixture I'd find useful: the same logical operation resumed in a new run, with one attempt in each. Both runs could pass maxAttempts: 1 while the operation has two attempts overall. A stable business-operation ID would let a separate rollup expose that without changing the per-run rule.

Collapse
 
raju_dandigam profile image
Raju Dandigam Google Developer Experts •

@mike-johnstone Exactly. maxAttempts here is scoped to one captured run, so two resumed runs can each pass while the business operation has two attempts overall. A stable operation ID across runs would support a separate rollup; it should stay distinct from each attempt ID and any downstream idempotency key. Your resumed-run fixture is a good test for that boundary.

Collapse
 
mudassirworks profile image
Mudassir Khan •

the table separating operation, attempt and delivery is the useful part, because the six rows with the same tool name question is exactly where our dashboards lied to us. we counted rows and called it retries, and a duplicated telemetry export made one flaky tool look like it failed three times. once we keyed on an attempt id the number dropped a lot. how do you handle traces from libraries that never emit an attempt id, infer from timing or mark the count as unknown?

Collapse
 
raju_dandigam profile image
Raju Dandigam Google Developer Experts •

@mudassirworks I would not infer attempts from timing or repeated tool names. If the library supplies validated attempt numbering or an explicit retry link, use that evidence; otherwise mark the count unknown for the legacy trace and add an attempt ID at the application invocation boundary going forward. Your duplicated-export case is the right negative control: exact redelivery should not increase attempts, while conflicting reuse of an ID should be surfaced rather than silently deduplicated.

Collapse
 
mudassirworks profile image
Mudassir Khan •

that negative control framing is cleaner than what i had. we ended up logging an attempt id at the invocation wrapper and tagging legacy rows as unknown instead of guessing, which made the dashboards less pretty but way more honest.

the conflicting reuse case is the gnarly part though. do you hard fail on it or just flag it and keep going?