Short answer: use a snapshot-bound cursor over a two-stage semantic search, and attach the snapshot ID to every page. That keeps a financial research dashboard from showing the same near-duplicate filing twice while an index refresh is running. The extra metadata costs a little latency; the alternative is an analyst chasing evidence that moved between pages.
The page that wakes me up is usually not “search is down.” It is an analyst reporting that page two contains a duplicate of page one, or that a filing visible ten minutes ago vanished when they clicked Next. In a fintech dashboard, those are evidence defects. A green 200 response does not make them harmless.
I work backwards from that alert. The signal should fire when a stable query returns a high near-duplicate rate across adjacent pages, or when the cursor's snapshot is no longer available. Log the query hash, embedding model version, index generation, candidate count, reranker latency, and result IDs. Never log the entire filing by default.
The most useful incident record is concrete. Capture the analyst's original query, the normalized query hash, page size, cursor age, and the ordered IDs returned on each request. Then record what the index reported for generation 481 and generation 482, including document counts and the embedding model digest. In one replay, an offset-based endpoint returned IDs A17 through A36, then an amendment inserted before A17 and shifted the next request to A36 through A55; the overlap was not visible in aggregate latency metrics. A preserved cursor would have pinned the boundary. A preserved generation would have made the result explainable six hours later, after the refresh job completed. During the fix, I would run the same query against both generations, compare the deduplication clusters, and check whether a representative changed because of a score tie or because the source record was re-embedded. This is slower than deleting duplicate rows after the fact, but it gives the on-call one deterministic question to answer: did the page change, or did the ranking change?
Page boundaries are evidence.
What should a financial research dashboard retrieval architecture guarantee?
Treat pagination as part of the retrieval contract, not a UI detail. At request time, resolve the query into a normalized form, select an index generation, and return an opaque cursor containing (or referring to) that generation. Every following request uses the same generation until the cursor expires. A refresh can build generation 482 while generation 481 continues serving existing cursors.
The cursor should sort by a deterministic tuple: relevance score, publication timestamp, and a unique record ID. Floating-point scores alone are not a tie-breaker. If two records have equal scores, the ID makes the boundary reproducible. Set a short expiry and return a clear “restart search” response when the snapshot is gone; silently switching generations is how duplicates enter an audit trail.
A practical response shape is small and boring:
type Page struct {
Snapshot string `json:"snapshot"`
Next string `json:"next_cursor,omitempty"`
Records []Record `json:"records"`
}
type Record struct {
ID string `json:"id"`
Score float64 `json:"score"`
Evidence string `json:"evidence"`
}
The evidence pointer matters. Store the source document ID and passage offsets, not only generated text. The RAG paper describes retrieval as a separate source of knowledge for generation; your dashboard still needs to make that source inspectable when an analyst opens a result.
How do retrieval, pagination, and semantic deduplication interact?
First retrieve a wider candidate set, then rerank, then paginate. For a page size of 20, retrieving only 20 vectors leaves no room to remove near-duplicates. I usually fetch 100 candidates, cluster records by a cosine-similarity threshold chosen from labeled pairs, keep the strongest representative per cluster, and only then apply the cursor boundary. The threshold is a policy knob, not a universal constant: false merges can hide a restatement, while missed merges waste an analyst's time.
Near-duplicate detection needs a record identity rule. Hash normalized issuer, filing type, period, and source revision for exact repeats; use embeddings for paraphrased headlines and amended text. Keep both identities in telemetry so a postmortem can say whether the error came from normalization, embedding drift, or cursor handling.
func unique(records []Record, similar func(Record, Record) bool) []Record {
kept := make([]Record, 0, len(records))
for _, candidate := range records {
duplicate := false
for _, prior := range kept {
if similar(candidate, prior) {
duplicate = true
break
}
}
if !duplicate {
kept = append(kept, candidate)
}
}
return kept
}
That loop is intentionally obvious, not a production-scale index. The production path should use the vector store's neighborhood query or an offline clustering job; the contract remains the same. A reranker can improve relevance, but it must receive stable document IDs and return them unchanged so pagination can be audited.
The alert-to-action runbook for drifting pages
When the duplicate-rate alert fires, freeze the query sample and compare the snapshot IDs across requests. If they differ, stop blaming embeddings: the pagination contract was broken. If snapshots match, compare candidate counts, model versions, and threshold configuration. Re-run the same query against the preserved generation and retain the exact cursor as an incident artifact.
I once started with the assumption that “next page” meant “run the query again with an offset.” That assumption fails as soon as inserts land ahead of the offset. Cursor pagination avoids that shift, but only when the sort tuple and generation are immutable. Short sentence. Keep the invariant visible in code review.
Instrument four timings separately: query encoding, candidate retrieval, deduplication, and reranking. The decision axis is retrieval quality versus latency, so one blended histogram hides the trade-off. Track page-level duplicate rate, citation coverage, and the percentage of cursors restarted after expiry. A latency improvement that increases restarts is not an improvement for an analyst.
Where this design is not a good fit
This design is not suitable for a rapidly changing, non-audited feed where “latest items” matters more than repeatable evidence; a simple time-based offset can be easier to operate there. It is also a poor fit when the index cannot retain generations long enough for a user's reading session. In that case, reduce cursor lifetime and make the restart explicit, or choose a storage layer with snapshot support.
The catch is operational cost: retaining old generations consumes storage, and reranking 100 candidates adds tail latency. Stick with a smaller candidate window when the corpus is tiny and quality labels show little duplication. Your mileage may vary because the right threshold depends on issuer language and amendment patterns; publish the evaluation set and the chosen trade-off with the service configuration.
Top comments (0)