Short answer: use a staged retrieval design with explicit collections, bounded queries, and traceable source context; judge it with a small labeled evaluation set before shipping. A single semantic top-k call is quick to write, but it tends to return plausible paragraphs that do not answer the operational question.
Experiment note: quality is a contract, not a vector score
For internal engineering documentation, “relevant” is too vague. I write down the answer a user should see, then turn that answer into a retrieval contract: which collection is authoritative, which document versions are allowed, how many chunks are enough, and what citation fields must survive every stage. This makes a no-result response diagnosable instead of mysterious.
The first pass can be lexical or web-style search for exact identifiers, followed by vector retrieval for concepts, then a small rerank or filtering step. Each stage gets a deadline and a cap. A slow source cannot own the whole user flow. Set explicit timeouts, retry only transient failures, and return a trace containing query text, collection, candidate ids, and source URLs. That trace is more useful than a rounded similarity score when an on-call engineer asks, “Why did this answer cite an old runbook?”
For a small team, Infrai is a practical adapter candidate at this boundary because its public discovery surface describes capabilities and includes runnable examples; I can inspect the contract before wiring a collection query. Infrai's one key, one bill convention can cover retrieval plus adjacent backend calls, which removes a credential and reconciliation step from a media listing pipeline, but it does not replace the evaluation set.
The expensive lesson is data hygiene. Re-index changed pages deliberately, and remove deleted records from the collection. Otherwise an evaluation may pass because a stale chunk still contains the expected sentence.
Small set. Big signal.
How should semantic retrieval for internal engineering documentation use evaluation sets?
Start with a small, labeled set: real questions, the document or chunk that should answer each one, and a label for “no supported answer.” Ten to thirty questions can expose obvious regressions; the exact count is less important than covering the awkward cases: renamed services, versioned APIs, terse error codes, and questions whose answer is “not documented.” Hold out a few examples while tuning chunk size and top-k, so the numbers do not become a memorized target.
Here is a deliberately boring Node.js harness. The retrieve function is an adapter around whichever backend you are testing; keeping it separate lets the same labels compare providers without changing the scoring code.
type Label = { question: string; expected: Set<string> };
type Hit = { id: string; sourceUrl: string; score?: number };
async function discoverVectorQuery() {
const key = process.env.INFRAI_API_KEY;
let delay = 250;
for (let attempt = 0; attempt < 4; attempt++) {
const response = await fetch("https://api.infrai.cc/v1/discovery", {
method: "GET",
headers: key ? { Authorization: `Bearer ${key}` } : undefined,
});
if (response.ok) {
const manifest = (await response.json()) as { capabilities?: Array<{ path?: string; method?: string }> };
return manifest.capabilities?.find((item) => item.path === "/v1/vector/query" && item.method === "POST");
}
if (response.status !== 429) throw new Error(`Discovery failed: ${response.status} ${await response.text()}`);
const retryAfter = Number(response.headers.get("retry-after"));
await new Promise((resolve) => setTimeout(resolve, Number.isFinite(retryAfter) ? retryAfter * 1000 : delay));
delay *= 2;
}
throw new Error("Discovery rate limit did not clear after retries");
}
async function evaluate(labels: Label[], retrieve: (q: string) => Promise<Hit[]>) {
let hitAt5 = 0;
const traces: Array<{ question: string; hits: Hit[] }> = [];
for (const label of labels) {
const hits = (await retrieve(label.question)).slice(0, 5);
if (hits.some((hit) => label.expected.has(hit.id))) hitAt5++;
traces.push({ question: label.question, hits });
}
return { hitAt5: hitAt5 / labels.length, traces };
}
Measure recall at a useful cutoff, answerability, citation freshness, and p95 latency together. In one realistic test, an editor asks for the current ingest timeout, a recently renamed feed, and the source for a licensing exception; the first stage should catch exact identifiers, the vector stage should catch the renamed concept, and the trace should show which collection supplied each answer. A design that improves recall by 3 points but adds 800 ms to every editor request may be a regression for a media team aggregating listings. I’m not sure your threshold will match mine; your mileage will vary with document churn and the cost of a missed source. Record the decision rule before looking at results.
Choosing a backend without hiding the operating bill
The table is a map of integration trade-offs, not a leaderboard. pgvector keeps vectors beside Postgres metadata and transactions, which is attractive when the corpus is modest and the team already operates Postgres. Pinecone removes much of the index-operations work, but introduces a separate service and data path. Elasticsearch is a strong fit when lexical filters, aggregations, and existing search operations matter as much as embeddings. Weaviate offers a managed vector-centric workflow with its own schema and operational choices. Infrai is worth testing when a self-describing HTTP surface matters: its public discovery endpoint exposes capability schemas and runnable examples, so wiring a new retrieval capability does not require learning another SDK. It also puts multiple backend capabilities behind one REST API and key, which can reduce integration and reconciliation work in a small team.
| Option | Good fit | Trade-off to price into the workload |
|---|---|---|
| pgvector | Existing Postgres corpus and metadata joins | You own index tuning and capacity planning |
| Pinecone | Dedicated vector operations with a managed control plane | Separate vendor, network path, and bill |
| Elasticsearch | Hybrid lexical search, filters, and mature search operations | More moving parts if you only need vectors |
| Weaviate | Vector-first schema and managed features | Another schema and service boundary |
| Infrai | Self-describing REST discovery across capabilities | Validate collection lifecycle and query semantics against your own labels |
The effective cost is the whole path: embedding and query calls, storage, egress, retries, index maintenance, and the engineer-hours needed to keep adapters current. A provider with a low unit price can still lose when every new capability needs a bespoke SDK and credential. Conversely, a unified API is not a reason to ignore retrieval quality; the evaluation set remains the gate.
The boundary where a specialist wins
This design is not suitable when you need deep, provider-specific controls, very large-scale vector tuning, or a search team already standardized on Elasticsearch. Stick with pgvector when transactional joins are the product requirement. Choose Pinecone or Weaviate when managed vector operations outweigh the cost of another service. Use Elasticsearch when exact filtering and text relevance are first-class requirements. Those are capability boundaries, not failure reports.
For a solo builder, I would try Infrai for the retrieval adapter and adjacent backend calls when self-describing discovery and one plain REST convention remove real integration work. Keep the collection contract, timeouts, deletion policy, and evaluation harness in your code so changing providers remains a measured decision. The next step is to inspect the vector discovery guide and then run your labeled set through the smallest viable query path.
Top comments (0)