I wrote a guide for Null Studio on what happens when the AI model under your feature gets retired or replaced. The advice is the boring kind: put every model call behind one interface, pin versions, keep the model name in server-side config, validate what comes back, and keep a test set so a swap is a measured change instead of a guess.
Then I did the thing I keep doing lately. I grepped my own repos for model names.
Two products came up. Upwork Scout, the job-alert SaaS I run, calls Claude to score jobs against a freelancer's profile and to import a pasted profile. VoiceDash, the white-label portal I build for voice AI agencies, calls OpenAI to summarize and score calls, and it has an agent settings screen with a "LLM Model" dropdown. One of these repos mostly follows my guide. The other one has a dropdown that saves your choice to the database and then changes nothing.
Here is what I found, and what a model upgrade would actually cost me in each.
Upwork Scout: close to the guide, with one shared knob
The matcher lives in one file, lib/ai.ts, and the cron route is its only caller. The model is read once at the top:
const MODEL = process.env.AI_MODEL || "claude-haiku-4-5";
That gets three things from my checklist right.
The model is config, not code. Swapping it is an environment variable change, not a code edit. The comment above it even says when I would flip it: use a bigger model "if match quality ever matters more than cost."
The output is constrained and then checked anyway. The call uses structured outputs with a JSON schema (score integer, reason string, no extra properties). I still parse defensively after:
if (res.stop_reason === "refusal") return null;
const parsed = JSON.parse(text) as { score: number; reason: string };
const score = Math.max(0, Math.min(100, Math.round(Number(parsed.score))));
if (!Number.isFinite(score)) return null;
return { score, reason: String(parsed.reason || "").slice(0, 200) };
A new model that returns 105, or a float, or a reason that runs to a paragraph, gets clamped instead of leaking into an email.
Failure degrades instead of breaking. Any error returns null, and the pipeline falls back to filters-only for that job. If a model gets retired under me on a Sunday, users get slightly worse alerts, not no alerts. I explained the cost side of this design in a post about metered model calls.
Where it falls short of my own guide:
-
One knob, two jobs. The profile import route has its own copy of the same line:
process.env.AI_MODEL || "claude-haiku-4-5". So the matcher and the importer share one variable. If I setAI_MODELto a bigger model to improve match quality, the importer moves too, whether I tested it or not. Two features, two different tasks (scoring versus extracting a profile from messy pasted page text), one switch. They should each have their own name, or at least the importer should not silently follow the matcher. -
The default is an alias, not a dated snapshot.
claude-haiku-4-5is the short name. My guide says pin the exact version in production so changes happen when you decide. I did not. - No test set. Nothing in the repo replays a fixed set of job and profile pairs and compares scores. If I swap models today, the only evidence I would have is whether the alert emails feel right. I wrote a whole post about testing outputs you cannot unit test, and this repo does not do it.
So: a swap here is a one-line config change, but a blind one.
VoiceDash: the model lives in five places
The call analysis route is the part users actually see. When a call ends, the Retell webhook fires a request to /api/conversations/analyze, which can run up to three model calls on the transcript: a summary, a 1 to 10 evaluation, and topic tagging. All three look like this:
const res = await openai.chat.completions.create({
model: "gpt-4o-mini",
// ...
});
The model name is written out three times in one file. There is no config, no shared helper, no constant. Changing models means editing three lines and redeploying, and forgetting one means two features run on the new model and one on the old.
It gets more interesting because the OpenAI key is not always mine. The route does this:
const apiKey = workspace?.aiApiKey || process.env.OPENAI_API_KEY;
An agency can bring its own key. That means the hardcoded model has to be available on every customer's OpenAI account, not just mine. A model retirement here is not only my migration. It is a silent change for every workspace that pasted in a key.
The validation is thinner too. The evaluation asks for JSON and parses it inside try { ... } catch {}, so a format change from a new model just means the evaluation quietly stops being saved. Nothing logs it. The tagging step is the one I would bet breaks first: the system prompt says "Return JSON array of matching topics," but the request uses response_format: { type: "json_object" }, which asks for an object, not an array. The code then guesses the key:
updates.tags = result.topics || result.tags || [];
That works only as long as the model happens to pick one of those two key names. That is model behavior holding my code together, which is exactly what changes in an upgrade.
The dropdown that goes nowhere
Now the part that made me title this post.
The agent settings tab has a model picker with four hardcoded options: GPT-4o mini, GPT-4o, Claude 3 Haiku and Claude 3.5 Sonnet. When you click publish, the component saves { language, model, voice, systemPrompt, firstMessage } to the agent's agentConfig in my database, then calls a sync route that pushes changes to Retell or Vapi.
I read the sync route. It sends the prompt, the first message and the voice. It never sends the model. So the dropdown is a database field with a label on it. You can pick Claude 3.5 Sonnet, see it saved, and your agent keeps running on whatever model it had in Retell.
It also does not read the real value. When there is no saved config, the tab loads the live agent from Retell, and the route returns this as the model:
model: engineType === "retell-llm" ? rEngine.llm_id || "gpt-4o-mini" : engineType,
llm_id is the ID of the Retell LLM object, not a model name. It will never match one of the four option values, so the select cannot show the agent's real model. The screen looks like it knows what model your agent uses. It does not.
And the option list itself is the deprecation problem in miniature: four model names typed into a component in March, with no way to update them except a code change.
What a forced migration would cost me today
Being honest with my own checklist:
| Question from my guide | Upwork Scout | VoiceDash |
|---|---|---|
| How many places call a model? | 2 files | 1 route, 3 calls |
| Model in server-side config? | Yes, one shared env var | No, hardcoded |
| Pinned version? | No, alias | No, alias |
| Output validated? | Schema plus clamping | Parse inside an empty catch |
| Test set? | No | No |
| Swap without a code change? | Yes | No |
None of the fixes below have shipped. They are my list, not a changelog:
-
VoiceDash: move the three analysis calls into one helper with the model read from config, log parse failures instead of swallowing them, and fix the tagging request so it asks for
{ "topics": [...] }explicitly. - VoiceDash: either make the model dropdown real, which means fetching the Retell LLM by ID and updating it there, or remove it. A setting that saves and does nothing is worse than no setting.
-
Upwork Scout: split
AI_MODELinto a matcher variable and an importer variable, and pin dated versions. - Both: a small replay set. For the matcher, thirty job and profile pairs with expected score bands. For VoiceDash, a handful of real transcripts with the summary and tags I would accept.
The pattern I keep relearning from these audits is that the gap is never in the code I wrote carefully. Upwork Scout's matcher was the feature I sat down and designed, so it got the config, the schema and the fallback. VoiceDash's analysis route was written fast to make a demo work, and nobody comes back to "working." A model retirement notice is just the thing that finds that code for you, on someone else's schedule.
The full checklist I measured myself against is on the Null Studio blog: AI model upgrades and deprecations.
How do you handle this in your own projects: one model setting per feature, or one per app?
Top comments (0)