Model deprecation & migration

A reality of building on someone else's model: it won't last forever. Providers deprecate model versions on a schedule, and even "the same" model can shift behavior after an update. If your prompts and product are tuned to one model, a migration can silently break things. Plan for it from day one.

Why this bites

• Hard deprecations — a provider sunsets a model version; your calls start failing on a deadline you must meet.
• Silent updates — a model alias (e.g. "latest") points to new weights; outputs drift even though your code didn't change (a distribution shift you didn't cause).
• Prompt brittleness — prompts and few-shot examples over-tuned to one model don't transfer cleanly.

Insulate the system up front

• Pin explicit versions, not floating aliases, so updates are a choice, not a surprise.
• Abstract the model behind a thin interface so swapping providers/models is a config change, not a rewrite.
• Keep prompts versioned in a registry and decoupled from app logic.
• Track the provider's deprecation calendar as an operational dependency.

Migrate with evals, not hope

A model swap is a behavior change — treat it like a deploy:

1. Run the new model against your eval set and diff against the current model (regression test).
2. Re-tune prompts for the new model where needed (don't assume they transfer).
3. Canary / shadow — run new alongside old on real traffic, compare, then ramp.
4. Keep a rollback path until the new model is proven.

Pitfall

Discovering a deprecation the week it takes effect, with prompts hand-tuned to the old model and no eval set to validate the replacement — so the migration is a scramble and ships regressions. The fix is boring and upfront: pin versions, version prompts, keep an eval set, and migrate behind a canary.

Connects to: prompt/model registry · regression testing · model selection