Model deprecation & migration
A reality of building on someone else's model: it won't last forever. Providers deprecate model versions on a schedule, and even "the same" model can shift behavior after an update. If your prompts and product are tuned to one model, a migration can silently break things. Plan for it from day one.
Why this bites
- Hard deprecations — a provider sunsets a model version; your calls start failing on a deadline you must meet.
- Silent updates — a model alias (e.g. "latest") points to new weights; outputs drift even though your code didn't change (a distribution shift you didn't cause).
- Prompt brittleness — prompts and few-shot examples over-tuned to one model don't transfer cleanly.
Insulate the system up front
- Pin explicit versions, not floating aliases, so updates are a choice, not a surprise.
- Abstract the model behind a thin interface so swapping providers/models is a config change, not a rewrite.
- Keep prompts versioned in a registry and decoupled from app logic.
- Track the provider's deprecation calendar as an operational dependency.
Migrate with evals, not hope
A model swap is a behavior change — treat it like a deploy:
- Run the new model against your eval set and diff against the current model (regression test).
- Re-tune prompts for the new model where needed (don't assume they transfer).
- Canary / shadow — run new alongside old on real traffic, compare, then ramp.
- Keep a rollback path until the new model is proven.
Pitfall
Discovering a deprecation the week it takes effect, with prompts hand-tuned to the old model and no eval set to validate the replacement — so the migration is a scramble and ships regressions. The fix is boring and upfront: pin versions, version prompts, keep an eval set, and migrate behind a canary.
Connects to: prompt/model registry · regression testing · model selection