Choosing a model: open vs closed, capability vs cost

Mechanism: held-out workload → admissible models → lowest-cost route

Model choice is a recurring product decision, not a one-time pick. The frontier moves monthly and your tasks differ, so the question is never "what's the best model?" but "what's the cheapest model that clears this task's quality bar?"

The axes that matter

• Capability — does it pass your eval set on this task? (Benchmarks are a weak proxy; test on your data.)
• Cost — price per token, and how it scales with volume.
• Latency — time-to-first-token and tokens/sec; reasoning models are slower.
• Context window — does your context fit?
• Privacy / control — can data leave your boundary? Do you need on-prem?
• Reliability — rate limits, uptime, and the provider's deprecation cadence (migration risk).

Open vs closed (hosted)

This is the build-vs-buy decision in practice.

A practical strategy

• Start with a strong hosted model to validate the product, then optimize cost.
• Right-size (down) once it works: many tasks run fine on a smaller/cheaper model.
• Route — send easy requests to a cheap model, hard ones to a strong one (model routing / cascades).
• Decouple your code from any single model behind a thin interface, because you will switch.

Pitfall

Defaulting to the biggest model "to be safe" burns money and latency on tasks a small model handles — and chasing the newest model without re-running your evals ships silent regressions. Decide with evals, not vibes or leaderboards.

Connects to: build vs buy · right-sizing · eval on your task

models = [{"name":"small", "quality":.86, "cost":.01}, {"name":"large", "quality":.91, "cost":.04}]
print(min((m for m in models if m["quality"] >= .90), key=lambda m: m["cost"])["name"])

Run with python3; expected output is large. Add safety, latency, region, availability, and fallback gates before release.

Sources

• HELM — scenario-based model evaluation.
• NIST AI RMF — risk-aware selection.