Build vs buy: provider API vs self-hosting
One of the highest-stakes infra decisions: consume a hosted API (OpenAI, Anthropic, …) or self-host an open-weights model on your own GPUs. There's no universal answer — it's a function of volume, privacy, control, and the engineering you can sustain.
What you're really trading
| Hosted API (buy) | Self-host (build) | |
|---|---|---|
| Time-to-first-value | minutes | days–weeks (set up serving) |
| Capability | frontier models | strong open models, slightly behind |
| Cost shape | per-token (opex, scales with use) | fixed GPU cost (capex-ish), cheaper at high volume |
| Privacy | data leaves your boundary | data stays in-house / on-prem |
| Control | provider's roadmap & deprecations | you own the model & version |
| Ops burden | provider handles it | you own uptime, scaling, GPUs |
The cost crossover
APIs win at low and spiky volume — you pay only for what you use and skip GPU ops. Self-hosting wins at high, steady volume, where a saturated GPU is cheaper per token than API pricing. The crossover depends on utilization: an idle GPU is pure waste, so self-hosting only pays if you keep it busy. Model it with real traffic (cost modeling), not a napkin.
When each clearly wins
- Buy (API): early stage, unpredictable load, need frontier capability, small team, no strict data-residency rule.
- Build (self-host): strict privacy/compliance, very high steady volume, need deep customization, or want insulation from provider changes.
- Hybrid is common: API for hard/rare calls, a cheap self-hosted model for the high-volume easy ones (a routing strategy).
Pitfall
Underestimating the operational cost of self-hosting — GPU supply, autoscaling, batching, KV-cache memory, upgrades, on-call. The model is the easy part; running it reliably at scale is a real platform. And don't self-host for "privacy" if a hosted provider's zero-retention enterprise tier already meets your requirement.
Connects to: choosing a model · cost modeling · serving