Claude: how Anthropic serves Claude at scale
In 2025 Anthropic said it serves Claude to millions of users on three kinds of chips, from three cloud partners, and to more than 300,000 business customers. Anthropic publishes little about its internal systems, but its API documentation and one detailed postmortem show a lot about how a model service behaves from the outside, and what went wrong inside. This lesson reads those sources carefully, separates them from general knowledge, and ends with a resilient client you can use with any LLM API.
By the end you can
- Say what Anthropic has published about its serving setup, and what it has not
- Explain how one model runs on different hardware, and why that makes quality hard to check
- Explain prompt caching: what is reused, why it cuts latency and cost, and how to structure prompts for it
- Use the documented limits, errors, streaming events and the Batch API correctly
- Build a streaming client with a time budget, safe retries, and cache-friendly prompts
The everyday picture
Imagine a restaurant chain whose famous rice dish, a biryani or a paella, is cooked in three kinds of kitchens: one with gas burners, one with electric ovens, one with wood fire. The recipe is the same, and head office promises the same taste in every branch. But each kitchen needs its own small adjustments: different heat, a different pot, a slightly different timing. If one kitchen's thermometer reads two degrees wrong, its rice is a little off, but only some days, only in some branches, and only for some dishes. Customers say "it tasted different last week" and head office cannot easily tell which kitchen, which day, or why.
Now picture the counter. Many customers order the same base, the plain rice, before choosing the topping. A smart counter keeps a pot of the base ready, so each order only adds its own topping. That is fast and cheap, but only if the base is exactly the same: one extra pinch of salt and it is a different pot. Anthropic's serving story has both parts: one model on several kinds of hardware, and reusing the identical start of a prompt.
What the product must do
- Functional. Answer chat and API requests with several models, stream answers, accept long inputs (up to 1 million tokens for current models), use tools, run large offline jobs, and reuse repeated context cheaply. Offer the same models on Anthropic's own API and on partner clouds (in 2025 these were Amazon Bedrock and Google Cloud's Vertex AI; the 2026 model docs also list Microsoft Foundry and Claude Platform on AWS).
- Non-functional. Same quality on every platform and every chip. Fair sharing of limited capacity between customers. Clear errors that tell a client when to wait and when to stop. Privacy rules strict enough that engineers cannot freely read user conversations, which, as you will see, makes debugging harder.