ChatGPT: serving a large language model to hundreds of millions
In October 2025, OpenAI's CEO said more than 800 million people use ChatGPT every week, and its API processes over 6 billion tokens a minute. Most of ChatGPT's internal design is private, but OpenAI has published real pieces: its PostgreSQL setup, its Kubernetes clusters, its outages, and how its engineers think about GPUs. This lesson puts those pieces together with how LLM serving works in general, and then you build a streaming chat proxy you can put in front of any model.
By the end you can
- Trace a message from the send button to a GPU and back as a stream of tokens
- Explain prefill, decode, the KV cache and continuous batching in plain words
- Explain why GPU capacity cannot autoscale, and what rate limits and queues do about it
- Read OpenAI's published PostgreSQL design like a DBA, and know which parts to copy
- Build a NestJS proxy with per-user limits, a bounded queue, SSE streaming and saved conversations
The everyday picture
Think of a busy flatbread oven at a kebab house on a Friday night, the kind you find from Dhaka to Istanbul to Lagos. The oven is the expensive part: there is one, it has room for a fixed number of flatbreads, and you cannot buy a second oven in the next ten minutes. When an order arrives, the cook first reads the whole order and prepares all the dough at once. That part is quick. Then the flatbreads come out one at a time, each one only after the one before. The waiter carries each one to the table as soon as it is ready, so the table starts eating early.
A clumsy cook fills the oven for four tables and waits until the slowest table's order is finished before taking new ones. A good cook puts a new flatbread into any gap the moment one comes out. Each table also needs space on the counter for its half-finished order, and the counter is small. When the counter is full, new customers wait at the door, or the manager says "we are full tonight". Serving ChatGPT is this kitchen: the oven is the GPU (the graphics chip that runs the model), the flatbreads are tokens, the counter is GPU memory, and the manager at the door is the rate limiter and queue.
What the product must do
- Functional. Sign in, type or speak a message, get an answer that appears word by word, keep the conversation history, attach files and images, and choose between models. Paid plans get more use than free ones. Developers reach the same models through an API with keys and usage limits.
- Non-functional. The first words should appear within a second or two. The answer must keep flowing without gaps. Capacity must be shared fairly between free users, paying users and API customers. Conversations must never leak between users. And because every answer runs on very expensive hardware, the cost of each token matters.
The unusual requirement is the last one. In most web systems, a request costs a few milliseconds of CPU. Here, one answer can keep a slice of a GPU busy for many seconds, and there is a fixed number of GPUs.