Files, object storage and CDNs
Photos, PDFs, exports and app updates do not belong in your database or on your API server's disk. This lesson moves them to object storage, lets phones upload straight to it, and puts a CDN in front so users download from a server near them.
By the end you can
- Explain why files should not live in Postgres or on an app server
- Describe buckets, keys and metadata, and design a good object key
- Build a presigned upload and download flow with AWS SDK v3 in NestJS
- Explain CDN hits, misses, cache keys and TTLs
- Use versioned filenames so you almost never need to invalidate a CDN
The everyday picture
The distribution centre holds everything the chain sells. It is huge, cheap per square foot, and safe. But it is far from most customers. If every customer had to travel to the DC to buy a bottle of water, nobody would shop with you.
So neighbourhood stores keep copies of the popular items close to customers. When a store runs out of something, it reorders from the DC, and the next customer finds it on the shelf. In this lesson, object storage is the DC and a CDN is the network of neighbourhood stores. The store only needs to know two things: what to stock (the cache key) and how long the stock stays good (the TTL).
Where files should not live
Not in the database. You can store images in a bytea column, and it works on day one. Then the table grows by gigabytes, backups and restores take hours longer (Backups and disaster recovery), every file is copied again into the WAL and to each replica (Replication and read replicas), and a database connection stays busy while each file streams out. Database storage is also the most expensive storage you have.
Not on the API server's disk. A file saved on copy 1 does not exist on copies 2 and 3 (Scaling up vs scaling out). The disk fills up and takes the API down. A redeploy that replaces the container deletes the files. And every download uses the API server's network and memory, which you need for real requests.
The database should store a short reference to the file, such as its key and size, and nothing else.
Quick check
Your API saves uploads to its local disk, and you deploy a new container image. What happens to the files?
A container's own disk belongs to that container. Replace the container and its files go with it. A volume can keep them, but they still exist on one machine only, which breaks as soon as you run a second copy.