CAP, PACELC and consistency
As soon as your data lives on more than one machine, the copies can disagree. This lesson gives you the words and the rules for that: what a system can promise when the network breaks, what it costs on a normal day, and how to choose the right promise for each feature.
By the end you can
- Explain CAP in one sentence, and why "pick two of three" is the wrong way to say it
- Use PACELC to talk about the everyday cost of consistency: latency
- Tell strong, eventual, read-your-writes, monotonic and causal consistency apart with a retail example each
- Check a quorum setting with R + W > N
- Build read-your-writes on top of a Prisma replica setup with a short Redis marker
The everyday picture
Imagine a wholesaler with two offices, North and South, in two cities a few hundred kilometres apart. Both keep a copy of the same stock ledger, and every time one office sells something, a clerk phones the other office so both ledgers stay the same. Then a storm cuts the phone line for three hours.
A customer walks into the South office and wants the last carton of cooking oil. The clerk has two choices. He can sell it, and hope North did not sell the same carton in the meantime. Or he can say "Sorry, I can't confirm stock until the line is back." The first choice keeps the shop open and risks selling a carton twice. The second keeps the ledgers correct and sends a customer away. There is no third choice while the line is down. That is the CAP theorem, and the rest of this lesson is about making that choice on purpose.
When one machine becomes many
A distributed system is any system where parts run on different machines and talk over a network. If you have followed the earlier courses, you have already met several: a primary with read replicas (Replication and read replicas), a Redis cache in front of Postgres (Caching), workers reading a queue (Queues, background jobs and pub/sub). Each copy of the data can be a little behind the others.
The network between machines fails in ways a single machine never does. A cable is cut, a switch reboots, a cloud zone loses its uplink, a store's internet drops during a power cut. When some machines can still talk to each other but not to the rest, that is a network partition. Each side keeps running and keeps getting requests, but cannot see what the other side is doing. Partitions are not rare events you can design away. In a large system, some link is broken somewhere most days.
CAP: during a split, refuse or guess
The CAP theorem (Eric Brewer's idea from 2000, proved by Seth Gilbert and Nancy Lynch in 2002) talks about three properties:
- Consistency (C). Every read sees the latest write, as if there were only one copy of the data. This strict meaning is called linearizable.
- Availability (A). Every request to a working machine gets a real answer, not an error and not an endless wait.
- Partition tolerance (P). The system keeps working when the network between machines breaks.
People often say "pick two of the three". That is misleading. You cannot choose to have no partitions, because the network decides that for you. The useful version is: when a partition happens, each side must either refuse requests it cannot confirm (choose C) or answer with what it knows and accept that the sides may disagree (choose A). A system that refuses is called CP. A system that answers anyway is called AP. When the network is healthy, a well-built system gives you both.
Quick check
Why is "pick any two of C, A and P" a misleading way to say CAP?
The network decides when a partition happens, so P is not something you give up. During a split, each side either refuses or answers with what it has. Adding replicas does not help: the replicas are on both sides of the cut too.
Which side should a feature take?
Neither choice is "better". It depends on what a wrong answer costs compared with no answer. Selling the last carton twice means a phone call to an angry partner shop and a refund, so ordering against scarce stock should usually be CP: if the order service cannot confirm with the stock database, it says "try again in a minute". Showing a product photo that is ten minutes old costs nothing, so the catalogue should be AP: answer from the cache or a replica even if the primary is unreachable.
A point-of-sale counter during an internet outage is the most famous AP case in retail. The store keeps selling from its local copy, and the sales are merged with head office later. That only works because the business accepts that the stock count may be wrong for a while. Offline-first apps and data sync builds exactly that.
Quick check
During a network split, which of these features should refuse rather than answer from a local copy?
A wrong answer on scarce stock means overselling, refunds and angry customers, so it should refuse when it cannot confirm. Photos and yesterday's chart barely change, and a slightly old copy costs nothing, so they should keep answering.
Try both policies in the simulator. It has two nodes: North, which holds the leader, and South. Start in CP mode, press "Cut the network", and press "Sell 1 here" at North and then at South. Then switch to AP, sell a few units on both sides until the total passes the 5 in stock, and press "Heal the network".
- In CP mode, South (the side without the leader) gives errors for both sales and reads, while North keeps selling. Nothing goes wrong with the data, but customers on that side are turned away.
- In AP mode, both sides keep working and their numbers drift apart. If the total sold is more than the stock, healing the network cannot fix it. A person has to.
PACELC: the price you pay on a normal day
Partitions are only part of the story. Daniel Abadi pointed out in 2010 that CAP says nothing about the 99% of the time when the network works. His extension is called PACELC: if there is a Partition, choose A or C; Else, choose Latency or Consistency.
The "else" part is the one you feel every day. To keep copies in step, a write has to wait until other machines confirm it. Each confirmation is a network round trip: about half a millisecond inside one data centre, and tens of milliseconds or more to another city or region. If you don't wait, writes are fast, but a read from another copy may be old.
Postgres shows this clearly. With the default asynchronous replication, a commit returns as soon as the primary has saved it, and replicas catch up a few milliseconds or seconds later. That is "else latency". If you set synchronous_standby_names and synchronous_commit = remote_apply, every commit waits until a standby has applied it, so a read on that standby always sees it. That is "else consistency", paid for with a slower commit on every write. If that standby is unreachable, commits wait until it comes back or until you change the setting, which is CP behaviour during a partition.
So a system gets a two-part label. Cassandra and Riak are PA/EL by default: they stay up and answer fast, and you ask for stronger reads per query when you need them. etcd, which stores the configuration of Kubernetes clusters, is PC/EC: a majority must agree on every write and, by default, on every read, and the minority side refuses. A single Postgres primary on its own counts as PC/EC too: there is only one copy to read and write, and when it is unreachable, nobody can write.
Consistency is a ladder, not a switch
"Consistent" and "eventually consistent" are the two words you hear most, but there are useful steps between them. Each step is a promise about what a reader may see.
| Model | The promise | Retail example |
|---|---|---|
| Strong (linearizable) | Once a write is confirmed, every reader everywhere sees it | Reserving the last cartons for a partner shop order |
| Causal | If B happened because of A, nobody sees B without A | A "GRN corrected" note is never shown before the GRN it corrects |
| Read-your-writes | You always see your own changes; others may see them later | A manager saves a GRN and sees it in the list on the next screen |
| Monotonic reads | Once you have seen a value, you never see an older one | Refreshing the order list never makes a new order disappear |
| Eventual | If writes stop, all copies agree after a while; no promise about when | Product descriptions and images on the Partner app |
The middle three are called session guarantees, because they are about one user's experience. They are cheap compared with strong consistency, and they remove most of the bugs users actually notice.
Monotonic reads break in a way that surprises people. Say you have two replicas, one 1 second behind and one 5 seconds behind, and each request goes to a random one. A partner shop owner refreshes the order list and sees three orders, refreshes again and sees two, refreshes again and sees three. Nothing is lost, but it looks broken. The fix is to send each user to the same replica every time, for example by hashing the user id. One replica only ever moves forward, so that user's view only moves forward too.
Quick check
A customer refreshes the delivery screen and the order goes from "Delivered" back to "On the way". Which guarantee is missing?
The customer saw a newer value and then an older one, so their view went back in time: that is a monotonic-reads bug, usually from bouncing between replicas with different lag. Read-your-writes is about your own changes, and the customer did not write the status. Strong consistency would fix it too, but at a much higher price than pinning the user to one replica.
Think first: a replica is 2 seconds behind. Is that a problem for a sales dashboard? For the screen right after "Save GRN"?
For the dashboard, no. Nobody can tell whether a chart is 2 seconds old, and the replica keeps heavy queries off the primary. For the screen after saving, yes. The manager just pressed Save, and if the list does not show the new GRN, he assumes it failed and saves it again. That screen needs read-your-writes. The data and the lag are the same; the need is different, so the routing should be different.
Quorums: how many copies must agree?
Some databases (Cassandra, DynamoDB internally, Riak) have no single primary. Every key is stored on N machines, and you choose two numbers. W is how many must confirm a write before it counts as done. R is how many you ask on a read; you take the newest version among their answers. A group of machines that is large enough to decide is called a quorum.
The rule to remember is R + W > N. If it holds, the machines you read from and the machines that took the last write must share at least one machine, so at least one answer contains the newest value. The common choice is N = 3, W = 2, R = 2. It survives one machine being down for both reads and writes, and every read overlaps every write.
You can also read the fault tolerance straight from the numbers. A write needs W machines, so it still succeeds with N − W machines down. A read needs R, so it still succeeds with N − R down. With N = 3 and W = 2, one machine can be down for writes.
Quick check
A key is stored on N = 5 replicas and writes wait for W = 3. What is the smallest R that makes every read overlap the latest write?
You need R + W > N, so R + 3 > 5, which means R is at least 3. With R = 2 the total is exactly 5, and a read of two replicas can land entirely on the two that missed the write. R = 4 or 5 also overlap, but they make reads slower and fail sooner when replicas are down.
Turning the dials
Change W and R and you move along the PACELC line. W = 1, R = 1 is the fastest and works with two machines down, but a read can easily miss a write. W = 3, R = 1 makes reads fast and always fresh, but a write fails as soon as one machine is down. W = 2, R = 2 is the balanced middle. Cassandra exposes this per query as consistency levels such as ONE, QUORUM and ALL. DynamoDB reads are eventually consistent by default; a read with ConsistentRead: true is strongly consistent and uses twice the read capacity.
R + W > N is a strong rule of thumb, not a complete guarantee. A write that failed after reaching only one replica, two writes to the same key at the same moment, or a "sloppy" quorum that borrows other machines during a failure can still produce surprising reads. When you need a hard guarantee, such as "never sell more than we have", use one place that decides: a single Postgres primary with an atomic conditional update (Transactions, ACID, isolation and locks), or a system built on a consensus protocol.
For most teams on Postgres, the practical picture is simpler. Writes go to one primary, so writes are strongly consistent. If the primary is unreachable, nobody can write (C over A). Replicas keep answering reads with slightly old data (A over C for reads). Your job is to decide, read by read, which ones may be a little old.
Build it
See replica lag break read-your-writes on your laptop, then fix it with a Redis marker that sends a user's reads to the primary for a few seconds after they write. The steps need a local primary and replica; the quorum arithmetic from this lesson is an exercise you can run in the browser, in the Practice section below.
-
Make lag visible
Use the primary and replica you set up in Replication and read replicas. Pausing WAL replay on the replica freezes it in the past, which is a big, easy-to-see version of normal lag.
SQL-- On the REPLICA, as a superuser: stop applying changes SELECT pg_wal_replay_pause(); -- On the PRIMARY: save a GRN INSERT INTO grn (site_id, supplier_id) VALUES (12, 305) RETURNING id; -- On the REPLICA: the new row is not there yet SELECT id, site_id, received_at FROM grn ORDER BY id DESC LIMIT 3; -- On the REPLICA: catch up again SELECT pg_wal_replay_resume();Before you pick a window, look at real lag:
SELECT application_name, replay_lag FROM pg_stat_replication;on the primary. Choose a marker time comfortably above the worst lag you see on a normal busy day. -
Write a read router
The service sets a short-lived key after a user writes, and picks the primary for that user while the key exists. If Redis is down, it reads from the primary: when in doubt, choose correct over fast.
Dbis the extended Prisma client from Replication and read replicas;REDISis an ioredis provider.TypeScript// read-router.service.ts import { Inject, Injectable, Logger } from '@nestjs/common'; import Redis from 'ioredis'; import { DB, Db } from './db.module'; import { REDIS } from './redis.module'; export type Reader = Omit<Db, '$primary' | '$replica'>; const WINDOW_SECONDS = 10; // well above normal replay_lag @Injectable() export class ReadRouter { private readonly log = new Logger(ReadRouter.name); constructor(@Inject(DB) private readonly db: Db, @Inject(REDIS) private readonly redis: Redis) {} /** Call right after a write has committed. */ async markWrote(scope: string): Promise<void> { try { await this.redis.set(`ryw:${scope}`, '1', 'EX', WINDOW_SECONDS); } catch (e) { this.log.warn(`read-your-writes marker not set for ${scope}: ${e}`); } } /** Primary for anyone who wrote in the last few seconds, replica for everyone else. */ async reader(scope: string): Promise<Reader> { try { const wroteRecently = await this.redis.exists(`ryw:${scope}`); return wroteRecently ? this.db.$primary() : this.db; } catch { return this.db.$primary(); // Redis is down: correct beats fast } } } -
Use it in a service
Choose the scope with the product question "whose writes must this person see?". Here the whole store's team should see a goods received note (GRN) that any of them saved, so the marker is per site (in the data model, a
siteis a store or a distribution centre).TypeScript// grn.service.ts @Injectable() export class GrnService { constructor(@Inject(DB) private readonly db: Db, private readonly reads: ReadRouter) {} async create(siteId: bigint, supplierId: bigint) { // Writes always go to the primary const grn = await this.db.grn.create({ data: { siteId, supplierId } }); await this.reads.markWrote(`site:${siteId}`); return grn; } async recent(siteId: bigint) { const db = await this.reads.reader(`site:${siteId}`); return db.grn.findMany({ where: { siteId }, orderBy: { receivedAt: 'desc' }, take: 20 }); } } -
Test it with replay paused
Pause replay again, then call the API. The fresh GRN shows up because the list comes from the primary. After the window ends, the list comes from the frozen replica and the GRN disappears. That shows the fix works, and also its limit: the window must be longer than your real lag.
Terminalpsql "$REPLICA_ADMIN_URL" -c "SELECT pg_wal_replay_pause();" curl -s -X POST localhost:3000/sites/12/grns -H 'content-type: application/json' -d '{"supplierId":305}' curl -s localhost:3000/sites/12/grns | jq '.[0].id' # new id: read from the primary sleep 11 curl -s localhost:3000/sites/12/grns | jq '.[0].id' # older id: replica is still paused psql "$REPLICA_ADMIN_URL" -c "SELECT pg_wal_replay_resume();" -
Write the consistency table into the design doc
The code is the easy part. The decision behind it belongs in the design doc, one row per feature, so nobody has to guess later.
Markdown## Consistency decisions | Feature | Needs | Read from | If primary unreachable | |-------------------------------|------------------------|-----------------------------|---------------------------------| | Reserve stock for an order | Strong | Primary, conditional UPDATE | Refuse: "try again in a minute" | | "My orders" after placing one | Read-your-writes | Primary for 10 s, then replica | Show cached list + banner | | Order status for the customer | Monotonic reads | Same replica per user | Show last known status | | Product catalogue and photos | Eventual (minutes) | Cache, then replica | Serve from cache | | Zone sales dashboard | Eventual (a few min) | Replica or warehouse | Show "data as of 10:40" |
Bring it to your team
Use these when a design has replicas, caches, or more than one data centre.
- If this service cannot reach the database, what does the user see: an error, or an answer that may be old?
- Which reads in this feature must show the user's own change right away, and which can be a few seconds old?
- What is our replica lag on a busy day, at p99? Is our read-your-writes window longer than that?
- If two places can accept a change to the same thing during a split, how do we merge them, and who is told when the merge goes wrong?
- Are we paying for strong consistency (synchronous replication, quorum reads) on data where nobody would notice a few seconds of delay?
- Is the consistency choice for each feature written in the design doc?
Red flags
- "We picked C and A" or "the network is reliable inside our cloud".
- "It's eventually consistent", with no idea whether "eventually" means 50 ms or 50 minutes.
- A stock or payment decision made from a replica or a cache read.
- Every read forced to the primary "to be safe", so the replicas sit idle and the primary is overloaded.
Practice
Check a quorum setting
Write checkQuorum(n, w, r) for a store that keeps every key on n replicas, waits for w acknowledgements on a write and asks r replicas on a read. Return an object with three fields: overlap (true when every read group shares at least one replica with every write group), writeCanLose (how many replicas can be down while writes still succeed) and readCanLose (the same for reads). If w or r is less than 1 or more than n, throw an error, because that setting can never work.
function checkQuorum(n, w, r) {
// 1. Throw if w or r is outside 1..n
// 2. overlap: do read and write groups always share a replica?
// 3. How many replicas can be down for writes? For reads?
return { overlap: false, writeCanLose: 0, readCanLose: 0 };
}test('N=3, W=2, R=2 overlaps and survives one replica down', () => {
expect(checkQuorum(3, 2, 2)).toEqual({ overlap: true, writeCanLose: 1, readCanLose: 1 });
});
test('N=3, W=1, R=1 is fast but a read can miss the write', () => {
expect(checkQuorum(3, 1, 1)).toEqual({ overlap: false, writeCanLose: 2, readCanLose: 2 });
});
test('N=3, W=3, R=1 overlaps, but one replica down blocks writes', () => {
expect(checkQuorum(3, 3, 1)).toEqual({ overlap: true, writeCanLose: 0, readCanLose: 2 });
});
test('R + W equal to N is not enough (N=4, W=2, R=2)', () => {
expect(checkQuorum(4, 2, 2).overlap).toBe(false);
});
test('W larger than N throws', () => {
expect(() => checkQuorum(3, 4, 1)).toThrow();
});
Show a solution
function checkQuorum(n, w, r) {
if (w < 1 || w > n || r < 1 || r > n) {
throw new Error('w and r must be between 1 and n');
}
return {
overlap: r + w > n, // the two groups must share a replica
writeCanLose: n - w, // a write still finds w replicas
readCanLose: n - r, // a read still finds r replicas
};
}Write the consistency table for the Rider app
The Rider app has four features. For each one, write a row with the consistency it needs, where it reads from, and what it does if the primary database is unreachable. (1) A rider taps "Accept" on a new order; only one rider may get each order. (2) A rider opens "Today's earnings" right after finishing a delivery. (3) A customer watches the rider's dot move on a map. (4) The list of delivery zones, which changes a few times a year.
- Accepting an order is strong: one place decides, and the app refuses when it cannot confirm
- Today's earnings uses read-your-writes, so the delivery the rider just finished shows up
- The map can be eventual and a few seconds old, and shows the last known position when the data is unreachable
- The zone list is served from a cache or replica and keeps working without the primary
- Every row says what the user sees on a bad day, not only on a good one
Show a model answer
| Feature | Needs | Read from | If primary unreachable |
|----------------------|-------------------|-------------------------------|-------------------------------------|
| Accept an order | Strong | Primary, conditional UPDATE | Refuse: "Could not confirm, retry" |
| Today's earnings | Read-your-writes | Primary for 10 s, then replica | Show cached total + "as of 18:05" |
| Rider dot on the map | Eventual (secs) | Latest position cache | Keep last known dot, grey it out |
| Delivery zones | Eventual (days) | Cache, then replica | Serve from cache |The accept step is the only one where a wrong answer (two riders on one order) is worse than no answer. It uses UPDATE "order" SET rider_id = $1 WHERE id = $2 AND rider_id IS NULL on the primary and checks that one row changed.
Check yourself
A network cut separates the South node from the North leader. The system is CP. What does a customer at South get when buying the last carton?
CP means the side that cannot confirm refuses. The first option is the AP behaviour. The last option goes too far: the North side still has the leader and can keep selling safely, so a well-built CP system stays up on the majority side.
You store each key on N = 3 replicas. You want every read to see the latest confirmed write, and you want reads and writes to keep working when one replica is down. Which setting fits?
2 + 2 = 4 is more than 3, so reads and writes overlap, and each needs only 2 of the 3 replicas. W = 3, R = 1 also passes R + W > N, but every write fails when any replica is down. W = 1, R = 1 and W = 2, R = 1 do not overlap, so a read can miss the latest write.
A store manager saves a GRN, and the list on the next screen does not show it. Replica lag is about 2 seconds. What is the most targeted fix?
Only the people who just wrote need fresh reads, and only for a few seconds. Moving every read to the primary or making every commit wait for the replica would fix the bug at a high cost for everyone. A cache adds another stale copy and makes the bug worse.
Flashcards
Key takeaways
- Partitions will happen. CAP says that during one, each side must refuse (CP) or answer with what it knows (AP).
- PACELC adds the everyday cost: when the network is fine, stronger consistency means waiting for other copies, which adds latency to writes.
- Between strong and eventual sit cheap session guarantees, such as read-your-writes and monotonic reads, which fix most bugs users notice.
- With quorums, R + W > N makes reads and writes overlap; N = 3, W = 2, R = 2 also survives one machine down.
- Choose consistency per feature, write it in the design doc, and keep money and stock decisions on one place that decides.
Sources
- Please stop calling databases CP or AP, Martin Kleppmann, 2015
- Log-shipping standby servers and synchronous replication, PostgreSQL documentation
- DynamoDB read consistency, AWS documentation
- Designing Data-Intensive Applications, Martin Kleppmann, O'Reilly, chapters 5 and 9