1. Courses
  2. System Design
  3. Databases at Scale
  4. Stop 1

35 minIntermediate

Partitioning big tables

When one table holds years of sales, every query, vacuum and clean-up gets slower. Partitioning splits that table into smaller tables behind one name, so Postgres reads only the months a query needs and you can remove an old month in about a second.

By the end you can

  • Explain how partitioning differs from an index, in two sentences
  • Say exactly how Postgres knows which partition a new row or a query belongs to
  • Write the SQL for monthly range partitions on sale, with a default partition and an automatic job for new months
  • Retire an old month with DETACH and DROP instead of a huge DELETE
  • Spot the common traps: the key in the primary key, queries without the key, too many partitions

The everyday picture

Think of a 30-volume encyclopedia in a school library. Volume 1 covers A to B, volume 2 covers C to D, and so on. The spine of each book shows its range. If you want "Rice", you do not open thirty books. You read the spines, take the volume for R, and ignore the rest.

Inside that volume there is an index at the back. It tells you the page number for "Rice". So the encyclopedia uses two tools at once. The split into volumes tells you which book. The index tells you which page inside that book.

Now the publisher prints a new edition of volume 1. The librarian takes the old volume 1 off the shelf and puts the new one in. Nobody erases the old pages one by one. Postgres partitioning gives you the same three things: the split, an index inside each piece, and a quick way to remove a whole piece.

Partitioning and indexing are different tools

A common first question is "Isn't partitioning the same as indexing?" Here is the short answer. An index is a separate sorted structure that points to rows, while the rows stay in one big table (Indexes from the inside). Partitioning moves the rows themselves into several smaller tables, and each of those tables gets its own indexes.

"If a table is split into three, does that mean three tables?" Yes. In Postgres each is a real table with its own file on disk. You also get one more object, the parent table. The parent stores zero rows. It holds the rule, called the , plus the list of partitions and the value range each one accepts. So "split into three" means four entries in the catalogue: one parent and three partitions.

"How does the API or query know which table to use?" It does not need to know. Your NestJS code, Prisma and the reporting tool always use the parent name, sale. Postgres does the rest in two places:

  • On INSERT, Postgres reads the row's sale_date, compares it with the stored ranges, and writes the row into the matching partition. This is called . If an UPDATE changes sale_date to another month, Postgres moves the row for you (since PG 11).
  • On SELECT, the planner compares your WHERE clause with the same ranges and drops every partition that cannot contain a match. This is . A query for September reads only the September table.

The two tools work together. Pruning picks the partition, and that partition's own B-tree index finds the rows inside it.

IndexPartitioning
What it isA sorted list of keys pointing to rowsThe rows split into separate tables by a rule
Where rows liveOne big tableMany smaller tables behind one parent
Speeds upFinding a few rows by a columnQueries that filter on the key; maintenance per piece
Removing old dataNo help; DELETE still touches every rowDrop a whole month in about a second
CostSlower writes, disk spaceMore objects to manage, rules on keys

Quick check

You partition sale into 12 monthly partitions. What does Postgres now hold?

NestJS API INSERT INTO sale sale_date = 14 Sep sale parent table stores 0 rows routes rows by sale_date sale_2026_08 Aug 2026 sale_2026_09 Sep 2026 sale_2026_10 Oct 2026 sale_default any other date Aug Sep Oct other
The app only knows the name sale. The parent holds the rules, not the rows. A sale dated 14 September is written into sale_2026_09. A date with no matching month lands in the default partition.

Pruning: how a query skips most of the table

Each partition stores its range in the catalogue, for example "from 2026-09-01 up to, but not including, 2026-10-01". When a query arrives, the planner checks every range against the WHERE clause. A filter like sale_date >= '2026-09-01' AND sale_date < '2026-10-01' overlaps one range, so only one partition stays in the plan.

Pruning can happen at two moments. If the values are written in the SQL, the planner prunes while it builds the plan. If the values arrive later, as parameters (Prisma always sends $1, $2) or from now(), Postgres prunes when the query runs. In EXPLAIN that second kind shows up as a line like Subplans Removed: 4.

Pruning compares the key column itself with the ranges. If you wrap the key in a function, such as date_trunc('month', sale_date) or extract(month FROM sale_date), the planner can no longer match it against the ranges, and it reads every partition. The rows that come back are the same, so nobody notices until the table is big. Write date filters as a half-open range on the raw column: >= the first day, < the first day of the next period.

The catch is in the word "key". Pruning only works for filters on the partition key. A query with WHERE site_id = 42 and no date has to visit every partition. Each partition's own index still helps, so it is many small index lookups instead of one, but the planner has more work and you lose most of the benefit.

Quick check

sale is partitioned by month on sale_date. Which filter lets the planner read only September's partition?

Filter on the partition key WHERE sale_date >= '2026-09-01' AND sale_date < '2026-10-01' only Sep May pruned Jun pruned Jul pruned Aug pruned Sep read Oct pruned Filter on another column WHERE site_id = 42 May read Jun read Jul read Aug read Sep read Oct read
Top: the date filter overlaps only September's range, so the other five partitions are pruned and never opened. Bottom: site_id is not the partition key, so Postgres must look inside every partition.

Try it yourself. Pick a month range and watch how many partitions are read and how many rows the planner has to consider. Then switch the filter to "By site_id only". At the bottom, compare deleting a year of old data with dropping whole partitions.

Simulator

Loading the simulator…

  • A two-month date filter reads 2 of 12 partitions. The other ten cost nothing.
  • The delete comparison is the bigger win in real life: dropping a partition writes almost nothing, while a DELETE touches every row.
Think first: the Store app has a "find receipt by number" screen that sends WHERE id = 98765. What happens on a table partitioned by month?

There is no date in the filter, so nothing can be pruned. Postgres checks the primary key index of every partition. With 36 monthly partitions that is 36 small index lookups instead of one. Each lookup is fast, so the screen may still feel fine, but the cost grows with every month you keep. The cleaner fix is a product one: the printed receipt already shows its date, so let the screen send the date too. Then the query reads one partition.

Range, list and hash

Postgres offers three kinds of rule. Pick the one that matches how the data is read and how it is removed.

  • Range: each partition takes a span of values, usually time. Use it for sales, goods received note (GRN) lines, handheld scanner events and logs. Almost every query has a date filter, and old data leaves by month. This is the common case.
  • List: each partition takes named values, for example FOR VALUES IN ('north', 'north-east'). Useful when queries always target one region or one business unit. Watch out for skew: if 60% of your stores are in the capital city, that partition stays huge.
  • Hash: Postgres hashes the key and spreads rows evenly, for example FOR VALUES WITH (MODULUS 8, REMAINDER 3). Use it when there is no natural range and you want smaller, even pieces for vacuum and index size. It does not help you remove old data.

Partition ranges do not need equal sizes. A festival month (Eid, Diwali or Christmas, depending on where you are) can be twice the size of a quiet month, and that is fine.

Quick check

The Rider app stores GPS pings. Every query asks for a time window, and pings are deleted after 30 days. Which rule fits?

The real win: removing old data in one second

Say you keep two years of sales and the oldest month holds about 50 million rows. A DELETE ... WHERE sale_date < ... visits all 50 million rows. It writes a change record for each one to the , so replicas and backups have to process all of it. It leaves 50 million dead rows that vacuum must clean later (Running Postgres in production), and the indexes stay bloated. It can run for hours.

With partitions you remove the month as a whole table. DETACH PARTITION unhooks it from the parent so queries no longer see it. Then you can archive it with pg_dump and DROP TABLE it. Dropping deletes the table's files. It writes almost no WAL and leaves nothing for vacuum.

Locks matter here, as they did in Changing the schema without downtime. A plain DETACH PARTITION takes an lock on the parent for a moment. It is quick, but it waits behind any long query and blocks everyone behind it. Since PG 14 you can write DETACH PARTITION ... CONCURRENTLY, which takes a lighter lock that lets reads and writes continue. It has two rules: it cannot run inside a transaction block, and it refuses to run while the table has a default partition.

Adding a partition has a lock too. CREATE TABLE ... PARTITION OF sale takes ACCESS EXCLUSIVE on sale. So create future months ahead of time, during quiet hours, with a lock_timeout. Never create them at midnight on the first of the month while the POS is busy. If even that short lock hurts, create the month as a normal table with a matching CHECK constraint and then ATTACH PARTITION it. ATTACH takes a lighter lock on the parent.

The traps

  • The primary key must include the partition key. Each partition checks uniqueness only inside itself, so Postgres insists that unique constraints contain sale_date. The key becomes (id, sale_date). Any table with a foreign key to sale, such as sale_line, must store both columns.
  • Queries without the key read every partition. Check your hot screens and API calls. If they do not filter on the key, partitioning will not speed them up.
  • Too many partitions slow planning. The PostgreSQL docs say the planner handles up to a few thousand partitions fairly well when queries prune most of them. Monthly partitions for five years are 60 tables, which is easy. Daily partitions for five years are over 1,800, so measure before you choose that.
  • The default partition can trap you. It catches rows with no matching month, so inserts never fail. But if it holds even one row for November, creating sale_2026_11 later fails with an error. Adding any partition also scans the default under a strong lock. Keep it empty and alert when it is not.
  • Partitioning is not sharding. All partitions live on the same server, with the same CPU, RAM and disk. It makes big tables manageable. It does not give you more write capacity. That is Sharding and consistent hashing.

Build it

We will partition sale by month, watch Postgres route and prune, automate new months, retire an old month, and map it in Prisma. Run this on a test database with PostgreSQL 16 or newer. Two of the steps, routing rows into partitions and making a report prune, are also in the Practice section below, where they run in your browser.

  1. Create the partitioned parent

    The parent declares the columns, the rule and the primary key. It will never hold rows itself.

    SQL
    CREATE TABLE sale (
      id           bigserial,                       -- what Prisma generates for autoincrement()
      site_id      bigint         NOT NULL,
      sale_date    date           NOT NULL,         -- the partition key
      sold_at      timestamptz(6) NOT NULL DEFAULT now(),
      total        numeric(12,2)  NOT NULL,
      PRIMARY KEY (id, sale_date)                   -- must include the partition key
    ) PARTITION BY RANGE (sale_date);
    
    -- Without sale_date in the key, Postgres refuses:
    -- ERROR: unique constraint on partitioned table must include all partitioning columns
  2. Create monthly partitions, a default and one index

    The lower bound is included and the upper bound is not, so months never overlap. An index created on the parent is created on every partition, now and in the future.

    SQL
    CREATE TABLE sale_2026_08 PARTITION OF sale FOR VALUES FROM ('2026-08-01') TO ('2026-09-01');
    CREATE TABLE sale_2026_09 PARTITION OF sale FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
    CREATE TABLE sale_2026_10 PARTITION OF sale FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
    CREATE TABLE sale_default PARTITION OF sale DEFAULT;   -- catches any other date
    
    CREATE INDEX sale_site_id_sale_date_idx ON sale (site_id, sale_date);

    On a big table that already has partitions, CREATE INDEX CONCURRENTLY on the parent is not allowed. Create the parent index with ON ONLY sale, build each partition's index concurrently, then attach each one with ALTER INDEX ... ATTACH PARTITION.

  3. Load test rows and see where they went

    tableoid::regclass tells you which physical table each row lives in.

    SQL
    INSERT INTO sale (site_id, sale_date, total)
    SELECT (g % 400) + 1, date '2026-08-01' + (g % 92), (g % 900) + 50
    FROM generate_series(1, 300000) AS g;
    ANALYZE sale;
    
    SELECT tableoid::regclass AS partition, count(*) AS rows
    FROM sale GROUP BY 1 ORDER BY 1;
    Output
      partition   |  rows
    --------------+--------
     sale_2026_08 | 101090
     sale_2026_09 |  97830
     sale_2026_10 | 101080
  4. Prove pruning with EXPLAIN

    A date filter reads one partition. A site_id filter alone reads all of them, each through its own index.

    SQL
    EXPLAIN (COSTS OFF)
    SELECT site_id, sum(total) FROM sale
    WHERE sale_date >= '2026-09-01' AND sale_date < '2026-10-01'
    GROUP BY site_id;
    
    EXPLAIN (COSTS OFF)
    SELECT sum(total) FROM sale WHERE site_id = 42;
    Output
     HashAggregate
       Group Key: sale.site_id
       ->  Seq Scan on sale_2026_09 sale          <- only September
             Filter: ((sale_date >= '2026-09-01'::date) AND (sale_date < '2026-10-01'::date))
    
     Aggregate
       ->  Append                                <- every partition
             ->  Bitmap Heap Scan on sale_2026_08 sale_1
                   ->  Bitmap Index Scan on sale_2026_08_site_id_sale_date_idx
             ->  Bitmap Heap Scan on sale_2026_09 sale_2
                   ->  Bitmap Index Scan on sale_2026_09_site_id_sale_date_idx
             ->  Bitmap Heap Scan on sale_2026_10 sale_3
                   ->  Bitmap Index Scan on sale_2026_10_site_id_sale_date_idx
             ->  Seq Scan on sale_default sale_4
  5. Create future months automatically

    Create partitions a few months ahead so inserts never hit a missing month. Use a small function run by a daily job, or the pg_partman extension.

    SQL
    CREATE OR REPLACE FUNCTION ensure_sale_partition(p_month date)
    RETURNS text LANGUAGE plpgsql
    SET lock_timeout = '5s'      -- fail fast instead of queueing behind a long query
    AS $$
    DECLARE
      v_from date := date_trunc('month', p_month)::date;
      v_to   date := (date_trunc('month', p_month) + interval '1 month')::date;
      v_name text := 'sale_' || to_char(v_from, 'YYYY_MM');
    BEGIN
      IF to_regclass(v_name) IS NULL THEN
        EXECUTE format('CREATE TABLE %I PARTITION OF sale FOR VALUES FROM (%L) TO (%L)',
                       v_name, v_from, v_to);
        RETURN 'created ' || v_name;
      END IF;
      RETURN 'exists ' || v_name;
    END $$;
    
    -- Keep this month and the next 3 ready
    SELECT ensure_sale_partition((date_trunc('month', current_date)
           + make_interval(months => m))::date)
    FROM generate_series(0, 3) AS m;
    
    -- Run it every night at 03:15 with the pg_cron extension
    SELECT cron.schedule('sale-partitions', '15 3 * * *',
      $$SELECT ensure_sale_partition((date_trunc('month', current_date)
        + make_interval(months => m))::date) FROM generate_series(0, 3) AS m$$);
  6. Retire an old month

    In this test, treat sale_2026_08 as the month that has aged out. Because we kept a default partition, the CONCURRENTLY form is refused, so we use a plain DETACH with a short lock timeout.

    SQL
    -- With a default partition, CONCURRENTLY is refused:
    -- ERROR: cannot detach partitions concurrently when a default partition exists
    SET lock_timeout = '3s';                      -- give up rather than block the POS
    ALTER TABLE sale DETACH PARTITION sale_2026_08;
    
    -- Without a default partition (PG 14+), use the lighter lock instead.
    -- Run it on its own, never inside BEGIN ... COMMIT:
    -- ALTER TABLE sale DETACH PARTITION sale_2026_08 CONCURRENTLY;
    -- If that was interrupted, finish it with:
    -- ALTER TABLE sale DETACH PARTITION sale_2026_08 FINALIZE;
    Terminal
    # Archive the detached month, check the file, then drop it
    pg_dump -Fc -t sale_2026_08 -d mangrove -f sale_2026_08.dump
    pg_restore -l sale_2026_08.dump | grep "TABLE DATA"
    psql -d mangrove -c "DROP TABLE sale_2026_08;"
  7. Map it in Prisma

    Prisma has no syntax for partitioning (as of Prisma 6). The model maps to the parent table, and the partitioning lives in raw SQL inside the migration. Generate the migration with npx prisma migrate dev --create-only --name partition_sale, add PARTITION BY RANGE ("sale_date") to the generated CREATE TABLE, add the partitions below it, then run npx prisma migrate dev.

    Prisma
    model Sale {
      id          BigInt   @default(autoincrement())
      siteId      BigInt   @map("site_id")
      saleDate    DateTime @map("sale_date") @db.Date
      soldAt      DateTime @default(now()) @map("sold_at") @db.Timestamptz(6)
      total       Decimal  @db.Decimal(12, 2)
    
      @@id([id, saleDate])           // composite key, includes the partition key
      @@index([siteId, saleDate])
      @@map("sale")
    }
    TypeScript
    // Always pass the date range so Postgres can prune
    const rows = await prisma.sale.findMany({
      where: {
        siteId: 42,
        saleDate: { gte: new Date('2026-09-01'), lt: new Date('2026-10-01') },
      },
    });
    // Lookups by id now need both parts of the key
    const one = await prisma.sale.findUnique({
      where: { id_saleDate: { id: 98765n, saleDate: new Date('2026-09-14') } },
    });

    Since Prisma 4.10, introspection ignores the child partitions and shows only the parent as a model, so migrate dev should not try to drop them. Old versions did, so test once: run migrate dev against a copy and read the SQL it generates before you trust it. Create future months with the job from step 5, not with Prisma migrations.

Bring it to your team

Use these when someone proposes partitioning, or when a big table keeps causing pain.

  • "Which column do almost all queries on this table filter by? Is that our partition key?"
  • "How big is the table today, how many rows a day does it grow, and how long must we keep them?"
  • "What creates next month's partition, how far ahead, and what alerts us if the job did not run?"
  • "How do we remove old data: by dropping partitions, or are we still planning a DELETE job?"
  • "Do we have a default partition? Who gets alerted if a row lands in it?"
  • "Does every report screen send a date range, and what is its default?"

Red flags

  • A nightly DELETE FROM sale WHERE sale_date < ... on a table with hundreds of millions of rows.
  • "Let's partition it to make it fast" for a table of a few GB that is missing an index.
  • Daily partitions with years of retention and nobody has measured planning time.
  • Hot API calls on the partitioned table that never include the partition key.
  • "Partitioning will let one database take more writes." That needs sharding or a bigger server.

Practice

Route rows into monthly partitions

The parent table sale and its August 2026 partition already exist, with two rows in August. Create the September and October 2026 partitions and a default partition named sale_default. Keep the second part of the starter at the end: it inserts three sales and shows which partition each one landed in. Remember that the upper bound of a range is not included.

SQL exerciseRuns in a real PostgreSQL inside your browser
-- 1. Create sale_2026_09, sale_2026_10 and sale_default here.


-- 2. Keep this part: it inserts three sales and shows where they went.
INSERT INTO sale VALUES
  (3, 3, '2026-09-14', 1250),
  (4, 3, '2026-10-02', 800),
  (5, 5, '2026-12-25', 4000);
SELECT tableoid::regclass::text AS partition, count(*) AS rows
FROM sale GROUP BY 1 ORDER BY 1;
Ctrl + Enter runs
Show a solution
SQL
CREATE TABLE sale_2026_09 PARTITION OF sale FOR VALUES FROM ('2026-09-01') TO ('2026-10-01');
CREATE TABLE sale_2026_10 PARTITION OF sale FOR VALUES FROM ('2026-10-01') TO ('2026-11-01');
CREATE TABLE sale_default PARTITION OF sale DEFAULT;

INSERT INTO sale VALUES
  (3, 3, '2026-09-14', 1250),
  (4, 3, '2026-10-02', 800),
  (5, 5, '2026-12-25', 4000);
SELECT tableoid::regclass::text AS partition, count(*) AS rows
FROM sale GROUP BY 1 ORDER BY 1;
Pass the exercise to finish this task.
Why it works

Each PARTITION OF table takes the range from its lower bound up to, but not including, its upper bound, so the months meet without overlapping. The Christmas sale has no month of its own and lands in sale_default. Without the default, that INSERT would fail with "no partition of relation sale found for row". In a real system you would alert on that row and create December ahead of time.

Make a store report prune

Ama's store is site 3. Her September 2026 report reads every partition, because its filter wraps sale_date in a function. The setup has a helper, check_report(query), that runs EXPLAIN on your query, lists the partitions the plan reads, and also runs the query to show its result. Rewrite only the WHERE clause so the plan reads September's partition alone. The revenue must stay the same, so watch the last day of the month.

SQL exerciseRuns in a real PostgreSQL inside your browser
-- Change only the WHERE clause inside $q$ ... $q$
SELECT * FROM check_report($q$
  SELECT sum(total_cents) FROM sale
  WHERE site_id = 3
    AND date_trunc('month', sale_date) = '2026-09-01'
$q$);
Ctrl + Enter runs
Show a solution
SQL
SELECT * FROM check_report($q$
  SELECT sum(total_cents) FROM sale
  WHERE site_id = 3
    AND sale_date >= '2026-09-01'
    AND sale_date <  '2026-10-01'
$q$);
Pass the exercise to finish this task.
Why it works

The planner can compare a plain range on sale_date with each partition's bounds, so August, October and the default are pruned. The function version returns the same 4,150 cents but reads all four partitions. A common slip is BETWEEN '2026-09-01' AND '2026-10-01': it includes 1 October, reads two partitions and adds 1,800 cents that belong to October. The half-open range avoids that trap, and it also works when the column is a timestamp.

Check yourself

Table sale is partitioned by month on sale_date. Which table name should your NestJS service use in prisma.sale.findMany(...) for September?

You must remove all sales older than 25 months from a 900 million row table partitioned by month. What is the best approach?

Why does CREATE TABLE sale (id bigserial PRIMARY KEY, sale_date date NOT NULL) PARTITION BY RANGE (sale_date) fail?

Flashcards

Open my review

Key takeaways

  • An index finds rows inside one table. Partitioning splits the rows into several real tables behind one parent name. They work together.
  • The app always uses the parent. Postgres routes inserts by the partition key and prunes partitions that cannot match the WHERE clause.
  • Pruning needs a plain filter on the key, with no function around it. Make date ranges part of every report screen.
  • The biggest win is maintenance: retire a month with DETACH and DROP instead of a giant DELETE.
  • Remember the rules: the key goes into every unique constraint, create partitions ahead with a lock timeout, keep the default partition empty, and keep the partition count modest.
Episode transcriptThirty volumes on one shelf · about 14 minutes · tap any line to play from there
  1. NadiaImran, last week you asked me a question I promised to answer properly today. Can you ask it again?
  2. ImranIsn't partitioning the same as indexing? Both are supposed to make big tables fast, so I've never understood the difference.
  3. NadiaIt's a good question, and the short answer is no. They're different tools that work together. By the end of this episode you'll be able to explain the difference in two sentences.
  4. NadiaLet's start in a school library, with a thirty-volume encyclopedia. Volume one covers A to B, volume two covers C to D, and so on. Each spine shows its range.
  5. ImranSo if I want "Rice", I read the spines, take the volume for R, and ignore the other twenty-nine.
  6. NadiaRight. And inside that volume, there's an index at the back that gives you the page for "Rice". So the encyclopedia uses two tools at once. The split into volumes tells you which book. The index tells you which page inside it.
  7. ImranAnd partitioning is the split into volumes.
  8. NadiaExactly. One more part of the story. The publisher prints a new edition of volume one. The librarian takes the old volume off the shelf and puts the new one in. Nobody erases the old pages one by one.
  9. NadiaThat's the quick way to remove a whole piece. Postgres partitioning gives you all three: the split, an index inside each piece, and fast removal. Now your two sentences.
  10. NadiaAn index is a separate sorted list that points to rows, while the rows stay in one big table. Partitioning moves the rows themselves into several smaller tables, and each of those gets its own indexes.
  11. ImranSo if a table is split into three, there really are three tables?
  12. NadiaThree real tables, each with its own file on disk. Plus one more object, the parent table. The parent stores zero rows. It holds the rule, called the partition key, and the list of partitions with the range each one accepts.
  13. ImranSo "split into three" means four entries. One parent, three partitions.
  14. NadiaRight. And for sales, the key is usually the sale date, with one partition per month.
  15. ImranThen here's my practical worry. Our NestJS code and Prisma talk to a table called sale. Now there's sale for August, sale for September. Does every query need to pick the right month's table?
  16. NadiaThat's the natural guess, and it's wrong in a nice way. The app doesn't need to know. Your code, Prisma and the reporting tool always use the parent name, sale. Postgres does the rest.
  17. NadiaIf you can see your screen, the first diagram shows how inserts work. At the top left, the NestJS API sends one insert into sale, dated fourteenth September. Below it sits the parent, which says "stores zero rows". On the right, four boxes: August, September, October, and a default.
  18. ImranAnd the row goes into the September box.
  19. NadiaPostgres reads the row's sale date, compares it with the stored ranges, and writes it into the matching partition. That's called tuple routing. A date with no matching month lands in the default partition.
  20. ImranAnd reading?
  21. NadiaOn a select, the planner compares your WHERE clause with the same ranges, and drops every partition that can't contain a match. That's partition pruning. A query for September reads only the September table.
  22. ImranSo pruning picks the volume, and the partition's own index finds the page.
  23. NadiaExactly. And it still works when dates arrive as parameters, which is how Prisma sends them. Postgres then prunes when the query runs, and EXPLAIN shows a line like "subplans removed, four".
  24. ImranIs there a catch?
  25. NadiaOne big one, in the word "key". Pruning only works for filters on the partition key. A query that asks for site forty-two, with no date, must visit every partition.
  26. ImranAnd if the query does filter on the date?
  27. NadiaThen there's a quieter trap. If you wrap the date in a function, like truncating it to the month, the planner can't compare it with the ranges. It reads every partition, and the same rows come back, so nobody notices until the table is big.
  28. ImranSo write the filter as a plain range. From the first of the month, up to but not including the first of the next month.
  29. NadiaExactly, a half-open range on the raw column.
  30. NadiaThe second diagram shows both cases as two rows of six monthly boxes, May to October. In the top row, a filter on September's dates, and only the September box says "read". The other five are pruned. In the bottom row, a filter on site id only, and every box says "read".
  31. ImranSo without the date, the split doesn't help.
  32. NadiaEach partition's index still helps, so it's many small index lookups instead of one. But you lose most of the benefit, and the planner has more work.
  33. ImranWhat about the Store app's "find receipt by number" screen? It sends just the receipt id.
  34. NadiaThat's the lesson's think-first question. No date, so nothing is pruned. With thirty-six monthly partitions, that's thirty-six small index lookups instead of one. Each is fast, so the screen may still feel fine, but the cost grows every month you keep.
  35. NadiaThe cleaner fix is a product one. The printed receipt already shows its date. Let the screen send the date too. Then the query reads one partition.
  36. NadiaAt your desk, try the lesson's simulator. Pick a month range and watch how many partitions are read. Then switch the filter to "By site id only". At the bottom, compare deleting a year of old data with dropping whole partitions.
  37. ImranBefore we get to deleting. Are monthly date ranges the only option?
  38. NadiaThere are three kinds of rule. Range, where each partition takes a span of values, usually time. That's right for sales, goods received lines, handheld scanner events and logs. It's the common case.
  39. NadiaList, where each partition takes named values, like a region. Watch for skew. If sixty percent of your stores are in the capital, that partition stays huge. And hash, which spreads rows evenly when there's no natural range. Hash doesn't help you remove old data.
  40. NadiaNow the real win. Say you keep two years of sales, and the oldest month holds about fifty million rows.
  41. ImranWe'd run a delete for everything older than that date. At night.
  42. NadiaThat delete visits all fifty million rows. It writes a change record for each one into the write-ahead log, the WAL, so replicas and backups have to process all of it. It leaves fifty million dead rows for vacuum to clean. It can run for hours.
  43. NadiaWith partitions, you detach that month from the parent, so queries no longer see it. Archive it with pg dump if you need to, then drop the table. Dropping deletes its files, writes almost no WAL, and leaves nothing for vacuum.
  44. ImranThe librarian swapping the volume.
  45. NadiaYes. But locks matter, just like in the safe migrations episode. A plain detach takes the strongest lock on the parent for a moment. Quick, but it waits behind any long query, and blocks everyone behind it.
  46. NadiaSince Postgres fourteen, you can detach concurrently, with a lighter lock that lets reads and writes continue. Two rules: it can't run inside a transaction block, and it refuses to run while the table has a default partition.
  47. ImranAnd adding next month's partition?
  48. NadiaThat takes the strong lock too. So create future months ahead of time, in quiet hours, with a lock timeout. Never at midnight on the first of the month while the POS is busy.
  49. ImranAny other traps?
  50. NadiaFour more. First, the primary key must include the partition key. Each partition checks uniqueness only inside itself, so the key becomes id plus sale date. And sale line, which points at sale, must store both columns.
  51. NadiaSecond, too many partitions slow planning. The Postgres docs say the planner handles up to a few thousand fairly well, when queries prune most of them. Monthly for five years is sixty tables, easy. Daily for five years is over eighteen hundred, so measure first.
  52. NadiaThird, the default partition. It catches rows with no matching month, so inserts never fail. But if it holds even one November row, creating the November partition later fails. Keep it empty, and alert when it isn't.
  53. ImranAnd fourth?
  54. NadiaPartitioning isn't sharding. All partitions live on the same server, same CPU, same disk. It makes big tables manageable. It doesn't give you more write capacity. That's the sharding episode, two episodes from now.
  55. ImranSo when should we partition at all?
  56. NadiaThe Postgres docs give a rough rule: consider it when a table is bigger than the server's memory. Two other strong signals are a regular "delete everything older than so many months" job, and queries that nearly always filter by date. A five gigabyte table with good indexes rarely needs it.
  57. ImranLet me size it for Mangrove.
  58. NadiaOur three hundred and twenty stores ring up about four hundred thousand checkouts a day, with about four lines each. That's roughly one point six million sale line rows a day. About forty-eight million a month, close to six hundred million a year.
  59. ImranAnd Tomás wants twenty-five months online, so the festival report can compare this year's rush with last year's.
  60. NadiaSo sale and sale line are both partitioned by month. The "last seven days by store" dashboard reads one or two partitions. On the second of each month, a job detaches the month that just turned twenty-six months old, archives it, and drops it.
  61. ImranAnd my part is a product decision. Every report screen in the Store app sends a date range, with a sensible default like "last seven days".
  62. NadiaThe Build-it does this on Postgres sixteen. You create the parent sale table, partitioned by range on sale date, with a primary key of id and sale date. Leave the date out, and Postgres refuses with a clear error.
  63. NadiaThen three monthly partitions, August to October, plus a default, and one index on the parent, which Postgres creates on every partition. You load three hundred thousand test rows, and ask which physical table each row landed in. About a hundred thousand per month.
  64. ImranThen I prove pruning.
  65. NadiaWith EXPLAIN. The September query shows a scan of only the September partition. The site id query shows a scan of every partition, each through its own index.
  66. NadiaNext, a small function, run every night, that keeps the next three months' partitions ready, with a lock timeout. Or the pg partman extension.
  67. ImranAnd retiring August?
  68. NadiaBecause there's a default partition, the concurrent detach is refused. So you use a plain detach with a three second lock timeout. Then dump the month to a file, check it, and drop the table.
  69. NadiaLast, Prisma. It has no syntax for partitioning, so the model maps to the parent, and you add the partitioning by hand in the migration SQL. Queries always pass the date range.
  70. ImranCan I try any of this without installing Postgres?
  71. NadiaYes. The practice section has two exercises that run in your browser. In the first you create the missing monthly partitions and watch where new sales land. In the second you fix a store report so it reads only one month.
  72. ImranCan I partition our existing big table in place?
  73. NadiaNo, that's the common mistake. You create a new partitioned table, copy rows over month by month in batches, then switch the app in a short window.
  74. ImranFor my next review. Which column do almost all queries on this table filter by, and is that our partition key?
  75. NadiaGood. Also: what creates next month's partition, and what alerts us if that job didn't run? Do we remove old data by dropping partitions, or are we still planning a delete job? And does every report screen send a date range?
  76. NadiaOne to think about. You must remove all sales older than twenty-five months from a nine hundred million row table, partitioned by month. Batched deletes every night, or something else? Take a second.
  77. ImranBatched deletes were the right answer on the old table. But here, detach the old months and drop them.
  78. NadiaExactly. Batched deletes still touch every row. Dropping a partition removes its files at once.
  79. ImranMy takeaways. One: an index finds the page, partitioning picks the volume, and they work together.
  80. ImranTwo: the app always uses the parent name, and pruning only works when the query filters on the key. Three: old months leave by detach and drop, not by a giant delete.
  81. NadiaAnd one nuance on the second point: put the key in every unique constraint, and keep the default partition empty, or next month's partition can't be created.
  82. NadiaThe lesson ends with seven flashcards. Try them tomorrow, before you've had time to forget.
  83. NadiaNext episode: replication and read replicas, and what happens when your report reads a copy that's a few seconds behind.
  84. ImranSee you there.
Finished reading?Mark it complete to fill in its stop on your course line.

Use it at work

Monthly sales report

Revenue, orders and average order value per month, with the change on the month before.

Open in the SQL Lab
Next stopReplication and read replicas
Was this lesson helpful?
Questions and discussion

Loading the discussion…