<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Backend-Development on Vitalii Sukach</title><link>https://vitaliisukach.dev/tags/backend-development/</link><description>Recent content in Backend-Development on Vitalii Sukach</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Tue, 27 Oct 2026 10:00:00 +0000</lastBuildDate><atom:link href="https://vitaliisukach.dev/tags/backend-development/index.xml" rel="self" type="application/rss+xml"/><item><title>5 PostgreSQL Mistakes I See in Every Spring Boot App</title><link>https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/</link><pubDate>Tue, 27 Oct 2026 10:00:00 +0000</pubDate><guid>https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/</guid><description>&lt;p>&lt;em>Your ORM is lying to you a little. Here&amp;rsquo;s where.&lt;/em>&lt;/p>
&lt;p>&lt;img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-01.jpg">&lt;/p>
&lt;p>Here&amp;rsquo;s how it usually goes. Spring Boot app, Spring Data JPA, Postgres behind it. Everything is fast in dev, because dev has 200 rows. Staging is fine too. Then production hits a few million rows and suddenly the database CPU is at 90%, a deploy hangs for ten minutes, and someone in Slack asks &amp;ldquo;did anyone change anything?&amp;rdquo;&lt;/p>
&lt;p>Usually nobody changed anything. The problem was there from day one, it just needed data to show up.&lt;/p></description><content:encoded><![CDATA[<p><em>Your ORM is lying to you a little. Here&rsquo;s where.</em></p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-01.jpg"></p>
<p>Here&rsquo;s how it usually goes. Spring Boot app, Spring Data JPA, Postgres behind it. Everything is fast in dev, because dev has 200 rows. Staging is fine too. Then production hits a few million rows and suddenly the database CPU is at 90%, a deploy hangs for ten minutes, and someone in Slack asks &ldquo;did anyone change anything?&rdquo;</p>
<p>Usually nobody changed anything. The problem was there from day one, it just needed data to show up.</p>
<p>I keep running into the same five issues in pretty much every Spring Boot + Postgres codebase I look at. None of them are exotic. They&rsquo;re all &ldquo;the defaults did something you didn&rsquo;t expect&rdquo; kind of problems. Let&rsquo;s go through them.</p>
<h3 id="mistake-1-assuming-postgres-indexes-your-foreign-keys">Mistake 1: Assuming Postgres indexes your foreign keys</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-02.jpg"></p>
<p>Quick quiz. You have this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Entity</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">Order</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nd">@Id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">private</span><span class="w"> </span><span class="n">Long</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nd">@ManyToOne</span><span class="p">(</span><span class="n">fetch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">FetchType</span><span class="p">.</span><span class="na">LAZY</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nd">@JoinColumn</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;customer_id&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">private</span><span class="w"> </span><span class="n">Customer</span><span class="w"> </span><span class="n">customer</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>and the migration has <code>customer_id BIGINT REFERENCES customer(id)</code>. Is there an index on <code>order.customer_id</code>?</p>
<p>Nope.</p>
<p>Postgres automatically creates indexes for <code>PRIMARY KEY</code> and <code>UNIQUE</code> constraints, and that&rsquo;s it. A foreign key is just a constraint. It makes sure the referenced row exists, but it does nothing to make lookups by that column fast. If you came from MySQL this one hurts, because InnoDB <em>does</em> create an index for every foreign key, so lots of people just assume every database does.</p>
<p>Hibernate won&rsquo;t save you either. Even with <code>ddl-auto=update</code> (please don&rsquo;t do that in prod) it creates the FK constraint and no index.</p>
<p><strong>Where it bites you, part one: reads.</strong> Every <code>findByCustomerId</code>, every lazy load of <code>customer.getOrders()</code>, every join from customer to orders is a sequential scan on <code>order</code>. At 10k rows you won&rsquo;t notice. At 50 million you will.</p>
<p><strong>Where it bites you, part two: deletes.</strong> This is the sneaky one. When you delete a customer, Postgres has to check that no order still points to it. Internally it runs something like this for <strong>every deleted row</strong>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">customer_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="err">$</span><span class="mi">1</span><span class="w"> </span><span class="k">FOR</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="k">SHARE</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>No index means a full scan of <code>order</code> per deleted customer. Delete 1,000 customers and you get 1,000 sequential scans of your biggest table. With <code>ON DELETE CASCADE</code> it&rsquo;s the same story, Postgres still has to find the child rows somehow. The worst part is that <code>EXPLAIN</code> on your <code>DELETE</code> won&rsquo;t show it, because the check runs inside a trigger. You just see a delete that &ldquo;randomly&rdquo; takes 40 seconds. (<code>EXPLAIN ANALYZE</code> does show trigger time at the bottom, look for <code>RI_ConstraintTrigger</code>.)</p>
<p><strong>The fix:</strong> Index your FK columns. Not necessarily all of them, a tiny lookup table that never gets deleted from is fine without one, but that should be a decision you actually made, not something you forgot.</p>
<p>Here&rsquo;s a query that finds foreign keys whose column isn&rsquo;t the leading column of any index (it covers single-column FKs, which is 95% of what JPA produces):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conrelid</span><span class="p">::</span><span class="n">regclass</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">table_name</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="n">a</span><span class="p">.</span><span class="n">attname</span><span class="w">            </span><span class="k">AS</span><span class="w"> </span><span class="n">fk_column</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">c</span><span class="p">.</span><span class="n">conname</span><span class="w">            </span><span class="k">AS</span><span class="w"> </span><span class="k">constraint_name</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_constraint</span><span class="w"> </span><span class="k">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">JOIN</span><span class="w"> </span><span class="n">pg_attribute</span><span class="w"> </span><span class="n">a</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">ON</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">attrelid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conrelid</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">attnum</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conkey</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">WHERE</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">contype</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;f&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">AND</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_index</span><span class="w"> </span><span class="n">i</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="k">WHERE</span><span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="n">indrelid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conrelid</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">AND</span><span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="n">indkey</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conkey</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">AND</span><span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="n">indisvalid</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Run it on your prod database. I&rsquo;d honestly be surprised if it comes back empty.</p>
<p>Then add the indexes in a migration, and do it with <code>CONCURRENTLY</code> (more on why in mistake 5):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">CONCURRENTLY</span><span class="w"> </span><span class="n">idx_order_customer_id</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="p">(</span><span class="n">customer_id</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><h3 id="mistake-2-fetching-the-whole-object-graph-in-one-query">Mistake 2: Fetching the whole object graph in one query</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-03.jpg"></p>
<p>This one starts with good intentions. Someone reads about the N+1 problem, gets scared (fair), and decides everything should be loaded in one query.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Query</span><span class="p">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">    SELECT o FROM Order o
</span></span></span><span class="line"><span class="cl"><span class="s">    JOIN FETCH o.items
</span></span></span><span class="line"><span class="cl"><span class="s">    JOIN FETCH o.payments
</span></span></span><span class="line"><span class="cl"><span class="s">    WHERE o.createdAt &gt; :since
</span></span></span><span class="line"><span class="cl"><span class="s">    &#34;&#34;&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">List</span><span class="o">&lt;</span><span class="n">Order</span><span class="o">&gt;</span><span class="w"> </span><span class="nf">findRecentWithEverything</span><span class="p">(</span><span class="n">Instant</span><span class="w"> </span><span class="n">since</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>First thing that happens: Hibernate throws <code>MultipleBagFetchException: cannot simultaneously fetch multiple bags</code>. Somebody googles it, the top answer says &ldquo;change <code>List</code> to <code>Set</code>&rdquo;, they do it, the exception goes away, and everyone moves on.</p>
<p>The exception was actually trying to help. Here&rsquo;s what the database sends back now. An order with 50 items and 20 payments isn&rsquo;t 1 row, or 71 rows. It&rsquo;s <strong>50 × 20 = 1,000 rows</strong>, because SQL joins multiply. Every item row gets repeated for every payment. Load 200 orders like that and Postgres is shipping 200,000 wide rows over the network, all carrying the same order columns again and again.</p>
<p>Then Hibernate has to deal with them. For every single row it reads the result set, builds the entity key, checks the persistence context, finds &ldquo;oh, I already have this one&rdquo;, and throws the duplicate away. That&rsquo;s what hydration costs: you pay CPU and memory for thousands of rows just to end up with a few hundred objects. I wrote a whole deep dive on this, <a href="/posts/when-one-fast-query-becomes-a-production-nightmare-the-dehydration-problem-in-hibernate/">The Dehydration Problem in Hibernate</a>, but the short version is this: <strong>the query looks fast in <code>pg_stat_statements</code> and your app is slow anyway</strong>, because the time goes into moving and deduplicating rows, not into executing the SQL.</p>
<p>And it grows with every join you add. Add <code>JOIN FETCH o.shipments</code> with 5 shipments and you&rsquo;re at 5,000 rows per order.</p>
<p><strong>The fix:</strong> Fetch at most one collection per query. Load the rest separately:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">List</span><span class="o">&lt;</span><span class="n">Order</span><span class="o">&gt;</span><span class="w"> </span><span class="n">orders</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">orderRepository</span><span class="p">.</span><span class="na">findRecentWithItems</span><span class="p">(</span><span class="n">since</span><span class="p">);</span><span class="w">   </span><span class="c1">// JOIN FETCH items</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">orderRepository</span><span class="p">.</span><span class="na">findWithPayments</span><span class="p">(</span><span class="n">orders</span><span class="p">);</span><span class="w">                           </span><span class="c1">// JOIN FETCH payments WHERE o IN :orders</span><span class="w">
</span></span></span></code></pre></div><p>Since both queries run in the same persistence context, Hibernate fills <code>payments</code> into the <code>Order</code> instances it already has. You get two queries and 50 + 20 rows per order, instead of one query and 1,000 rows.</p>
<p>Even easier, turn on batch fetching globally and let lazy collections load in chunks:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-properties" data-lang="properties"><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.default_batch_fetch_size</span><span class="o">=</span><span class="s">50</span>
</span></span></code></pre></div><p>Now touching <code>order.getPayments()</code> loads payments for up to 50 orders in one <code>IN (...)</code> query. You get roughly N/50 queries instead of N+1, and no cartesian product.</p>
<p>And if this is a read-only screen or an API response, skip entities entirely and use a DTO projection. No persistence context, no dirty checking, no hydration of stuff you don&rsquo;t need.</p>
<p><strong>How to spot it:</strong> turn on <code>hibernate.generate_statistics</code> in staging and look for queries where rows fetched is way higher than entities loaded. Or look in <code>pg_stat_statements</code> for queries with a huge <code>rows / calls</code> ratio.</p>
<h3 id="mistake-3-treating-delete-as-free">Mistake 3: Treating DELETE as free</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-04.jpg"></p>
<p>In your head, <code>DELETE</code> makes data go away. In Postgres it doesn&rsquo;t, at least not right away.</p>
<p>Because of <a href="/posts/mvcc-invisible-superpower-inside-every-query/">MVCC</a>, a deleted row stays on the page. Postgres just marks it as dead for transactions that start after yours. The space comes back only after vacuum cleans it up. Until then every index still points to it, sequential scans still read it, and the table is exactly as big as before.</p>
<p>So a few Spring patterns that look totally innocent are actually expensive.</p>
<p><strong><code>deleteAll()</code> and derived <code>deleteBy...</code> methods.</strong> Spring Data JPA&rsquo;s <code>deleteAll()</code> doesn&rsquo;t run <code>DELETE FROM table</code>. It loads every entity into memory and then removes them one by one. Same for <code>deleteByStatus(...)</code>: it selects first, then deletes each row separately, firing cascades and lifecycle callbacks on the way. For 100k rows that&rsquo;s 100k <code>DELETE</code> statements plus a heap full of entities.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">repository</span><span class="p">.</span><span class="na">deleteAll</span><span class="p">();</span><span class="w">          </span><span class="c1">// SELECT * + N deletes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">repository</span><span class="p">.</span><span class="na">deleteAllInBatch</span><span class="p">();</span><span class="w">   </span><span class="c1">// one DELETE statement</span><span class="w">
</span></span></span></code></pre></div><p>If you don&rsquo;t need the JPA lifecycle, use <code>deleteAllInBatch()</code> or a <code>@Modifying @Query(&quot;delete from ...&quot;)</code>.</p>
<p><strong>The &ldquo;replace the collection&rdquo; pattern.</strong> This one is everywhere:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">order</span><span class="p">.</span><span class="na">getItems</span><span class="p">().</span><span class="na">clear</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">order</span><span class="p">.</span><span class="na">getItems</span><span class="p">().</span><span class="na">addAll</span><span class="p">(</span><span class="n">newItems</span><span class="p">);</span><span class="w">   </span><span class="c1">// with orphanRemoval = true</span><span class="w">
</span></span></span></code></pre></div><p>Every time the order is saved, Hibernate deletes all the old item rows and inserts new ones, even if 49 of the 50 items didn&rsquo;t change. That&rsquo;s 49 dead tuples and 49 new tuples for nothing, plus index entries for all of them. Do it on a hot path and you get a table that bloats constantly and autovacuum that never gets to rest. I measured exactly this, down to the page level, in <a href="/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/">Your DELETE + INSERT Is Burning PostgreSQL&rsquo;s CPU</a>. Spoiler: diffing the collection and only touching changed rows makes a huge difference.</p>
<p><strong>The nightly cleanup job.</strong> Everyone has one:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">DELETE</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">event_log</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">created_at</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">now</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">interval</span><span class="w"> </span><span class="s1">&#39;90 days&#39;</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>As a single statement on a big table, this:</p>
<ul>
<li>creates millions of dead tuples in one go</li>
<li>writes a huge burst of WAL, so your replicas fall behind</li>
<li>holds one long transaction, and as long as it&rsquo;s open, vacuum can&rsquo;t clean dead rows anywhere in the database that are newer than its snapshot</li>
<li>then autovacuum kicks in on a giant table and chews CPU and I/O for an hour, right when morning traffic starts</li>
</ul>
<p><strong>The fix:</strong> Delete in batches, with each batch committed in its own transaction:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">purgeOldEvents</span><span class="p">(</span><span class="n">Instant</span><span class="w"> </span><span class="n">cutoff</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kt">int</span><span class="w"> </span><span class="n">deleted</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">do</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">deleted</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="na">execute</span><span class="p">(</span><span class="n">status</span><span class="w"> </span><span class="o">-&gt;</span><span class="w"> </span><span class="n">jdbc</span><span class="p">.</span><span class="na">update</span><span class="p">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">            DELETE FROM event_log
</span></span></span><span class="line"><span class="cl"><span class="s">            WHERE id IN (
</span></span></span><span class="line"><span class="cl"><span class="s">                SELECT id FROM event_log
</span></span></span><span class="line"><span class="cl"><span class="s">                WHERE created_at &lt; ?
</span></span></span><span class="line"><span class="cl"><span class="s">                LIMIT 5000
</span></span></span><span class="line"><span class="cl"><span class="s">            )
</span></span></span><span class="line"><span class="cl"><span class="s">            &#34;&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">Timestamp</span><span class="p">.</span><span class="na">from</span><span class="p">(</span><span class="n">cutoff</span><span class="p">)));</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">deleted</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="n">0</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>tx</code> is a <code>TransactionTemplate</code>. Don&rsquo;t put <code>@Transactional</code> on the method, or the whole loop becomes one big transaction again and you&rsquo;re back where you started. And yes, <code>created_at</code> needs an index, otherwise every batch does a full scan.</p>
<p>For time-based data, the real fix is <strong>partitioning</strong>. Partition <code>event_log</code> by month, and cleanup becomes <code>DROP TABLE event_log_2026_07</code>. That&rsquo;s a metadata operation: no dead tuples, no vacuum, basically instant. If you&rsquo;re throwing away the whole table, <code>TRUNCATE</code> does the same thing for you.</p>
<p>And for tables that churn a lot, tune autovacuum per table instead of living with the defaults (which wait until 20% of the table is dead):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">order_item</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="p">(</span><span class="n">autovacuum_vacuum_scale_factor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">02</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>Keep an eye on <code>n_dead_tup</code> and <code>last_autovacuum</code> in <code>pg_stat_user_tables</code>. If dead tuples keep climbing and vacuum never catches up, you&rsquo;ve found your CPU problem.</p>
<h3 id="mistake-4-using-identity-and-wondering-why-batch-inserts-are-slow">Mistake 4: Using IDENTITY and wondering why batch inserts are slow</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-05.jpg"></p>
<p>Every tutorial shows this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">strategy</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">GenerationType</span><span class="p">.</span><span class="na">IDENTITY</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">private</span><span class="w"> </span><span class="n">Long</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Works fine. Looks clean. And it silently <strong>turns off JDBC insert batching</strong>.</p>
<p>Here&rsquo;s why. With <code>IDENTITY</code> the database generates the id during the <code>INSERT</code>. But Hibernate needs the id as soon as you call <code>persist()</code>, because that&rsquo;s how it tracks the entity in the persistence context. So it can&rsquo;t queue inserts and send them in a batch. It has to run each insert right away and read back the generated key.</p>
<p>So you set <code>hibernate.jdbc.batch_size=50</code>, call <code>saveAll()</code> with 10,000 entities, and get&hellip; 10,000 separate round trips. The batch setting is just ignored. No warning, nothing in the logs unless you go looking.</p>
<p><strong>The fix:</strong> Switch to a sequence with a proper allocation size:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">strategy</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">GenerationType</span><span class="p">.</span><span class="na">SEQUENCE</span><span class="p">,</span><span class="w"> </span><span class="n">generator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;order_seq&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nd">@SequenceGenerator</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;order_seq&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">sequenceName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;order_id_seq&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">allocationSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">50</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">private</span><span class="w"> </span><span class="n">Long</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Now Hibernate calls <code>nextval</code> once, gets a block of 50 ids, hands them out in memory, and can batch the inserts freely.</p>
<p>The important part: <strong>the database sequence has to use the same increment</strong>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="n">SEQUENCE</span><span class="w"> </span><span class="n">order_id_seq</span><span class="w"> </span><span class="k">INCREMENT</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="mi">50</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>If they don&rsquo;t match, recent Hibernate versions refuse to start (which is good). Older ones would happily hand out overlapping ids across instances and you&rsquo;d find out from duplicate key errors in prod. Good news if your table uses <code>bigserial</code>: the sequence already exists (<code>&lt;table&gt;_&lt;column&gt;_seq</code>), you just point the generator at it and change the increment. No data migration needed.</p>
<p>Then turn batching on properly:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-properties" data-lang="properties"><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.jdbc.batch_size</span><span class="o">=</span><span class="s">50</span>
</span></span><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.order_inserts</span><span class="o">=</span><span class="s">true</span>
</span></span><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.order_updates</span><span class="o">=</span><span class="s">true</span>
</span></span></code></pre></div><p>And add <code>reWriteBatchedInserts=true</code> to your JDBC URL. With it, the Postgres driver rewrites a batch into multi-row <code>INSERT ... VALUES (...), (...), (...)</code> statements, which is noticeably faster again.</p>
<pre tabindex="0"><code>jdbc:postgresql://db:5432/app?reWriteBatchedInserts=true
</code></pre><p>Yes, you&rsquo;ll get gaps in your ids when an instance restarts and throws away the rest of its block. That&rsquo;s fine. Ids are identifiers, not row counters. If someone needs them to have no gaps, that&rsquo;s a different column.</p>
<h3 id="mistake-5-migrations-that-lock-your-hottest-table">Mistake 5: Migrations that lock your hottest table</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-06.jpg"></p>
<p>Flyway or Liquibase runs on startup, applies a migration, and the app starts. Easy. Until the migration is this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">50</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>That&rsquo;s harmless, right? On Postgres 11+ adding a nullable column is instant, it only touches metadata.</p>
<p>Except it still needs an <code>ACCESS EXCLUSIVE</code> lock on the table, even if just for a millisecond. And if a long-running query (say, a 3-minute analytics report) is reading from <code>order</code> at that moment, the <code>ALTER</code> has to wait. Here&rsquo;s the bad part: <strong>while it waits, every new query on <code>order</code> queues up behind it</strong>, even simple <code>SELECT</code>s. Lock requests in Postgres are first-come-first-served, so nothing can jump ahead of the waiting <code>ALTER</code>. Your one-millisecond migration just made the most important table in your system unavailable for three minutes.</p>
<p>In Kubernetes it gets worse. The new pod is stuck on the migration, the readiness probe fails, the liveness probe kills the pod, a new pod starts, tries the migration again, and the whole thing repeats.</p>
<p><strong>Always set a lock timeout in migrations:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SET</span><span class="w"> </span><span class="n">lock_timeout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;5s&#39;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">50</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>If it can&rsquo;t get the lock within 5 seconds, it fails fast instead of taking production down with it. Retry the deploy when it&rsquo;s quieter.</p>
<p>Then there&rsquo;s the list of operations that are slow and locking on top of that:</p>
<p><strong><code>CREATE INDEX</code> without <code>CONCURRENTLY</code></strong> blocks all writes to the table for as long as the index takes to build. On a big table that can be many minutes. Use <code>CREATE INDEX CONCURRENTLY</code>. It&rsquo;s slower, but writes keep going. Catch: it can&rsquo;t run inside a transaction, and Flyway wraps every migration in one. Put it in its own migration file and Flyway will notice and run it outside a transaction (it refuses to mix it with other statements unless you set <code>mixed=true</code>). In Liquibase use <code>runInTransaction=&quot;false&quot;</code> on the changeset. If a concurrent build fails, it leaves an <code>INVALID</code> index behind, so drop it and try again.</p>
<p><strong>Adding <code>NOT NULL</code> to an existing column</strong> scans the whole table while holding <code>ACCESS EXCLUSIVE</code>. The safe way is in steps:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">order_source_not_null</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">CHECK</span><span class="w"> </span><span class="p">(</span><span class="k">source</span><span class="w"> </span><span class="k">IS</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">VALID</span><span class="p">;</span><span class="w">           </span><span class="c1">-- instant
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="n">VALIDATE</span><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">order_source_not_null</span><span class="p">;</span><span class="w">  </span><span class="c1">-- scans, but doesn&#39;t block writes
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ALTER</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">;</span><span class="w">  </span><span class="c1">-- PG12+ uses the check, no scan
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">DROP</span><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">order_source_not_null</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Same trick for new foreign keys: add with <code>NOT VALID</code>, then <code>VALIDATE CONSTRAINT</code> separately.</p>
<p><strong>Changing a column type</strong> (the classic <code>int</code> → <code>bigint</code> when you&rsquo;re about to run out of ids) rewrites the whole table under an exclusive lock. For a big table that means a new column, backfilling it in batches, and a swap. Not a one-line migration.</p>
<p><strong>Adding a column with a volatile default</strong> like <code>DEFAULT gen_random_uuid()</code> or <code>clock_timestamp()</code> also rewrites the table. Constant defaults are fine on Postgres 11+.</p>
<p>The general rule: before any migration touches a big table, ask two questions. &ldquo;What lock does this take?&rdquo; and &ldquo;Does this rewrite or scan the table?&rdquo; The <a href="https://www.postgresql.org/docs/current/sql-altertable.html">Postgres docs for ALTER TABLE</a> list the lock level for each operation, and it&rsquo;s worth five minutes of reading.</p>
<h3 id="a-quick-checklist">A quick checklist</h3>
<p><strong>Indexes</strong></p>
<ul>
<li>Did you run the missing FK index query on prod?</li>
<li>Are there <code>DELETE</code>s on parent tables that take way longer than they should?</li>
</ul>
<p><strong>Fetching</strong></p>
<ul>
<li>Any query with more than one <code>JOIN FETCH</code> on collections?</li>
<li>Did anyone &ldquo;fix&rdquo; <code>MultipleBagFetchException</code> by switching <code>List</code> to <code>Set</code>?</li>
<li>Is <code>default_batch_fetch_size</code> set?</li>
</ul>
<p><strong>Deletes</strong></p>
<ul>
<li>Are you calling <code>deleteAll()</code> or derived <code>deleteBy...</code> methods on big tables?</li>
<li>Do you <code>clear()</code> + <code>addAll()</code> collections with <code>orphanRemoval</code>?</li>
<li>Does your cleanup job delete everything in one statement?</li>
<li>What does <code>n_dead_tup</code> look like on your busiest tables?</li>
</ul>
<p><strong>Inserts</strong></p>
<ul>
<li>Are you using <code>GenerationType.IDENTITY</code> on entities you insert in bulk?</li>
<li>Does <code>allocationSize</code> match the sequence&rsquo;s <code>INCREMENT BY</code>?</li>
<li>Is <code>reWriteBatchedInserts=true</code> in your JDBC URL?</li>
</ul>
<p><strong>Migrations</strong></p>
<ul>
<li>Do your migrations set <code>lock_timeout</code>?</li>
<li>Does every <code>CREATE INDEX</code> on an existing table use <code>CONCURRENTLY</code>?</li>
<li>Does anyone review migrations for lock level before they merge?</li>
</ul>
<p>None of these mean your team did anything stupid. Spring Boot and Hibernate make the easy path really easy, and the easy path works great right up until you have real data. The good part is that every single fix here is small, can be done one table and one PR at a time, and you&rsquo;ll see the difference in your database CPU graph the same day.</p>
]]></content:encoded></item><item><title>Your DELETE + INSERT Is Burning PostgreSQL’s CPU. Here’s What’s Actually Happening on the Page.</title><link>https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/</link><pubDate>Mon, 17 Aug 2026 18:56:51 +0000</pubDate><guid>https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/</guid><description>&lt;p>&lt;em>A deep dive into MVCC, heap pages, vacuum, and what to do when you have billions of rows to update.&lt;/em>&lt;/p>
&lt;p>&lt;img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-01.png">&lt;/p>
&lt;h3 id="the-symptom">The symptom&lt;/h3>
&lt;p>You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:&lt;/p>
&lt;pre tabindex="0">&lt;code>BEGIN;
DELETE FROM user_state WHERE user_id = $1;
INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());
COMMIT;
&lt;/code>&lt;/pre>&lt;p>It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.&lt;/p></description><content:encoded><![CDATA[<p><em>A deep dive into MVCC, heap pages, vacuum, and what to do when you have billions of rows to update.</em></p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-01.png"></p>
<h3 id="the-symptom">The symptom</h3>
<p>You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:</p>
<pre tabindex="0"><code>BEGIN;
DELETE FROM user_state WHERE user_id = $1;
INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());
COMMIT;
</code></pre><p>It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.</p>
<p>Then event volume grows, and PostgreSQL’s CPU starts climbing in a way that nothing in your query plans explains. Your queries are still index scans. Your pg_stat_statements top entries look reasonable. But CPU is pinned, replication lag creeps up, and every so often the whole database gets slow for a few minutes and then recovers on its own.</p>
<p>If you look at pg_stat_activity during one of those windows, you&rsquo;ll usually find the culprit sitting right there:</p>
<pre tabindex="0"><code>autovacuum: VACUUM public.user_state
</code></pre><p>That’s the story. But “autovacuum is expensive” is not an explanation — it’s a restatement of the symptom. To actually fix this you need to understand what a DELETE followed by an INSERT does to the physical 8 KB pages on disk, and why that specific shape of write is close to the worst case for PostgreSQL&rsquo;s storage engine.</p>
<p>Let’s go all the way down.</p>
<h3 id="part-1-postgresql-never-modifies-a-row-inplace">Part 1: PostgreSQL never modifies a row in place</h3>
<p>This is the single fact that everything else follows from.</p>
<p>PostgreSQL implements MVCC (Multi-Version Concurrency Control) by keeping <strong>multiple physical versions of every row</strong> and letting each transaction see the version appropriate to its snapshot. There is no undo log (unlike Oracle or MySQL’s InnoDB, which keep old versions in a separate rollback segment). In PostgreSQL, old row versions live <strong>in the table itself</strong>, mixed in with the live ones.</p>
<p>Concretely:</p>
<ul>
<li>INSERT writes a new tuple and stamps it with xmin = the inserting transaction ID.</li>
<li>DELETE <strong>does not remove anything</strong>. It stamps the existing tuple with xmax = the deleting transaction ID. The bytes stay exactly where they were.</li>
<li>UPDATE is internally a DELETE + INSERT: it stamps xmax on the old tuple and writes a brand-new tuple somewhere.</li>
</ul>
<p>So a “deleted” row is still fully present on disk, occupying its bytes, until some later process comes along and physically reclaims it. That process is vacuum.</p>
<p>Here’s this happening for real. A 100-row table, then delete 50 rows and insert 50 new ones:</p>
<pre tabindex="0"><code>CREATE EXTENSION pageinspect;
CREATE TABLE demo (id int PRIMARY KEY, state text);
INSERT INTO demo SELECT g, &#39;active&#39; FROM generate_series(1,100) g;
-- delete half, re-insert half (the pattern from our app)
DELETE FROM demo WHERE id &lt;= 50;
INSERT INTO demo SELECT g, &#39;active&#39; FROM generate_series(1,50) g;
SELECT lp, lp_flags, lp_len, t_xmin, t_xmax
FROM heap_page_items(get_raw_page(&#39;demo&#39;, 0))
WHERE lp BETWEEN 48 AND 54;
</code></pre><pre tabindex="0"><code> lp | lp_flags | lp_len | t_xmin | t_xmax
----+----------+--------+--------+--------
 48 |        1 |     35 |    801 |    802   &lt;- deleted, still on the page
 49 |        1 |     35 |    801 |    802   &lt;- deleted, still on the page
 50 |        1 |     35 |    801 |    802   &lt;- deleted, still on the page
 51 |        1 |     35 |    801 |      0   &lt;- live
 52 |        1 |     35 |    801 |      0   &lt;- live
 53 |        1 |     35 |    801 |      0   &lt;- live
 54 |        1 |     35 |    801 |      0   &lt;- live
</code></pre><p>The deleted tuples have t_xmax = 802 (the transaction that deleted them) and are still LP_NORMAL (lp_flags = 1). They occupy real space. The table now has 150 line pointers where the logical row count is 100.</p>
<pre tabindex="0"><code>SELECT tuple_count, dead_tuple_count, dead_tuple_percent, free_percent
FROM pgstattuple(&#39;demo&#39;);
</code></pre><pre tabindex="0"><code> tuple_count | dead_tuple_count | dead_tuple_percent | free_percent
-------------+------------------+--------------------+--------------
         100 |               50 |              21.36 |        19.09
</code></pre><p><strong>21% of that table is garbage.</strong> And that’s after one round.</p>
<h3 id="part-2-anatomy-of-an-8-kbpage">Part 2: Anatomy of an 8 KB page</h3>
<p>PostgreSQL stores everything in fixed-size blocks — 8 KB by default (SHOW block_size). Understanding the layout of one page is the key to understanding why churn is expensive.</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-02.png"></p>
<p>Two structures grow toward each other from opposite ends: the <strong>line pointer array</strong> from the top, the <strong>tuple data</strong> from the bottom. The page is full when they meet.</p>
<p>The <strong>line pointer</strong> (also called an item identifier, ItemIdData) is a tiny 4-byte struct that says &ldquo;the tuple at slot N lives at byte offset X and is Y bytes long.&rdquo; It has four possible states, and you&rsquo;ll see all of them in this article:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-03.png"></p>
<p>The <strong>tuple header</strong> (HeapTupleHeader, 23 bytes padded to 24) is where MVCC lives:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-04.png"></p>
<p>That’s <strong>24 bytes of overhead per row version, plus 4 bytes for the line pointer.</strong> For a table storing a user ID, a short state string, and a timestamp, the MVCC bookkeeping can easily be 40% of the row’s on-disk size.</p>
<p>Now here’s the critical bit for our problem:</p>
<blockquote>
<p><em>A page’s free space is only usable for a</em> new <em>tuple if it is</em> <em><strong>contiguous</strong></em> <em>and the page has a free line pointer or room to add one. Dead tuples fragment the page. Only vacuum can defragment it.</em></p></blockquote>
<h3 id="part-3-why-delete--insert-is-the-worstshape">Part 3: Why DELETE + INSERT is the worst shape</h3>
<p>Compare what the two write patterns do to a page.</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-05.png"></p>
<p>A DELETE + INSERT can <strong>never</strong> be a HOT update. HOT (Heap-Only Tuple) is an optimization that only applies to UPDATE, and only when two conditions hold:</p>
<ol>
<li>No indexed column changed.</li>
<li>The new version fits <strong>on the same page</strong> as the old one.</li>
</ol>
<p>When both hold, PostgreSQL writes the new version onto the same page and links it via t_ctid, leaving the index entries alone. That&rsquo;s an enormous saving: index maintenance is usually the dominant cost of a write, because B-tree inserts are random-access, and every index on the table pays it.</p>
<p>By splitting your logical update into two statements, you’ve told PostgreSQL: <em>this is a brand-new row that has nothing to do with the old one.</em> It has no choice but to write full index entries into every index, and to leave a dead tuple behind that vacuum must chase down.</p>
<p>You can see HOT working with pageinspect. In a table where updates <em>are</em> HOT-eligible, the page fills up with redirect pointers:</p>
<pre tabindex="0"><code>SELECT lp, lp_flags, lp_len, t_ctid
FROM heap_page_items(get_raw_page(&#39;user_state_hot&#39;, 0)) LIMIT 5;
</code></pre><pre tabindex="0"><code> lp | lp_flags | lp_len | t_ctid
----+----------+--------+--------
  1 |        2 |      0 |          &lt;- LP_REDIRECT
  2 |        2 |      0 |          &lt;- LP_REDIRECT
  3 |        2 |      0 |          &lt;- LP_REDIRECT
  4 |        2 |      0 |          &lt;- LP_REDIRECT
  5 |        2 |      0 |          &lt;- LP_REDIRECT
</code></pre><p>Those redirect pointers are the physical trace of updates that cost <strong>zero index writes</strong>.</p>
<h3 id="part-4-what-vacuum-actuallydoes">Part 4: What vacuum actually does</h3>
<p>Vacuum is not one operation. A standard (non-full) VACUUM of a table with indexes runs in phases, and understanding the phases explains where the CPU goes.</p>
<h4 id="phase-1heapscan">Phase 1 — Heap scan</h4>
<p>Vacuum scans the table’s pages (skipping ones the <strong>visibility map</strong> marks all-visible, which is the main reason vacuum on an append-only table is cheap and vacuum on a churned table is not). For each page it:</p>
<ul>
<li>Prunes HOT chains and marks reclaimable tuples.</li>
<li>Collects the <strong>TIDs (page number, line pointer) of dead tuples</strong> into memory.</li>
<li>Sets LP_DEAD on their line pointers.</li>
</ul>
<p>The dead TID list is bounded by maintenance_work_mem (or autovacuum_work_mem). In PostgreSQL 17+ this list is stored in an adaptive radix tree (TidStore) rather than a flat array, which cut memory usage dramatically and removed the old 1 GB cap — a genuinely large improvement for high-churn tables.</p>
<h4 id="phase-2indexvacuum">Phase 2 — Index vacuum</h4>
<p><strong>This is the expensive part.</strong> For every index on the table, vacuum performs a full scan (a btbulkdelete) and removes every index entry pointing at a dead TID.</p>
<p>Note what that means: <strong>the cost is per index, and it’s a full index scan, not a lookup.</strong> A table with five indexes pays five full index scans per vacuum cycle. This is why “just add an index” decisions compound badly on high-churn tables.</p>
<h4 id="phase-3heapvacuum">Phase 3 — Heap vacuum</h4>
<p>Vacuum revisits the pages and turns LP_DEAD line pointers into LP_UNUSED, then <strong>defragments</strong> the page: it slides the remaining tuples together so the free space becomes one contiguous block. Now that space is genuinely reusable.</p>
<p>If the dead-TID list overflowed maintenance_work_mem in phase 1, vacuum has to loop: scan more heap → vacuum all indexes again → vacuum heap again. A single vacuum of a big churned table can do the full index pass <strong>multiple times</strong>. This is the classic &ldquo;why has autovacuum been running for six hours&rdquo; answer.</p>
<h4 id="phase-4freezing">Phase 4 — Freezing</h4>
<p>Separately, vacuum replaces old xmin values with a frozen marker so transaction IDs can wrap around safely. Anti-wraparound vacuums are non-cancelable and will run whether you like it or not. PostgreSQL 18 added <em>eager freezing</em> (vacuum_max_eager_freeze_failure_rate, default 0.03) so that normal vacuums opportunistically freeze pages, spreading this cost out instead of saving it up for one giant emergency vacuum.</p>
<h4 id="phase-5truncation">Phase 5 — Truncation</h4>
<p>If there are empty pages at the <em>end</em> of the table, vacuum tries to return them to the OS. This requires an ACCESS EXCLUSIVE lock (briefly). Note the word <em>end</em>: <strong>vacuum almost never shrinks a bloated table</strong>, because the free space is scattered throughout, not conveniently at the tail. It makes the space reusable by PostgreSQL, not available to your filesystem.</p>
<p>Here’s the before/after on our demo table:</p>
<pre tabindex="0"><code>VACUUM demo;
SELECT lp_flags, count(*) 
FROM heap_page_items(get_raw_page(&#39;demo&#39;,0)) GROUP BY 1;
</code></pre><p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-06.png"></p>
<p>Free space on the page went from <strong>19.09% → 43.51%</strong>. That’s the work vacuum did. Now multiply it by every page you churned.</p>
<h3 id="part-5-the-amplification-math">Part 5: The amplification math</h3>
<p>Here is the part people underestimate. One logical “user state changed” event does not cause one write. It causes, at minimum:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-07.png"></p>
<p>Steps 7–13 are <em>invisible in your application metrics</em>. Your app measures step 1–6 latency and reports “the insert took 2 ms.” The other two-thirds of the cost lands on a background worker, asynchronously, in a burst, on the same CPU.</p>
<p>That’s the answer to “why does CPU go up but my queries look fine.”</p>
<h4 id="point-5b-the-full-page-image-trap"><strong>Point 5b: the full-page-image trap</strong></h4>
<p>After each checkpoint, the first write to any page triggers a <strong>full 8 KB page image</strong> written into WAL (full_page_writes, on by default, and required for crash safety on most filesystems). If your churn touches many distinct pages, you pay a lot of FPIs. If your churn is concentrated on a few pages, you pay far fewer. Delete+insert scatters writes across pages — the new tuple often lands in a <em>different</em> page from the one you dirtied with the xmax write. Two dirty pages instead of one, two FPI candidates instead of one.</p>
<h4 id="point-5c-the-read-side-cost-nobodymentions">Point 5c: the read-side cost nobody mentions</h4>
<p>Dead tuples also make <strong>reads</strong> slower and CPU-hungrier:</p>
<ul>
<li>An index scan follows an index entry to the heap, reads the tuple, checks visibility, discovers it’s dead, and discards it. That’s a wasted buffer access plus a visibility check. You can see this directly: EXPLAIN (ANALYZE, BUFFERS) will show a rising Buffers: shared hit count for a query returning the same number of rows.</li>
<li>Bloated tables have worse cache locality. The same 100k live rows spread over 2× the pages means 2× the shared buffers to hold them, which evicts <em>other</em> things.</li>
<li>On a read, PostgreSQL may perform <strong>opportunistic HOT pruning</strong> on the page — real CPU work done inside your SELECT.</li>
</ul>
<h3 id="part-6-measurements">Part 6: Measurements</h3>
<p>I ran a controlled comparison: 100,000 rows, ten rounds of modifying 20,000 of them, with an explicit VACUUM after each round so we&rsquo;re looking at steady state, not a runaway. Three variants:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-08.png"></p>
<p>Read those rows carefully, because the middle one is the trap.</p>
<p><strong>Switching from</strong> **DELETE+**<strong>INSERT to</strong> <strong>UPDATE bought essentially nothing on its own.</strong> The second variant still generated <em>more</em> WAL than the first. Why? Because the UPDATE touched updated_at, and updated_at had an index on it. An UPDATE that modifies any indexed column is not HOT-eligible, so it does exactly what delete+insert does: new heap tuple, new entries in every index, dead tuple left behind.</p>
<p>The third variant — same UPDATE, but with no index on the mutated columns and fillfactor lowered to 80 — generated <strong>less than half the WAL</strong> and made 73% of its updates HOT.</p>
<p>The cost of that win is visible too: the HOT table starts at 1042 heap pages instead of 834, because fillfactor=80 deliberately leaves 20% of every page empty. That empty space is what new row versions land in. <strong>You are trading disk space for CPU and I/O</strong>, which on modern hardware is almost always the right trade.</p>
<p><em>(These are synthetic single-client numbers on PostgreSQL 16 — treat the ratios as directional, not as a benchmark of your workload.)</em></p>
<h3 id="part-7-fixing-it-in-order-ofleverage">Part 7: Fixing it, in order of leverage</h3>
<h4 id="71-dont-write-when-nothingchanged">7.1 Don’t write when nothing changed</h4>
<p>The cheapest write is the one that doesn’t happen. A shocking amount of churn in “collect user state” systems is <strong>rewriting a row to the exact same value</strong> because an event arrived that didn’t actually change anything.</p>
<pre tabindex="0"><code>INSERT INTO user_state (user_id, state, score, updated_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id) DO UPDATE
SET state      = EXCLUDED.state,
    score      = EXCLUDED.score,
    updated_at = now()
WHERE user_state.state  IS DISTINCT FROM EXCLUDED.state
   OR user_state.score  IS DISTINCT FROM EXCLUDED.score;
</code></pre><p>That trailing WHERE on the DO UPDATE clause is the whole trick: if nothing changed, PostgreSQL writes <strong>nothing</strong> — no new tuple, no dead tuple, no index entry, no WAL. In event-driven systems where the same state gets recomputed repeatedly, this alone often removes 50–90% of write volume.</p>
<p>Measure how much you’d save before you build anything else:</p>
<pre tabindex="0"><code>-- how many of your writes are no-ops?
SELECT count(*) FILTER (WHERE state = $2 AND score = $3) AS would_be_noop,
       count(*) AS total
FROM user_state WHERE user_id = ANY($1);
</code></pre><h4 id="72-make-your-updates-hot-eligible">7.2 Make your updates HOT-eligible</h4>
<p>Two requirements, both actionable:</p>
<p><strong>(a) Don’t index the columns that change.</strong> Audit every index against your write path:</p>
<pre tabindex="0"><code>SELECT i.relname AS index_name,
       array_agg(a.attname ORDER BY x.n) AS indexed_columns,
       s.idx_scan AS times_used
FROM pg_index ix
JOIN pg_class i ON i.oid = ix.indexrelid
JOIN pg_class t ON t.oid = ix.indrelid
JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS x(attnum, n) ON true
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = x.attnum
LEFT JOIN pg_stat_user_indexes s ON s.indexrelid = i.oid
WHERE t.relname = &#39;user_state&#39;
GROUP BY i.relname, s.idx_scan
ORDER BY s.idx_scan NULLS FIRST;
</code></pre><p>An index on updated_at is the single most common HOT-killer in exactly this kind of table. Ask whether you truly need it, or whether a partial index, a coarser index (date_trunc(&lsquo;hour&rsquo;, updated_at) doesn&rsquo;t help — but a separate low-churn audit table might), or no index at all would do.</p>
<p>Unused indexes are pure loss: they cost writes and vacuum passes and return nothing. Drop them.</p>
<p><strong>(b) Leave room on the page.</strong></p>
<pre tabindex="0"><code>ALTER TABLE user_state SET (fillfactor = 80);
-- fillfactor only applies to newly written pages;
-- rewrite the table to apply it to existing data:
VACUUM FULL user_state;   -- takes ACCESS EXCLUSIVE lock
-- or, online:
-- pg_repack -t user_state
</code></pre><p>For a heavily-updated table, 70–85 is a reasonable range. Below ~70 you’re wasting a lot of buffer cache on empty space.</p>
<p>Then confirm it’s working:</p>
<pre tabindex="0"><code>SELECT relname,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd,0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE relname = &#39;user_state&#39;;
</code></pre><p><strong>If</strong> <strong>hot_pct is near zero on an update-heavy table, that is your bug.</strong> Aim for 80%+.</p>
<h4 id="73-tune-autovacuum-for-this-table-notglobally">7.3 Tune autovacuum for this table, not globally</h4>
<p>The defaults are tuned for a small table on a small machine. autovacuum_vacuum_scale_factor = 0.2 means vacuum waits until <strong>20% of the table is dead</strong> before acting. On a 500-million-row table that&rsquo;s 100 million dead tuples before anything happens — and then one enormous, disruptive vacuum.</p>
<p>You want the opposite: vacuum <strong>more often, doing less each time.</strong></p>
<pre tabindex="0"><code>ALTER TABLE user_state SET (
  autovacuum_vacuum_scale_factor  = 0.01,   -- act at 1% dead, not 20%
  autovacuum_vacuum_threshold     = 1000,
  autovacuum_analyze_scale_factor = 0.02,
  autovacuum_vacuum_cost_delay    = 0,      -- don&#39;t throttle this table
  autovacuum_vacuum_cost_limit    = 3000
);
</code></pre><p>And globally, if your hardware is modern (NVMe, many cores), the shipped throttle is far too conservative:</p>
<pre tabindex="0"><code>autovacuum_max_workers      = 6          # default 3
autovacuum_naptime          = 15s        # default 1min
autovacuum_vacuum_cost_delay = 2ms       # default 2ms; consider 0 on fast storage
vacuum_cost_limit           = 2000       # default 200
maintenance_work_mem        = 2GB        # gives vacuum a big dead-TID store
autovacuum_work_mem         = 1GB
</code></pre><p>The cost-delay mechanism is the one most often left at a value that makes vacuum unable to keep up: vacuum accumulates “cost” (1 per buffer hit, 2 per miss, 20 per dirtied page) and sleeps cost_delay whenever it exceeds cost_limit. With the defaults, vacuum is deliberately slowed to something like a few MB/s. On a table generating hundreds of MB of dead tuples per minute, it will never catch up, and dead tuples accumulate forever.</p>
<p>PostgreSQL 18 also added autovacuum_vacuum_max_threshold (default 100,000,000), which caps the scale-factor calculation so very large tables can&rsquo;t defer vacuum indefinitely. It helps, but a per-table scale factor is still better.</p>
<h4 id="74-reconsider-the-datamodel">7.4 Reconsider the data model</h4>
<p>If your state genuinely is “replace the whole thing on every event,” ask whether it belongs in a row-per-user table at all:</p>
<ul>
<li><strong>Collapse many rows into one.</strong> If you’re storing N state rows per user and replacing them all, one row with a jsonb column can turn N deletes + N inserts into one update — and if jsonb is under the TOAST threshold and no indexed column changes, that update can be HOT. (Careful: a large jsonb that gets TOASTed is a whole separate write amplification problem. This works when the document is small, roughly under 2 KB.)</li>
<li><strong>Append-only + a materialized “current” view.</strong> Write events to a partitioned append-only table (no updates, no deletes, so almost no vacuum work — just insert-triggered vacuums for freezing), and maintain the current state separately, or compute it with a DISTINCT ON query over a well-indexed event log. Append-only tables are the cheapest thing PostgreSQL does.</li>
<li><strong>Move the hottest state out of PostgreSQL.</strong> If a piece of state changes many times per second per user and you only ever need the latest value, Redis or a dedicated KV store is the honest answer. Not everything needs to be transactional and durable.</li>
</ul>
<h3 id="part-8-the-hard-questionupdating-billions-ofrows">Part 8: The hard question — updating billions of rows</h3>
<p>This is the case where all the usual advice breaks, so it deserves its own treatment.</p>
<h4 id="why-the-naive-approach-fails-precisely">Why the naive approach fails, precisely</h4>
<pre tabindex="0"><code>UPDATE huge_table SET status = &#39;migrated&#39; WHERE status = &#39;pending&#39;;
-- 2,000,000,000 rows
</code></pre><p>Four separate disasters:</p>
<p><strong>1. Table size doubles.</strong> Every updated row writes a new version while the old one stays. A 2 TB table becomes 4 TB mid-statement. Then vacuum makes the space reusable but does <em>not</em> return it to the OS. You now permanently need 4 TB of disk for 2 TB of data.</p>
<p><strong>2. It generates terabytes of WAL</strong> in one burst. Replicas fall behind, archiving falls behind, and if you’re on a cloud provider you may fill the WAL volume and take the primary down.</p>
<p><strong>3. It holds a single transaction open for hours — and this is the one that hurts most.</strong> Vacuum can only remove tuples older than the oldest running transaction’s snapshot (the <em>xmin horizon</em>). A long transaction pins that horizon. So while your giant UPDATE runs, <strong>vacuum cannot clean up dead tuples anywhere in the entire database</strong> — not just this table. Every other table quietly bloats. This is how one migration takes down a whole cluster.</p>
<p><strong>4. If it fails at 95%, all the work is thrown away</strong> — but the dead tuples it created are not. You get the bloat with none of the benefit.</p>
<p>You can watch the horizon problem happen:</p>
<pre tabindex="0"><code>SELECT pid, age(backend_xid) AS xid_age, state,
       now() - xact_start AS duration, left(query, 60)
FROM pg_stat_activity
WHERE backend_xid IS NOT NULL
ORDER BY xact_start
LIMIT 5;
</code></pre><p>Also check for the other two horizon-pinners people forget: <strong>replication slots</strong> and <strong>prepared transactions</strong>.</p>
<pre tabindex="0"><code>SELECT slot_name, active, age(xmin) AS xmin_age,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
SELECT gid, prepared, age(transaction) FROM pg_prepared_xacts;
</code></pre><p>An inactive replication slot will pin the xmin horizon <strong>forever</strong> and no amount of vacuum tuning will save you.</p>
<h4 id="strategy-a-batched-backfill-with-keyset-pagination">Strategy A: Batched backfill with keyset pagination</h4>
<p>The workhorse. Small transactions, each one committing so the xmin horizon advances and vacuum can keep pace.</p>
<pre tabindex="0"><code>-- Track progress in a real table so the job is resumable
CREATE TABLE IF NOT EXISTS backfill_progress (
  job_name text PRIMARY KEY,
  last_id  bigint NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);
</code></pre><pre tabindex="0"><code>-- One batch. Run this in a loop from your application or a scheduler.
WITH bounds AS (
  SELECT last_id FROM backfill_progress WHERE job_name = &#39;status_migration&#39;
),
batch AS (
  SELECT id FROM huge_table
  WHERE id &gt; (SELECT last_id FROM bounds)
  ORDER BY id
  LIMIT 5000
  FOR UPDATE SKIP LOCKED
),
upd AS (
  UPDATE huge_table t
  SET status = &#39;migrated&#39;
  FROM batch b
  WHERE t.id = b.id
    AND t.status IS DISTINCT FROM &#39;migrated&#39;   -- skip no-op writes
  RETURNING t.id
)
UPDATE backfill_progress
SET last_id = COALESCE((SELECT max(id) FROM batch), last_id),
    updated_at = now()
WHERE job_name = &#39;status_migration&#39;
RETURNING last_id, (SELECT count(*) FROM upd) AS rows_updated;
</code></pre><p>Rules that make the difference between this working and this being a slow disaster:</p>
<ul>
<li><strong>Keyset pagination, never</strong> <strong>OFFSET.</strong> OFFSET 10000000 re-scans ten million rows every batch. WHERE id &gt; $last uses the index.</li>
<li><strong>Batch size 1,000–10,000 rows.</strong> Big enough to amortize overhead, small enough that each transaction finishes in well under a second.</li>
<li><strong>AND col IS DISTINCT FROM $new</strong> so re-running a batch is free and no-ops don&rsquo;t write.</li>
<li><strong>Throttle deliberately.</strong> Sleep 50–200 ms between batches. You are racing autovacuum; give it room to win. Adaptive throttling — sleep longer when replication lag or dead-tuple count rises — is worth the extra code:</li>
</ul>
<pre tabindex="0"><code>-- feed this into your loop&#39;s sleep decision
SELECT (SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname=&#39;huge_table&#39;) AS dead,
       (SELECT coalesce(max(pg_wal_lsn_diff(sent_lsn, replay_lsn)), 0)
        FROM pg_stat_replication) AS replica_lag_bytes;
</code></pre><ul>
<li><strong>Run it during a low-traffic window</strong> if you have one, but design it to be safe at any time.</li>
</ul>
<p><strong>Expected duration is the thing to be honest about:</strong> 2 billion rows at 5,000 rows/batch with a 100 ms sleep is 400,000 batches ≈ <strong>11+ hours minimum</strong>, realistically a few days. That’s fine. Plan for days, make it resumable, make it observable.</p>
<h4 id="strategy-b-partitioning--partition-swap">Strategy B: Partitioning + partition swap</h4>
<p>If your data is naturally partitionable (by time, tenant, hash of user ID), this changes the game completely, because you can rebuild a partition <strong>offline</strong> and swap it in atomically.</p>
<pre tabindex="0"><code>-- Assume: huge_table PARTITION BY RANGE (created_at)
--         with partition huge_table_2026_03
-- 1. Build the new version at full speed, outside the live table.
CREATE TABLE huge_table_2026_03_new (LIKE huge_table INCLUDING ALL);
INSERT INTO huge_table_2026_03_new
SELECT id, user_id, &#39;migrated&#39;::text AS status, payload, created_at
FROM huge_table_2026_03;
-- 2. Indexes AFTER the load - far faster than maintaining them during it.
--    No CONCURRENTLY needed: nothing is reading this table yet.
CREATE INDEX ON huge_table_2026_03_new (user_id);
ANALYZE huge_table_2026_03_new;
-- 3. Atomic swap: milliseconds of ACCESS EXCLUSIVE lock, not hours.
BEGIN;
  ALTER TABLE huge_table DETACH PARTITION huge_table_2026_03;
  ALTER TABLE huge_table ATTACH PARTITION huge_table_2026_03_new
    FOR VALUES FROM (&#39;2026-03-01&#39;) TO (&#39;2026-04-01&#39;);
COMMIT;
-- 4. Keep the old one around until you&#39;re confident, then drop it.
-- DROP TABLE huge_table_2026_03;
</code></pre><p>Why this is so much better:</p>
<ul>
<li>The bulk INSERT into an empty, unindexed table is <strong>the fastest write path PostgreSQL has</strong>. No dead tuples are created at all.</li>
<li>Zero vacuum debt: you never made garbage, so there’s nothing to collect.</li>
<li>The lock window is milliseconds.</li>
<li>Rollback is trivial: swap the old partition back.</li>
<li>You can do partitions <strong>one at a time</strong>, over weeks, with a rollback point after each.</li>
</ul>
<p>There’s a caveat worth knowing: ATTACH PARTITION validates that existing rows satisfy the partition constraint, which requires a scan — unless you add a matching CHECK constraint to the new table first, in which case PostgreSQL skips the scan.</p>
<pre tabindex="0"><code>ALTER TABLE huge_table_2026_03_new
  ADD CONSTRAINT part_check
  CHECK (created_at &gt;= &#39;2026-03-01&#39; AND created_at &lt; &#39;2026-04-01&#39;) NOT VALID;
ALTER TABLE huge_table_2026_03_new VALIDATE CONSTRAINT part_check;
-- now ATTACH is instant
</code></pre><p><strong>If you have billions of rows and no partitioning, partitioning is probably the actual project</strong> — not the update itself. Everything downstream gets easier: DROP PARTITION replaces mass DELETE (instant, and creates zero dead tuples), vacuum works per-partition in parallel, and index sizes stay manageable.</p>
<h4 id="strategy-c-rebuild-and-swap-the-wholetable">Strategy C: Rebuild and swap the whole table</h4>
<p>When the change is uniform and you can’t partition, build a new table and cut over.</p>
<p>For a bloated table with no schema change, pg_repack does this online: it builds a fresh copy, captures concurrent changes via triggers, replays them, and swaps — taking the exclusive lock only for the final instant.</p>
<pre tabindex="0"><code>pg_repack -h host -d mydb -t huge_table --jobs 4
</code></pre><p>For a schema/data change, the same idea by hand with logical replication or triggers; tools like pgroll and Percona&rsquo;s approach automate it. The pattern is always: <strong>new table, bulk load, keep in sync, atomic rename.</strong></p>
<p>Budget for double the disk space. That’s the price of every swap-based strategy.</p>
<h3 id="strategy-d-dont-update-atall">Strategy D: Don’t update at all</h3>
<p>The strategy people skip past. Ask whether the update is necessary:</p>
<ul>
<li>Add a <strong>new column with a default</strong> instead of rewriting values — since PostgreSQL 11, adding a column with a non-volatile default is a metadata-only change that rewrites nothing, no matter how big the table.</li>
<li>Write a <strong>view</strong> that computes the new value from the old one, and migrate readers to it.</li>
<li>Use a <strong>generated column</strong> if the new value is derivable.</li>
<li>Let it happen <strong>lazily</strong>: update rows as they’re naturally touched by the application, with a background job mopping up the long tail over months.</li>
</ul>
<p>A billion-row update you don’t have to run is strictly better than one you run well.</p>
<h3 id="part-9-what-to-actuallywatch">Part 9: What to actually watch</h3>
<p>Four queries. Put them on a dashboard.</p>
<p><strong>1. Is vacuum keeping up?</strong></p>
<pre tabindex="0"><code>SELECT relname,
       n_live_tup,
       n_dead_tup,
       round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
       last_autovacuum,
       autovacuum_count,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd, 0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE n_dead_tup &gt; 10000
ORDER BY n_dead_tup DESC
LIMIT 20;
</code></pre><p>Sustained dead_pct above ~10% on a busy table means vacuum is losing. hot_pct near zero on an update-heavy table means you&rsquo;re paying full index cost on every write.</p>
<p><strong>2. What is vacuum doing right now?</strong></p>
<pre tabindex="0"><code>SELECT p.pid, p.relid::regclass AS table, p.phase,
       pg_size_pretty(p.heap_blks_total * 8192::bigint) AS total,
       round(100.0 * p.heap_blks_scanned / NULLIF(p.heap_blks_total,0), 1) AS pct_scanned,
       p.index_vacuum_count,
       now() - a.xact_start AS running_for
FROM pg_stat_progress_vacuum p
JOIN pg_stat_activity a USING (pid);
</code></pre><p>index_vacuum_count &gt; 1 is the smoking gun for maintenance_work_mem being too small — vacuum had to do the index pass more than once.</p>
<p><strong>3. Is anything pinning the xmin horizon?</strong></p>
<pre tabindex="0"><code>SELECT &#39;backend&#39; AS src, pid::text AS id, age(backend_xmin) AS xmin_age
FROM pg_stat_activity WHERE backend_xmin IS NOT NULL
UNION ALL
SELECT &#39;slot&#39;, slot_name, age(xmin) FROM pg_replication_slots WHERE xmin IS NOT NULL
UNION ALL
SELECT &#39;prepared_xact&#39;, gid, age(transaction) FROM pg_prepared_xacts
ORDER BY xmin_age DESC NULLS LAST
LIMIT 10;
</code></pre><p>Anything with a large age here is preventing vacuum from cleaning up <strong>database-wide.</strong></p>
<p><strong>4. How bloated is it, really?</strong></p>
<pre tabindex="0"><code>CREATE EXTENSION IF NOT EXISTS pgstattuple;
-- accurate but does a full scan; use the sampling version on huge tables
SELECT * FROM pgstattuple(&#39;user_state&#39;);
SELECT * FROM pgstattuple_approx(&#39;huge_table&#39;);
SELECT * FROM pgstatindex(&#39;user_state_user_id_idx&#39;);
</code></pre><p>For indexes, avg_leaf_density below ~50 means the index is worth rebuilding:</p>
<pre tabindex="0"><code>REINDEX INDEX CONCURRENTLY user_state_user_id_idx;
</code></pre><h3 id="the-checklist">The checklist</h3>
<p>If you’re in the situation this article opened with, do these in order:</p>
<ol>
<li><strong>Measure first.</strong> dead_pct, hot_pct, pg_stat_progress_vacuum, xmin horizon. Don&rsquo;t tune blind.</li>
<li><strong>Kill no-op writes</strong> with ON CONFLICT DO UPDATE &hellip; WHERE &hellip; IS DISTINCT FROM. Usually the biggest single win, and the cheapest to ship.</li>
<li><strong>Convert delete+insert to</strong> <strong>UPDATE</strong> — but only after step 4, or it buys you nothing.</li>
<li><strong>Make updates HOT-eligible.</strong> Drop or move indexes on frequently-changed columns (especially updated_at). Set fillfactor to 80. Verify hot_pct climbs above 80%.</li>
<li><strong>Tune autovacuum per-table</strong>: scale factor down to 0.01, cost delay to 0, cost limit way up, maintenance_work_mem to gigabytes.</li>
<li><strong>Drop unused indexes.</strong> Every one is a write tax and an extra full scan per vacuum.</li>
<li><strong>Partition</strong> if the table is large and the data has a natural key. This makes every future problem easier.</li>
<li><strong>For mass updates</strong>: batched keyset backfill, or partition rebuild-and-swap. Never one giant statement.</li>
</ol>
<p>The underlying idea is simple enough to hold in your head: <strong>PostgreSQL is a copy-on-write storage engine that trades write amplification for concurrency.</strong> Delete+insert opts into the maximum possible amplification — no HOT, full index maintenance, guaranteed dead tuples, guaranteed vacuum work. Understanding what happens on the 8 KB page is what turns “the database is slow” into a list of specific, fixable things.</p>
<p><em>If this was useful, the two commands worth internalizing are</em> <em>pageinspect&rsquo;s</em> <em>heap_page_items() and</em> <em>pgstattuple(). Point them at your hottest table. What you find will probably explain your CPU graph.</em></p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://www.postgresql.org/docs/current/routine-vacuuming.html">PostgreSQL 18: Routine Vacuuming</a></li>
<li><a href="https://www.postgresql.org/docs/current/runtime-config-vacuum.html">PostgreSQL 18: Vacuuming configuration parameters</a></li>
<li><a href="https://www.postgresql.org/about/news/postgresql-18-released-3142/">PostgreSQL 18 Released</a></li>
<li><a href="https://techcommunity.microsoft.com/blog/adforpostgresql/postgresql-18-vacuuming-improvements-explained/4459484">PostgreSQL 18 Vacuuming Improvements Explained (Microsoft)</a></li>
<li><a href="https://www.dbi-services.com/blog/postgresql-18-introduce-autovacuum_vacuum_max_threshold/">autovacuum_vacuum_max_threshold in PostgreSQL 18 (dbi services)</a></li>
<li><a href="https://pganalyze.com/blog/5mins-postgres-17-faster-vacuum-adaptive-radix-trees">Faster VACUUM with Adaptive Radix Trees in PostgreSQL 17 (pganalyze)</a></li>
<li><a href="https://boringsql.com/posts/vacuum-at-the-page-level/">VACUUM at the Page Level (boringSQL)</a></li>
</ul>
]]></content:encoded></item></channel></rss>