<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Postgresql on Vitalii Sukach</title><link>https://vitaliisukach.dev/tags/postgresql/</link><description>Recent content in Postgresql on Vitalii Sukach</description><generator>Hugo</generator><language>en-us</language><lastBuildDate>Tue, 27 Oct 2026 10:00:00 +0000</lastBuildDate><atom:link href="https://vitaliisukach.dev/tags/postgresql/index.xml" rel="self" type="application/rss+xml"/><item><title>5 PostgreSQL Mistakes I See in Every Spring Boot App</title><link>https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/</link><pubDate>Tue, 27 Oct 2026 10:00:00 +0000</pubDate><guid>https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/</guid><description>&lt;p>&lt;em>Your ORM is lying to you a little. Here&amp;rsquo;s where.&lt;/em>&lt;/p>
&lt;p>&lt;img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-01.jpg">&lt;/p>
&lt;p>Here&amp;rsquo;s how it usually goes. Spring Boot app, Spring Data JPA, Postgres behind it. Everything is fast in dev, because dev has 200 rows. Staging is fine too. Then production hits a few million rows and suddenly the database CPU is at 90%, a deploy hangs for ten minutes, and someone in Slack asks &amp;ldquo;did anyone change anything?&amp;rdquo;&lt;/p>
&lt;p>Usually nobody changed anything. The problem was there from day one, it just needed data to show up.&lt;/p></description><content:encoded><![CDATA[<p><em>Your ORM is lying to you a little. Here&rsquo;s where.</em></p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-01.jpg"></p>
<p>Here&rsquo;s how it usually goes. Spring Boot app, Spring Data JPA, Postgres behind it. Everything is fast in dev, because dev has 200 rows. Staging is fine too. Then production hits a few million rows and suddenly the database CPU is at 90%, a deploy hangs for ten minutes, and someone in Slack asks &ldquo;did anyone change anything?&rdquo;</p>
<p>Usually nobody changed anything. The problem was there from day one, it just needed data to show up.</p>
<p>I keep running into the same five issues in pretty much every Spring Boot + Postgres codebase I look at. None of them are exotic. They&rsquo;re all &ldquo;the defaults did something you didn&rsquo;t expect&rdquo; kind of problems. Let&rsquo;s go through them.</p>
<h3 id="mistake-1-assuming-postgres-indexes-your-foreign-keys">Mistake 1: Assuming Postgres indexes your foreign keys</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-02.jpg"></p>
<p>Quick quiz. You have this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Entity</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">public</span><span class="w"> </span><span class="kd">class</span> <span class="nc">Order</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nd">@Id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">private</span><span class="w"> </span><span class="n">Long</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nd">@ManyToOne</span><span class="p">(</span><span class="n">fetch</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">FetchType</span><span class="p">.</span><span class="na">LAZY</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="nd">@JoinColumn</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;customer_id&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kd">private</span><span class="w"> </span><span class="n">Customer</span><span class="w"> </span><span class="n">customer</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p>and the migration has <code>customer_id BIGINT REFERENCES customer(id)</code>. Is there an index on <code>order.customer_id</code>?</p>
<p>Nope.</p>
<p>Postgres automatically creates indexes for <code>PRIMARY KEY</code> and <code>UNIQUE</code> constraints, and that&rsquo;s it. A foreign key is just a constraint. It makes sure the referenced row exists, but it does nothing to make lookups by that column fast. If you came from MySQL this one hurts, because InnoDB <em>does</em> create an index for every foreign key, so lots of people just assume every database does.</p>
<p>Hibernate won&rsquo;t save you either. Even with <code>ddl-auto=update</code> (please don&rsquo;t do that in prod) it creates the FK constraint and no index.</p>
<p><strong>Where it bites you, part one: reads.</strong> Every <code>findByCustomerId</code>, every lazy load of <code>customer.getOrders()</code>, every join from customer to orders is a sequential scan on <code>order</code>. At 10k rows you won&rsquo;t notice. At 50 million you will.</p>
<p><strong>Where it bites you, part two: deletes.</strong> This is the sneaky one. When you delete a customer, Postgres has to check that no order still points to it. Internally it runs something like this for <strong>every deleted row</strong>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">customer_id</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="err">$</span><span class="mi">1</span><span class="w"> </span><span class="k">FOR</span><span class="w"> </span><span class="k">KEY</span><span class="w"> </span><span class="k">SHARE</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>No index means a full scan of <code>order</code> per deleted customer. Delete 1,000 customers and you get 1,000 sequential scans of your biggest table. With <code>ON DELETE CASCADE</code> it&rsquo;s the same story, Postgres still has to find the child rows somehow. The worst part is that <code>EXPLAIN</code> on your <code>DELETE</code> won&rsquo;t show it, because the check runs inside a trigger. You just see a delete that &ldquo;randomly&rdquo; takes 40 seconds. (<code>EXPLAIN ANALYZE</code> does show trigger time at the bottom, look for <code>RI_ConstraintTrigger</code>.)</p>
<p><strong>The fix:</strong> Index your FK columns. Not necessarily all of them, a tiny lookup table that never gets deleted from is fine without one, but that should be a decision you actually made, not something you forgot.</p>
<p>Here&rsquo;s a query that finds foreign keys whose column isn&rsquo;t the leading column of any index (it covers single-column FKs, which is 95% of what JPA produces):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SELECT</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conrelid</span><span class="p">::</span><span class="n">regclass</span><span class="w"> </span><span class="k">AS</span><span class="w"> </span><span class="k">table_name</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="n">a</span><span class="p">.</span><span class="n">attname</span><span class="w">            </span><span class="k">AS</span><span class="w"> </span><span class="n">fk_column</span><span class="p">,</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">       </span><span class="k">c</span><span class="p">.</span><span class="n">conname</span><span class="w">            </span><span class="k">AS</span><span class="w"> </span><span class="k">constraint_name</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_constraint</span><span class="w"> </span><span class="k">c</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">JOIN</span><span class="w"> </span><span class="n">pg_attribute</span><span class="w"> </span><span class="n">a</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">ON</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">attrelid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conrelid</span><span class="w"> </span><span class="k">AND</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">attnum</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conkey</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">WHERE</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">contype</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;f&#39;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="k">AND</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">EXISTS</span><span class="w"> </span><span class="p">(</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="k">SELECT</span><span class="w"> </span><span class="mi">1</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">pg_index</span><span class="w"> </span><span class="n">i</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">      </span><span class="k">WHERE</span><span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="n">indrelid</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conrelid</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">AND</span><span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="n">indkey</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="k">c</span><span class="p">.</span><span class="n">conkey</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="k">AND</span><span class="w"> </span><span class="n">i</span><span class="p">.</span><span class="n">indisvalid</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">  </span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ORDER</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="mi">1</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Run it on your prod database. I&rsquo;d honestly be surprised if it comes back empty.</p>
<p>Then add the indexes in a migration, and do it with <code>CONCURRENTLY</code> (more on why in mistake 5):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">CREATE</span><span class="w"> </span><span class="k">INDEX</span><span class="w"> </span><span class="n">CONCURRENTLY</span><span class="w"> </span><span class="n">idx_order_customer_id</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="p">(</span><span class="n">customer_id</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><h3 id="mistake-2-fetching-the-whole-object-graph-in-one-query">Mistake 2: Fetching the whole object graph in one query</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-03.jpg"></p>
<p>This one starts with good intentions. Someone reads about the N+1 problem, gets scared (fair), and decides everything should be loaded in one query.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Query</span><span class="p">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">    SELECT o FROM Order o
</span></span></span><span class="line"><span class="cl"><span class="s">    JOIN FETCH o.items
</span></span></span><span class="line"><span class="cl"><span class="s">    JOIN FETCH o.payments
</span></span></span><span class="line"><span class="cl"><span class="s">    WHERE o.createdAt &gt; :since
</span></span></span><span class="line"><span class="cl"><span class="s">    &#34;&#34;&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">List</span><span class="o">&lt;</span><span class="n">Order</span><span class="o">&gt;</span><span class="w"> </span><span class="nf">findRecentWithEverything</span><span class="p">(</span><span class="n">Instant</span><span class="w"> </span><span class="n">since</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>First thing that happens: Hibernate throws <code>MultipleBagFetchException: cannot simultaneously fetch multiple bags</code>. Somebody googles it, the top answer says &ldquo;change <code>List</code> to <code>Set</code>&rdquo;, they do it, the exception goes away, and everyone moves on.</p>
<p>The exception was actually trying to help. Here&rsquo;s what the database sends back now. An order with 50 items and 20 payments isn&rsquo;t 1 row, or 71 rows. It&rsquo;s <strong>50 × 20 = 1,000 rows</strong>, because SQL joins multiply. Every item row gets repeated for every payment. Load 200 orders like that and Postgres is shipping 200,000 wide rows over the network, all carrying the same order columns again and again.</p>
<p>Then Hibernate has to deal with them. For every single row it reads the result set, builds the entity key, checks the persistence context, finds &ldquo;oh, I already have this one&rdquo;, and throws the duplicate away. That&rsquo;s what hydration costs: you pay CPU and memory for thousands of rows just to end up with a few hundred objects. I wrote a whole deep dive on this, <a href="/posts/when-one-fast-query-becomes-a-production-nightmare-the-dehydration-problem-in-hibernate/">The Dehydration Problem in Hibernate</a>, but the short version is this: <strong>the query looks fast in <code>pg_stat_statements</code> and your app is slow anyway</strong>, because the time goes into moving and deduplicating rows, not into executing the SQL.</p>
<p>And it grows with every join you add. Add <code>JOIN FETCH o.shipments</code> with 5 shipments and you&rsquo;re at 5,000 rows per order.</p>
<p><strong>The fix:</strong> Fetch at most one collection per query. Load the rest separately:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">List</span><span class="o">&lt;</span><span class="n">Order</span><span class="o">&gt;</span><span class="w"> </span><span class="n">orders</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">orderRepository</span><span class="p">.</span><span class="na">findRecentWithItems</span><span class="p">(</span><span class="n">since</span><span class="p">);</span><span class="w">   </span><span class="c1">// JOIN FETCH items</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">orderRepository</span><span class="p">.</span><span class="na">findWithPayments</span><span class="p">(</span><span class="n">orders</span><span class="p">);</span><span class="w">                           </span><span class="c1">// JOIN FETCH payments WHERE o IN :orders</span><span class="w">
</span></span></span></code></pre></div><p>Since both queries run in the same persistence context, Hibernate fills <code>payments</code> into the <code>Order</code> instances it already has. You get two queries and 50 + 20 rows per order, instead of one query and 1,000 rows.</p>
<p>Even easier, turn on batch fetching globally and let lazy collections load in chunks:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-properties" data-lang="properties"><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.default_batch_fetch_size</span><span class="o">=</span><span class="s">50</span>
</span></span></code></pre></div><p>Now touching <code>order.getPayments()</code> loads payments for up to 50 orders in one <code>IN (...)</code> query. You get roughly N/50 queries instead of N+1, and no cartesian product.</p>
<p>And if this is a read-only screen or an API response, skip entities entirely and use a DTO projection. No persistence context, no dirty checking, no hydration of stuff you don&rsquo;t need.</p>
<p><strong>How to spot it:</strong> turn on <code>hibernate.generate_statistics</code> in staging and look for queries where rows fetched is way higher than entities loaded. Or look in <code>pg_stat_statements</code> for queries with a huge <code>rows / calls</code> ratio.</p>
<h3 id="mistake-3-treating-delete-as-free">Mistake 3: Treating DELETE as free</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-04.jpg"></p>
<p>In your head, <code>DELETE</code> makes data go away. In Postgres it doesn&rsquo;t, at least not right away.</p>
<p>Because of <a href="/posts/mvcc-invisible-superpower-inside-every-query/">MVCC</a>, a deleted row stays on the page. Postgres just marks it as dead for transactions that start after yours. The space comes back only after vacuum cleans it up. Until then every index still points to it, sequential scans still read it, and the table is exactly as big as before.</p>
<p>So a few Spring patterns that look totally innocent are actually expensive.</p>
<p><strong><code>deleteAll()</code> and derived <code>deleteBy...</code> methods.</strong> Spring Data JPA&rsquo;s <code>deleteAll()</code> doesn&rsquo;t run <code>DELETE FROM table</code>. It loads every entity into memory and then removes them one by one. Same for <code>deleteByStatus(...)</code>: it selects first, then deletes each row separately, firing cascades and lifecycle callbacks on the way. For 100k rows that&rsquo;s 100k <code>DELETE</code> statements plus a heap full of entities.</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">repository</span><span class="p">.</span><span class="na">deleteAll</span><span class="p">();</span><span class="w">          </span><span class="c1">// SELECT * + N deletes</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">repository</span><span class="p">.</span><span class="na">deleteAllInBatch</span><span class="p">();</span><span class="w">   </span><span class="c1">// one DELETE statement</span><span class="w">
</span></span></span></code></pre></div><p>If you don&rsquo;t need the JPA lifecycle, use <code>deleteAllInBatch()</code> or a <code>@Modifying @Query(&quot;delete from ...&quot;)</code>.</p>
<p><strong>The &ldquo;replace the collection&rdquo; pattern.</strong> This one is everywhere:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="n">order</span><span class="p">.</span><span class="na">getItems</span><span class="p">().</span><span class="na">clear</span><span class="p">();</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="n">order</span><span class="p">.</span><span class="na">getItems</span><span class="p">().</span><span class="na">addAll</span><span class="p">(</span><span class="n">newItems</span><span class="p">);</span><span class="w">   </span><span class="c1">// with orphanRemoval = true</span><span class="w">
</span></span></span></code></pre></div><p>Every time the order is saved, Hibernate deletes all the old item rows and inserts new ones, even if 49 of the 50 items didn&rsquo;t change. That&rsquo;s 49 dead tuples and 49 new tuples for nothing, plus index entries for all of them. Do it on a hot path and you get a table that bloats constantly and autovacuum that never gets to rest. I measured exactly this, down to the page level, in <a href="/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/">Your DELETE + INSERT Is Burning PostgreSQL&rsquo;s CPU</a>. Spoiler: diffing the collection and only touching changed rows makes a huge difference.</p>
<p><strong>The nightly cleanup job.</strong> Everyone has one:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">DELETE</span><span class="w"> </span><span class="k">FROM</span><span class="w"> </span><span class="n">event_log</span><span class="w"> </span><span class="k">WHERE</span><span class="w"> </span><span class="n">created_at</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">now</span><span class="p">()</span><span class="w"> </span><span class="o">-</span><span class="w"> </span><span class="nb">interval</span><span class="w"> </span><span class="s1">&#39;90 days&#39;</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>As a single statement on a big table, this:</p>
<ul>
<li>creates millions of dead tuples in one go</li>
<li>writes a huge burst of WAL, so your replicas fall behind</li>
<li>holds one long transaction, and as long as it&rsquo;s open, vacuum can&rsquo;t clean dead rows anywhere in the database that are newer than its snapshot</li>
<li>then autovacuum kicks in on a giant table and chews CPU and I/O for an hour, right when morning traffic starts</li>
</ul>
<p><strong>The fix:</strong> Delete in batches, with each batch committed in its own transaction:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="kd">public</span><span class="w"> </span><span class="kt">void</span><span class="w"> </span><span class="nf">purgeOldEvents</span><span class="p">(</span><span class="n">Instant</span><span class="w"> </span><span class="n">cutoff</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="kt">int</span><span class="w"> </span><span class="n">deleted</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">do</span><span class="w"> </span><span class="p">{</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">        </span><span class="n">deleted</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">tx</span><span class="p">.</span><span class="na">execute</span><span class="p">(</span><span class="n">status</span><span class="w"> </span><span class="o">-&gt;</span><span class="w"> </span><span class="n">jdbc</span><span class="p">.</span><span class="na">update</span><span class="p">(</span><span class="s">&#34;&#34;&#34;
</span></span></span><span class="line"><span class="cl"><span class="s">            DELETE FROM event_log
</span></span></span><span class="line"><span class="cl"><span class="s">            WHERE id IN (
</span></span></span><span class="line"><span class="cl"><span class="s">                SELECT id FROM event_log
</span></span></span><span class="line"><span class="cl"><span class="s">                WHERE created_at &lt; ?
</span></span></span><span class="line"><span class="cl"><span class="s">                LIMIT 5000
</span></span></span><span class="line"><span class="cl"><span class="s">            )
</span></span></span><span class="line"><span class="cl"><span class="s">            &#34;&#34;&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">Timestamp</span><span class="p">.</span><span class="na">from</span><span class="p">(</span><span class="n">cutoff</span><span class="p">)));</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="p">}</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="p">(</span><span class="n">deleted</span><span class="w"> </span><span class="o">&gt;</span><span class="w"> </span><span class="n">0</span><span class="p">);</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="p">}</span><span class="w">
</span></span></span></code></pre></div><p><code>tx</code> is a <code>TransactionTemplate</code>. Don&rsquo;t put <code>@Transactional</code> on the method, or the whole loop becomes one big transaction again and you&rsquo;re back where you started. And yes, <code>created_at</code> needs an index, otherwise every batch does a full scan.</p>
<p>For time-based data, the real fix is <strong>partitioning</strong>. Partition <code>event_log</code> by month, and cleanup becomes <code>DROP TABLE event_log_2026_07</code>. That&rsquo;s a metadata operation: no dead tuples, no vacuum, basically instant. If you&rsquo;re throwing away the whole table, <code>TRUNCATE</code> does the same thing for you.</p>
<p>And for tables that churn a lot, tune autovacuum per table instead of living with the defaults (which wait until 20% of the table is dead):</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="n">order_item</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="p">(</span><span class="n">autovacuum_vacuum_scale_factor</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">0</span><span class="p">.</span><span class="mi">02</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>Keep an eye on <code>n_dead_tup</code> and <code>last_autovacuum</code> in <code>pg_stat_user_tables</code>. If dead tuples keep climbing and vacuum never catches up, you&rsquo;ve found your CPU problem.</p>
<h3 id="mistake-4-using-identity-and-wondering-why-batch-inserts-are-slow">Mistake 4: Using IDENTITY and wondering why batch inserts are slow</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-05.jpg"></p>
<p>Every tutorial shows this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">strategy</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">GenerationType</span><span class="p">.</span><span class="na">IDENTITY</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">private</span><span class="w"> </span><span class="n">Long</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Works fine. Looks clean. And it silently <strong>turns off JDBC insert batching</strong>.</p>
<p>Here&rsquo;s why. With <code>IDENTITY</code> the database generates the id during the <code>INSERT</code>. But Hibernate needs the id as soon as you call <code>persist()</code>, because that&rsquo;s how it tracks the entity in the persistence context. So it can&rsquo;t queue inserts and send them in a batch. It has to run each insert right away and read back the generated key.</p>
<p>So you set <code>hibernate.jdbc.batch_size=50</code>, call <code>saveAll()</code> with 10,000 entities, and get&hellip; 10,000 separate round trips. The batch setting is just ignored. No warning, nothing in the logs unless you go looking.</p>
<p><strong>The fix:</strong> Switch to a sequence with a proper allocation size:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-java" data-lang="java"><span class="line"><span class="cl"><span class="nd">@Id</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nd">@GeneratedValue</span><span class="p">(</span><span class="n">strategy</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">GenerationType</span><span class="p">.</span><span class="na">SEQUENCE</span><span class="p">,</span><span class="w"> </span><span class="n">generator</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;order_seq&#34;</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="nd">@SequenceGenerator</span><span class="p">(</span><span class="n">name</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;order_seq&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">sequenceName</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s">&#34;order_id_seq&#34;</span><span class="p">,</span><span class="w"> </span><span class="n">allocationSize</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">50</span><span class="p">)</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="kd">private</span><span class="w"> </span><span class="n">Long</span><span class="w"> </span><span class="n">id</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Now Hibernate calls <code>nextval</code> once, gets a block of 50 ids, hands them out in memory, and can batch the inserts freely.</p>
<p>The important part: <strong>the database sequence has to use the same increment</strong>:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="n">SEQUENCE</span><span class="w"> </span><span class="n">order_id_seq</span><span class="w"> </span><span class="k">INCREMENT</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="mi">50</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>If they don&rsquo;t match, recent Hibernate versions refuse to start (which is good). Older ones would happily hand out overlapping ids across instances and you&rsquo;d find out from duplicate key errors in prod. Good news if your table uses <code>bigserial</code>: the sequence already exists (<code>&lt;table&gt;_&lt;column&gt;_seq</code>), you just point the generator at it and change the increment. No data migration needed.</p>
<p>Then turn batching on properly:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-properties" data-lang="properties"><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.jdbc.batch_size</span><span class="o">=</span><span class="s">50</span>
</span></span><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.order_inserts</span><span class="o">=</span><span class="s">true</span>
</span></span><span class="line"><span class="cl"><span class="na">spring.jpa.properties.hibernate.order_updates</span><span class="o">=</span><span class="s">true</span>
</span></span></code></pre></div><p>And add <code>reWriteBatchedInserts=true</code> to your JDBC URL. With it, the Postgres driver rewrites a batch into multi-row <code>INSERT ... VALUES (...), (...), (...)</code> statements, which is noticeably faster again.</p>
<pre tabindex="0"><code>jdbc:postgresql://db:5432/app?reWriteBatchedInserts=true
</code></pre><p>Yes, you&rsquo;ll get gaps in your ids when an instance restarts and throws away the rest of its block. That&rsquo;s fine. Ids are identifiers, not row counters. If someone needs them to have no gaps, that&rsquo;s a different column.</p>
<h3 id="mistake-5-migrations-that-lock-your-hottest-table">Mistake 5: Migrations that lock your hottest table</h3>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/5-postgresql-mistakes-i-see-in-every-spring-boot-app/image-06.jpg"></p>
<p>Flyway or Liquibase runs on startup, applies a migration, and the app starts. Easy. Until the migration is this:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">50</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>That&rsquo;s harmless, right? On Postgres 11+ adding a nullable column is instant, it only touches metadata.</p>
<p>Except it still needs an <code>ACCESS EXCLUSIVE</code> lock on the table, even if just for a millisecond. And if a long-running query (say, a 3-minute analytics report) is reading from <code>order</code> at that moment, the <code>ALTER</code> has to wait. Here&rsquo;s the bad part: <strong>while it waits, every new query on <code>order</code> queues up behind it</strong>, even simple <code>SELECT</code>s. Lock requests in Postgres are first-come-first-served, so nothing can jump ahead of the waiting <code>ALTER</code>. Your one-millisecond migration just made the most important table in your system unavailable for three minutes.</p>
<p>In Kubernetes it gets worse. The new pod is stuck on the migration, the readiness probe fails, the liveness probe kills the pod, a new pod starts, tries the migration again, and the whole thing repeats.</p>
<p><strong>Always set a lock timeout in migrations:</strong></p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">SET</span><span class="w"> </span><span class="n">lock_timeout</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="s1">&#39;5s&#39;</span><span class="p">;</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="nb">VARCHAR</span><span class="p">(</span><span class="mi">50</span><span class="p">);</span><span class="w">
</span></span></span></code></pre></div><p>If it can&rsquo;t get the lock within 5 seconds, it fails fast instead of taking production down with it. Retry the deploy when it&rsquo;s quieter.</p>
<p>Then there&rsquo;s the list of operations that are slow and locking on top of that:</p>
<p><strong><code>CREATE INDEX</code> without <code>CONCURRENTLY</code></strong> blocks all writes to the table for as long as the index takes to build. On a big table that can be many minutes. Use <code>CREATE INDEX CONCURRENTLY</code>. It&rsquo;s slower, but writes keep going. Catch: it can&rsquo;t run inside a transaction, and Flyway wraps every migration in one. Put it in its own migration file and Flyway will notice and run it outside a transaction (it refuses to mix it with other statements unless you set <code>mixed=true</code>). In Liquibase use <code>runInTransaction=&quot;false&quot;</code> on the changeset. If a concurrent build fails, it leaves an <code>INVALID</code> index behind, so drop it and try again.</p>
<p><strong>Adding <code>NOT NULL</code> to an existing column</strong> scans the whole table while holding <code>ACCESS EXCLUSIVE</code>. The safe way is in steps:</p>
<div class="highlight"><pre tabindex="0" class="chroma"><code class="language-sql" data-lang="sql"><span class="line"><span class="cl"><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ADD</span><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">order_source_not_null</span><span class="w">
</span></span></span><span class="line"><span class="cl"><span class="w">    </span><span class="k">CHECK</span><span class="w"> </span><span class="p">(</span><span class="k">source</span><span class="w"> </span><span class="k">IS</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">)</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">VALID</span><span class="p">;</span><span class="w">           </span><span class="c1">-- instant
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="n">VALIDATE</span><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">order_source_not_null</span><span class="p">;</span><span class="w">  </span><span class="c1">-- scans, but doesn&#39;t block writes
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">ALTER</span><span class="w"> </span><span class="k">COLUMN</span><span class="w"> </span><span class="k">source</span><span class="w"> </span><span class="k">SET</span><span class="w"> </span><span class="k">NOT</span><span class="w"> </span><span class="k">NULL</span><span class="p">;</span><span class="w">  </span><span class="c1">-- PG12+ uses the check, no scan
</span></span></span><span class="line"><span class="cl"><span class="c1"></span><span class="k">ALTER</span><span class="w"> </span><span class="k">TABLE</span><span class="w"> </span><span class="s2">&#34;order&#34;</span><span class="w"> </span><span class="k">DROP</span><span class="w"> </span><span class="k">CONSTRAINT</span><span class="w"> </span><span class="n">order_source_not_null</span><span class="p">;</span><span class="w">
</span></span></span></code></pre></div><p>Same trick for new foreign keys: add with <code>NOT VALID</code>, then <code>VALIDATE CONSTRAINT</code> separately.</p>
<p><strong>Changing a column type</strong> (the classic <code>int</code> → <code>bigint</code> when you&rsquo;re about to run out of ids) rewrites the whole table under an exclusive lock. For a big table that means a new column, backfilling it in batches, and a swap. Not a one-line migration.</p>
<p><strong>Adding a column with a volatile default</strong> like <code>DEFAULT gen_random_uuid()</code> or <code>clock_timestamp()</code> also rewrites the table. Constant defaults are fine on Postgres 11+.</p>
<p>The general rule: before any migration touches a big table, ask two questions. &ldquo;What lock does this take?&rdquo; and &ldquo;Does this rewrite or scan the table?&rdquo; The <a href="https://www.postgresql.org/docs/current/sql-altertable.html">Postgres docs for ALTER TABLE</a> list the lock level for each operation, and it&rsquo;s worth five minutes of reading.</p>
<h3 id="a-quick-checklist">A quick checklist</h3>
<p><strong>Indexes</strong></p>
<ul>
<li>Did you run the missing FK index query on prod?</li>
<li>Are there <code>DELETE</code>s on parent tables that take way longer than they should?</li>
</ul>
<p><strong>Fetching</strong></p>
<ul>
<li>Any query with more than one <code>JOIN FETCH</code> on collections?</li>
<li>Did anyone &ldquo;fix&rdquo; <code>MultipleBagFetchException</code> by switching <code>List</code> to <code>Set</code>?</li>
<li>Is <code>default_batch_fetch_size</code> set?</li>
</ul>
<p><strong>Deletes</strong></p>
<ul>
<li>Are you calling <code>deleteAll()</code> or derived <code>deleteBy...</code> methods on big tables?</li>
<li>Do you <code>clear()</code> + <code>addAll()</code> collections with <code>orphanRemoval</code>?</li>
<li>Does your cleanup job delete everything in one statement?</li>
<li>What does <code>n_dead_tup</code> look like on your busiest tables?</li>
</ul>
<p><strong>Inserts</strong></p>
<ul>
<li>Are you using <code>GenerationType.IDENTITY</code> on entities you insert in bulk?</li>
<li>Does <code>allocationSize</code> match the sequence&rsquo;s <code>INCREMENT BY</code>?</li>
<li>Is <code>reWriteBatchedInserts=true</code> in your JDBC URL?</li>
</ul>
<p><strong>Migrations</strong></p>
<ul>
<li>Do your migrations set <code>lock_timeout</code>?</li>
<li>Does every <code>CREATE INDEX</code> on an existing table use <code>CONCURRENTLY</code>?</li>
<li>Does anyone review migrations for lock level before they merge?</li>
</ul>
<p>None of these mean your team did anything stupid. Spring Boot and Hibernate make the easy path really easy, and the easy path works great right up until you have real data. The good part is that every single fix here is small, can be done one table and one PR at a time, and you&rsquo;ll see the difference in your database CPU graph the same day.</p>
]]></content:encoded></item><item><title>Your DELETE + INSERT Is Burning PostgreSQL’s CPU. Here’s What’s Actually Happening on the Page.</title><link>https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/</link><pubDate>Mon, 17 Aug 2026 18:56:51 +0000</pubDate><guid>https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/</guid><description>&lt;p>&lt;em>A deep dive into MVCC, heap pages, vacuum, and what to do when you have billions of rows to update.&lt;/em>&lt;/p>
&lt;p>&lt;img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-01.png">&lt;/p>
&lt;h3 id="the-symptom">The symptom&lt;/h3>
&lt;p>You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:&lt;/p>
&lt;pre tabindex="0">&lt;code>BEGIN;
DELETE FROM user_state WHERE user_id = $1;
INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());
COMMIT;
&lt;/code>&lt;/pre>&lt;p>It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.&lt;/p></description><content:encoded><![CDATA[<p><em>A deep dive into MVCC, heap pages, vacuum, and what to do when you have billions of rows to update.</em></p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-01.png"></p>
<h3 id="the-symptom">The symptom</h3>
<p>You have a table that stores user state. Something happens — an event arrives — and your code does the obvious thing:</p>
<pre tabindex="0"><code>BEGIN;
DELETE FROM user_state WHERE user_id = $1;
INSERT INTO user_state (user_id, state, score, updated_at) VALUES ($1, $2, $3, now());
COMMIT;
</code></pre><p>It’s clean. It’s idempotent. It doesn’t require you to think about which columns changed. For a long time it works fine.</p>
<p>Then event volume grows, and PostgreSQL’s CPU starts climbing in a way that nothing in your query plans explains. Your queries are still index scans. Your pg_stat_statements top entries look reasonable. But CPU is pinned, replication lag creeps up, and every so often the whole database gets slow for a few minutes and then recovers on its own.</p>
<p>If you look at pg_stat_activity during one of those windows, you&rsquo;ll usually find the culprit sitting right there:</p>
<pre tabindex="0"><code>autovacuum: VACUUM public.user_state
</code></pre><p>That’s the story. But “autovacuum is expensive” is not an explanation — it’s a restatement of the symptom. To actually fix this you need to understand what a DELETE followed by an INSERT does to the physical 8 KB pages on disk, and why that specific shape of write is close to the worst case for PostgreSQL&rsquo;s storage engine.</p>
<p>Let’s go all the way down.</p>
<h3 id="part-1-postgresql-never-modifies-a-row-inplace">Part 1: PostgreSQL never modifies a row in place</h3>
<p>This is the single fact that everything else follows from.</p>
<p>PostgreSQL implements MVCC (Multi-Version Concurrency Control) by keeping <strong>multiple physical versions of every row</strong> and letting each transaction see the version appropriate to its snapshot. There is no undo log (unlike Oracle or MySQL’s InnoDB, which keep old versions in a separate rollback segment). In PostgreSQL, old row versions live <strong>in the table itself</strong>, mixed in with the live ones.</p>
<p>Concretely:</p>
<ul>
<li>INSERT writes a new tuple and stamps it with xmin = the inserting transaction ID.</li>
<li>DELETE <strong>does not remove anything</strong>. It stamps the existing tuple with xmax = the deleting transaction ID. The bytes stay exactly where they were.</li>
<li>UPDATE is internally a DELETE + INSERT: it stamps xmax on the old tuple and writes a brand-new tuple somewhere.</li>
</ul>
<p>So a “deleted” row is still fully present on disk, occupying its bytes, until some later process comes along and physically reclaims it. That process is vacuum.</p>
<p>Here’s this happening for real. A 100-row table, then delete 50 rows and insert 50 new ones:</p>
<pre tabindex="0"><code>CREATE EXTENSION pageinspect;
CREATE TABLE demo (id int PRIMARY KEY, state text);
INSERT INTO demo SELECT g, &#39;active&#39; FROM generate_series(1,100) g;
-- delete half, re-insert half (the pattern from our app)
DELETE FROM demo WHERE id &lt;= 50;
INSERT INTO demo SELECT g, &#39;active&#39; FROM generate_series(1,50) g;
SELECT lp, lp_flags, lp_len, t_xmin, t_xmax
FROM heap_page_items(get_raw_page(&#39;demo&#39;, 0))
WHERE lp BETWEEN 48 AND 54;
</code></pre><pre tabindex="0"><code> lp | lp_flags | lp_len | t_xmin | t_xmax
----+----------+--------+--------+--------
 48 |        1 |     35 |    801 |    802   &lt;- deleted, still on the page
 49 |        1 |     35 |    801 |    802   &lt;- deleted, still on the page
 50 |        1 |     35 |    801 |    802   &lt;- deleted, still on the page
 51 |        1 |     35 |    801 |      0   &lt;- live
 52 |        1 |     35 |    801 |      0   &lt;- live
 53 |        1 |     35 |    801 |      0   &lt;- live
 54 |        1 |     35 |    801 |      0   &lt;- live
</code></pre><p>The deleted tuples have t_xmax = 802 (the transaction that deleted them) and are still LP_NORMAL (lp_flags = 1). They occupy real space. The table now has 150 line pointers where the logical row count is 100.</p>
<pre tabindex="0"><code>SELECT tuple_count, dead_tuple_count, dead_tuple_percent, free_percent
FROM pgstattuple(&#39;demo&#39;);
</code></pre><pre tabindex="0"><code> tuple_count | dead_tuple_count | dead_tuple_percent | free_percent
-------------+------------------+--------------------+--------------
         100 |               50 |              21.36 |        19.09
</code></pre><p><strong>21% of that table is garbage.</strong> And that’s after one round.</p>
<h3 id="part-2-anatomy-of-an-8-kbpage">Part 2: Anatomy of an 8 KB page</h3>
<p>PostgreSQL stores everything in fixed-size blocks — 8 KB by default (SHOW block_size). Understanding the layout of one page is the key to understanding why churn is expensive.</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-02.png"></p>
<p>Two structures grow toward each other from opposite ends: the <strong>line pointer array</strong> from the top, the <strong>tuple data</strong> from the bottom. The page is full when they meet.</p>
<p>The <strong>line pointer</strong> (also called an item identifier, ItemIdData) is a tiny 4-byte struct that says &ldquo;the tuple at slot N lives at byte offset X and is Y bytes long.&rdquo; It has four possible states, and you&rsquo;ll see all of them in this article:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-03.png"></p>
<p>The <strong>tuple header</strong> (HeapTupleHeader, 23 bytes padded to 24) is where MVCC lives:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-04.png"></p>
<p>That’s <strong>24 bytes of overhead per row version, plus 4 bytes for the line pointer.</strong> For a table storing a user ID, a short state string, and a timestamp, the MVCC bookkeeping can easily be 40% of the row’s on-disk size.</p>
<p>Now here’s the critical bit for our problem:</p>
<blockquote>
<p><em>A page’s free space is only usable for a</em> new <em>tuple if it is</em> <em><strong>contiguous</strong></em> <em>and the page has a free line pointer or room to add one. Dead tuples fragment the page. Only vacuum can defragment it.</em></p></blockquote>
<h3 id="part-3-why-delete--insert-is-the-worstshape">Part 3: Why DELETE + INSERT is the worst shape</h3>
<p>Compare what the two write patterns do to a page.</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-05.png"></p>
<p>A DELETE + INSERT can <strong>never</strong> be a HOT update. HOT (Heap-Only Tuple) is an optimization that only applies to UPDATE, and only when two conditions hold:</p>
<ol>
<li>No indexed column changed.</li>
<li>The new version fits <strong>on the same page</strong> as the old one.</li>
</ol>
<p>When both hold, PostgreSQL writes the new version onto the same page and links it via t_ctid, leaving the index entries alone. That&rsquo;s an enormous saving: index maintenance is usually the dominant cost of a write, because B-tree inserts are random-access, and every index on the table pays it.</p>
<p>By splitting your logical update into two statements, you’ve told PostgreSQL: <em>this is a brand-new row that has nothing to do with the old one.</em> It has no choice but to write full index entries into every index, and to leave a dead tuple behind that vacuum must chase down.</p>
<p>You can see HOT working with pageinspect. In a table where updates <em>are</em> HOT-eligible, the page fills up with redirect pointers:</p>
<pre tabindex="0"><code>SELECT lp, lp_flags, lp_len, t_ctid
FROM heap_page_items(get_raw_page(&#39;user_state_hot&#39;, 0)) LIMIT 5;
</code></pre><pre tabindex="0"><code> lp | lp_flags | lp_len | t_ctid
----+----------+--------+--------
  1 |        2 |      0 |          &lt;- LP_REDIRECT
  2 |        2 |      0 |          &lt;- LP_REDIRECT
  3 |        2 |      0 |          &lt;- LP_REDIRECT
  4 |        2 |      0 |          &lt;- LP_REDIRECT
  5 |        2 |      0 |          &lt;- LP_REDIRECT
</code></pre><p>Those redirect pointers are the physical trace of updates that cost <strong>zero index writes</strong>.</p>
<h3 id="part-4-what-vacuum-actuallydoes">Part 4: What vacuum actually does</h3>
<p>Vacuum is not one operation. A standard (non-full) VACUUM of a table with indexes runs in phases, and understanding the phases explains where the CPU goes.</p>
<h4 id="phase-1heapscan">Phase 1 — Heap scan</h4>
<p>Vacuum scans the table’s pages (skipping ones the <strong>visibility map</strong> marks all-visible, which is the main reason vacuum on an append-only table is cheap and vacuum on a churned table is not). For each page it:</p>
<ul>
<li>Prunes HOT chains and marks reclaimable tuples.</li>
<li>Collects the <strong>TIDs (page number, line pointer) of dead tuples</strong> into memory.</li>
<li>Sets LP_DEAD on their line pointers.</li>
</ul>
<p>The dead TID list is bounded by maintenance_work_mem (or autovacuum_work_mem). In PostgreSQL 17+ this list is stored in an adaptive radix tree (TidStore) rather than a flat array, which cut memory usage dramatically and removed the old 1 GB cap — a genuinely large improvement for high-churn tables.</p>
<h4 id="phase-2indexvacuum">Phase 2 — Index vacuum</h4>
<p><strong>This is the expensive part.</strong> For every index on the table, vacuum performs a full scan (a btbulkdelete) and removes every index entry pointing at a dead TID.</p>
<p>Note what that means: <strong>the cost is per index, and it’s a full index scan, not a lookup.</strong> A table with five indexes pays five full index scans per vacuum cycle. This is why “just add an index” decisions compound badly on high-churn tables.</p>
<h4 id="phase-3heapvacuum">Phase 3 — Heap vacuum</h4>
<p>Vacuum revisits the pages and turns LP_DEAD line pointers into LP_UNUSED, then <strong>defragments</strong> the page: it slides the remaining tuples together so the free space becomes one contiguous block. Now that space is genuinely reusable.</p>
<p>If the dead-TID list overflowed maintenance_work_mem in phase 1, vacuum has to loop: scan more heap → vacuum all indexes again → vacuum heap again. A single vacuum of a big churned table can do the full index pass <strong>multiple times</strong>. This is the classic &ldquo;why has autovacuum been running for six hours&rdquo; answer.</p>
<h4 id="phase-4freezing">Phase 4 — Freezing</h4>
<p>Separately, vacuum replaces old xmin values with a frozen marker so transaction IDs can wrap around safely. Anti-wraparound vacuums are non-cancelable and will run whether you like it or not. PostgreSQL 18 added <em>eager freezing</em> (vacuum_max_eager_freeze_failure_rate, default 0.03) so that normal vacuums opportunistically freeze pages, spreading this cost out instead of saving it up for one giant emergency vacuum.</p>
<h4 id="phase-5truncation">Phase 5 — Truncation</h4>
<p>If there are empty pages at the <em>end</em> of the table, vacuum tries to return them to the OS. This requires an ACCESS EXCLUSIVE lock (briefly). Note the word <em>end</em>: <strong>vacuum almost never shrinks a bloated table</strong>, because the free space is scattered throughout, not conveniently at the tail. It makes the space reusable by PostgreSQL, not available to your filesystem.</p>
<p>Here’s the before/after on our demo table:</p>
<pre tabindex="0"><code>VACUUM demo;
SELECT lp_flags, count(*) 
FROM heap_page_items(get_raw_page(&#39;demo&#39;,0)) GROUP BY 1;
</code></pre><p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-06.png"></p>
<p>Free space on the page went from <strong>19.09% → 43.51%</strong>. That’s the work vacuum did. Now multiply it by every page you churned.</p>
<h3 id="part-5-the-amplification-math">Part 5: The amplification math</h3>
<p>Here is the part people underestimate. One logical “user state changed” event does not cause one write. It causes, at minimum:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-07.png"></p>
<p>Steps 7–13 are <em>invisible in your application metrics</em>. Your app measures step 1–6 latency and reports “the insert took 2 ms.” The other two-thirds of the cost lands on a background worker, asynchronously, in a burst, on the same CPU.</p>
<p>That’s the answer to “why does CPU go up but my queries look fine.”</p>
<h4 id="point-5b-the-full-page-image-trap"><strong>Point 5b: the full-page-image trap</strong></h4>
<p>After each checkpoint, the first write to any page triggers a <strong>full 8 KB page image</strong> written into WAL (full_page_writes, on by default, and required for crash safety on most filesystems). If your churn touches many distinct pages, you pay a lot of FPIs. If your churn is concentrated on a few pages, you pay far fewer. Delete+insert scatters writes across pages — the new tuple often lands in a <em>different</em> page from the one you dirtied with the xmax write. Two dirty pages instead of one, two FPI candidates instead of one.</p>
<h4 id="point-5c-the-read-side-cost-nobodymentions">Point 5c: the read-side cost nobody mentions</h4>
<p>Dead tuples also make <strong>reads</strong> slower and CPU-hungrier:</p>
<ul>
<li>An index scan follows an index entry to the heap, reads the tuple, checks visibility, discovers it’s dead, and discards it. That’s a wasted buffer access plus a visibility check. You can see this directly: EXPLAIN (ANALYZE, BUFFERS) will show a rising Buffers: shared hit count for a query returning the same number of rows.</li>
<li>Bloated tables have worse cache locality. The same 100k live rows spread over 2× the pages means 2× the shared buffers to hold them, which evicts <em>other</em> things.</li>
<li>On a read, PostgreSQL may perform <strong>opportunistic HOT pruning</strong> on the page — real CPU work done inside your SELECT.</li>
</ul>
<h3 id="part-6-measurements">Part 6: Measurements</h3>
<p>I ran a controlled comparison: 100,000 rows, ten rounds of modifying 20,000 of them, with an explicit VACUUM after each round so we&rsquo;re looking at steady state, not a runaway. Three variants:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/your-delete-insert-is-burning-postgresqls-cpu-here-s-what-s-actually-happening-on-the-page/image-08.png"></p>
<p>Read those rows carefully, because the middle one is the trap.</p>
<p><strong>Switching from</strong> **DELETE+**<strong>INSERT to</strong> <strong>UPDATE bought essentially nothing on its own.</strong> The second variant still generated <em>more</em> WAL than the first. Why? Because the UPDATE touched updated_at, and updated_at had an index on it. An UPDATE that modifies any indexed column is not HOT-eligible, so it does exactly what delete+insert does: new heap tuple, new entries in every index, dead tuple left behind.</p>
<p>The third variant — same UPDATE, but with no index on the mutated columns and fillfactor lowered to 80 — generated <strong>less than half the WAL</strong> and made 73% of its updates HOT.</p>
<p>The cost of that win is visible too: the HOT table starts at 1042 heap pages instead of 834, because fillfactor=80 deliberately leaves 20% of every page empty. That empty space is what new row versions land in. <strong>You are trading disk space for CPU and I/O</strong>, which on modern hardware is almost always the right trade.</p>
<p><em>(These are synthetic single-client numbers on PostgreSQL 16 — treat the ratios as directional, not as a benchmark of your workload.)</em></p>
<h3 id="part-7-fixing-it-in-order-ofleverage">Part 7: Fixing it, in order of leverage</h3>
<h4 id="71-dont-write-when-nothingchanged">7.1 Don’t write when nothing changed</h4>
<p>The cheapest write is the one that doesn’t happen. A shocking amount of churn in “collect user state” systems is <strong>rewriting a row to the exact same value</strong> because an event arrived that didn’t actually change anything.</p>
<pre tabindex="0"><code>INSERT INTO user_state (user_id, state, score, updated_at)
VALUES ($1, $2, $3, now())
ON CONFLICT (user_id) DO UPDATE
SET state      = EXCLUDED.state,
    score      = EXCLUDED.score,
    updated_at = now()
WHERE user_state.state  IS DISTINCT FROM EXCLUDED.state
   OR user_state.score  IS DISTINCT FROM EXCLUDED.score;
</code></pre><p>That trailing WHERE on the DO UPDATE clause is the whole trick: if nothing changed, PostgreSQL writes <strong>nothing</strong> — no new tuple, no dead tuple, no index entry, no WAL. In event-driven systems where the same state gets recomputed repeatedly, this alone often removes 50–90% of write volume.</p>
<p>Measure how much you’d save before you build anything else:</p>
<pre tabindex="0"><code>-- how many of your writes are no-ops?
SELECT count(*) FILTER (WHERE state = $2 AND score = $3) AS would_be_noop,
       count(*) AS total
FROM user_state WHERE user_id = ANY($1);
</code></pre><h4 id="72-make-your-updates-hot-eligible">7.2 Make your updates HOT-eligible</h4>
<p>Two requirements, both actionable:</p>
<p><strong>(a) Don’t index the columns that change.</strong> Audit every index against your write path:</p>
<pre tabindex="0"><code>SELECT i.relname AS index_name,
       array_agg(a.attname ORDER BY x.n) AS indexed_columns,
       s.idx_scan AS times_used
FROM pg_index ix
JOIN pg_class i ON i.oid = ix.indexrelid
JOIN pg_class t ON t.oid = ix.indrelid
JOIN LATERAL unnest(ix.indkey) WITH ORDINALITY AS x(attnum, n) ON true
JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = x.attnum
LEFT JOIN pg_stat_user_indexes s ON s.indexrelid = i.oid
WHERE t.relname = &#39;user_state&#39;
GROUP BY i.relname, s.idx_scan
ORDER BY s.idx_scan NULLS FIRST;
</code></pre><p>An index on updated_at is the single most common HOT-killer in exactly this kind of table. Ask whether you truly need it, or whether a partial index, a coarser index (date_trunc(&lsquo;hour&rsquo;, updated_at) doesn&rsquo;t help — but a separate low-churn audit table might), or no index at all would do.</p>
<p>Unused indexes are pure loss: they cost writes and vacuum passes and return nothing. Drop them.</p>
<p><strong>(b) Leave room on the page.</strong></p>
<pre tabindex="0"><code>ALTER TABLE user_state SET (fillfactor = 80);
-- fillfactor only applies to newly written pages;
-- rewrite the table to apply it to existing data:
VACUUM FULL user_state;   -- takes ACCESS EXCLUSIVE lock
-- or, online:
-- pg_repack -t user_state
</code></pre><p>For a heavily-updated table, 70–85 is a reasonable range. Below ~70 you’re wasting a lot of buffer cache on empty space.</p>
<p>Then confirm it’s working:</p>
<pre tabindex="0"><code>SELECT relname,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd,0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE relname = &#39;user_state&#39;;
</code></pre><p><strong>If</strong> <strong>hot_pct is near zero on an update-heavy table, that is your bug.</strong> Aim for 80%+.</p>
<h4 id="73-tune-autovacuum-for-this-table-notglobally">7.3 Tune autovacuum for this table, not globally</h4>
<p>The defaults are tuned for a small table on a small machine. autovacuum_vacuum_scale_factor = 0.2 means vacuum waits until <strong>20% of the table is dead</strong> before acting. On a 500-million-row table that&rsquo;s 100 million dead tuples before anything happens — and then one enormous, disruptive vacuum.</p>
<p>You want the opposite: vacuum <strong>more often, doing less each time.</strong></p>
<pre tabindex="0"><code>ALTER TABLE user_state SET (
  autovacuum_vacuum_scale_factor  = 0.01,   -- act at 1% dead, not 20%
  autovacuum_vacuum_threshold     = 1000,
  autovacuum_analyze_scale_factor = 0.02,
  autovacuum_vacuum_cost_delay    = 0,      -- don&#39;t throttle this table
  autovacuum_vacuum_cost_limit    = 3000
);
</code></pre><p>And globally, if your hardware is modern (NVMe, many cores), the shipped throttle is far too conservative:</p>
<pre tabindex="0"><code>autovacuum_max_workers      = 6          # default 3
autovacuum_naptime          = 15s        # default 1min
autovacuum_vacuum_cost_delay = 2ms       # default 2ms; consider 0 on fast storage
vacuum_cost_limit           = 2000       # default 200
maintenance_work_mem        = 2GB        # gives vacuum a big dead-TID store
autovacuum_work_mem         = 1GB
</code></pre><p>The cost-delay mechanism is the one most often left at a value that makes vacuum unable to keep up: vacuum accumulates “cost” (1 per buffer hit, 2 per miss, 20 per dirtied page) and sleeps cost_delay whenever it exceeds cost_limit. With the defaults, vacuum is deliberately slowed to something like a few MB/s. On a table generating hundreds of MB of dead tuples per minute, it will never catch up, and dead tuples accumulate forever.</p>
<p>PostgreSQL 18 also added autovacuum_vacuum_max_threshold (default 100,000,000), which caps the scale-factor calculation so very large tables can&rsquo;t defer vacuum indefinitely. It helps, but a per-table scale factor is still better.</p>
<h4 id="74-reconsider-the-datamodel">7.4 Reconsider the data model</h4>
<p>If your state genuinely is “replace the whole thing on every event,” ask whether it belongs in a row-per-user table at all:</p>
<ul>
<li><strong>Collapse many rows into one.</strong> If you’re storing N state rows per user and replacing them all, one row with a jsonb column can turn N deletes + N inserts into one update — and if jsonb is under the TOAST threshold and no indexed column changes, that update can be HOT. (Careful: a large jsonb that gets TOASTed is a whole separate write amplification problem. This works when the document is small, roughly under 2 KB.)</li>
<li><strong>Append-only + a materialized “current” view.</strong> Write events to a partitioned append-only table (no updates, no deletes, so almost no vacuum work — just insert-triggered vacuums for freezing), and maintain the current state separately, or compute it with a DISTINCT ON query over a well-indexed event log. Append-only tables are the cheapest thing PostgreSQL does.</li>
<li><strong>Move the hottest state out of PostgreSQL.</strong> If a piece of state changes many times per second per user and you only ever need the latest value, Redis or a dedicated KV store is the honest answer. Not everything needs to be transactional and durable.</li>
</ul>
<h3 id="part-8-the-hard-questionupdating-billions-ofrows">Part 8: The hard question — updating billions of rows</h3>
<p>This is the case where all the usual advice breaks, so it deserves its own treatment.</p>
<h4 id="why-the-naive-approach-fails-precisely">Why the naive approach fails, precisely</h4>
<pre tabindex="0"><code>UPDATE huge_table SET status = &#39;migrated&#39; WHERE status = &#39;pending&#39;;
-- 2,000,000,000 rows
</code></pre><p>Four separate disasters:</p>
<p><strong>1. Table size doubles.</strong> Every updated row writes a new version while the old one stays. A 2 TB table becomes 4 TB mid-statement. Then vacuum makes the space reusable but does <em>not</em> return it to the OS. You now permanently need 4 TB of disk for 2 TB of data.</p>
<p><strong>2. It generates terabytes of WAL</strong> in one burst. Replicas fall behind, archiving falls behind, and if you’re on a cloud provider you may fill the WAL volume and take the primary down.</p>
<p><strong>3. It holds a single transaction open for hours — and this is the one that hurts most.</strong> Vacuum can only remove tuples older than the oldest running transaction’s snapshot (the <em>xmin horizon</em>). A long transaction pins that horizon. So while your giant UPDATE runs, <strong>vacuum cannot clean up dead tuples anywhere in the entire database</strong> — not just this table. Every other table quietly bloats. This is how one migration takes down a whole cluster.</p>
<p><strong>4. If it fails at 95%, all the work is thrown away</strong> — but the dead tuples it created are not. You get the bloat with none of the benefit.</p>
<p>You can watch the horizon problem happen:</p>
<pre tabindex="0"><code>SELECT pid, age(backend_xid) AS xid_age, state,
       now() - xact_start AS duration, left(query, 60)
FROM pg_stat_activity
WHERE backend_xid IS NOT NULL
ORDER BY xact_start
LIMIT 5;
</code></pre><p>Also check for the other two horizon-pinners people forget: <strong>replication slots</strong> and <strong>prepared transactions</strong>.</p>
<pre tabindex="0"><code>SELECT slot_name, active, age(xmin) AS xmin_age,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots;
SELECT gid, prepared, age(transaction) FROM pg_prepared_xacts;
</code></pre><p>An inactive replication slot will pin the xmin horizon <strong>forever</strong> and no amount of vacuum tuning will save you.</p>
<h4 id="strategy-a-batched-backfill-with-keyset-pagination">Strategy A: Batched backfill with keyset pagination</h4>
<p>The workhorse. Small transactions, each one committing so the xmin horizon advances and vacuum can keep pace.</p>
<pre tabindex="0"><code>-- Track progress in a real table so the job is resumable
CREATE TABLE IF NOT EXISTS backfill_progress (
  job_name text PRIMARY KEY,
  last_id  bigint NOT NULL,
  updated_at timestamptz NOT NULL DEFAULT now()
);
</code></pre><pre tabindex="0"><code>-- One batch. Run this in a loop from your application or a scheduler.
WITH bounds AS (
  SELECT last_id FROM backfill_progress WHERE job_name = &#39;status_migration&#39;
),
batch AS (
  SELECT id FROM huge_table
  WHERE id &gt; (SELECT last_id FROM bounds)
  ORDER BY id
  LIMIT 5000
  FOR UPDATE SKIP LOCKED
),
upd AS (
  UPDATE huge_table t
  SET status = &#39;migrated&#39;
  FROM batch b
  WHERE t.id = b.id
    AND t.status IS DISTINCT FROM &#39;migrated&#39;   -- skip no-op writes
  RETURNING t.id
)
UPDATE backfill_progress
SET last_id = COALESCE((SELECT max(id) FROM batch), last_id),
    updated_at = now()
WHERE job_name = &#39;status_migration&#39;
RETURNING last_id, (SELECT count(*) FROM upd) AS rows_updated;
</code></pre><p>Rules that make the difference between this working and this being a slow disaster:</p>
<ul>
<li><strong>Keyset pagination, never</strong> <strong>OFFSET.</strong> OFFSET 10000000 re-scans ten million rows every batch. WHERE id &gt; $last uses the index.</li>
<li><strong>Batch size 1,000–10,000 rows.</strong> Big enough to amortize overhead, small enough that each transaction finishes in well under a second.</li>
<li><strong>AND col IS DISTINCT FROM $new</strong> so re-running a batch is free and no-ops don&rsquo;t write.</li>
<li><strong>Throttle deliberately.</strong> Sleep 50–200 ms between batches. You are racing autovacuum; give it room to win. Adaptive throttling — sleep longer when replication lag or dead-tuple count rises — is worth the extra code:</li>
</ul>
<pre tabindex="0"><code>-- feed this into your loop&#39;s sleep decision
SELECT (SELECT n_dead_tup FROM pg_stat_user_tables WHERE relname=&#39;huge_table&#39;) AS dead,
       (SELECT coalesce(max(pg_wal_lsn_diff(sent_lsn, replay_lsn)), 0)
        FROM pg_stat_replication) AS replica_lag_bytes;
</code></pre><ul>
<li><strong>Run it during a low-traffic window</strong> if you have one, but design it to be safe at any time.</li>
</ul>
<p><strong>Expected duration is the thing to be honest about:</strong> 2 billion rows at 5,000 rows/batch with a 100 ms sleep is 400,000 batches ≈ <strong>11+ hours minimum</strong>, realistically a few days. That’s fine. Plan for days, make it resumable, make it observable.</p>
<h4 id="strategy-b-partitioning--partition-swap">Strategy B: Partitioning + partition swap</h4>
<p>If your data is naturally partitionable (by time, tenant, hash of user ID), this changes the game completely, because you can rebuild a partition <strong>offline</strong> and swap it in atomically.</p>
<pre tabindex="0"><code>-- Assume: huge_table PARTITION BY RANGE (created_at)
--         with partition huge_table_2026_03
-- 1. Build the new version at full speed, outside the live table.
CREATE TABLE huge_table_2026_03_new (LIKE huge_table INCLUDING ALL);
INSERT INTO huge_table_2026_03_new
SELECT id, user_id, &#39;migrated&#39;::text AS status, payload, created_at
FROM huge_table_2026_03;
-- 2. Indexes AFTER the load - far faster than maintaining them during it.
--    No CONCURRENTLY needed: nothing is reading this table yet.
CREATE INDEX ON huge_table_2026_03_new (user_id);
ANALYZE huge_table_2026_03_new;
-- 3. Atomic swap: milliseconds of ACCESS EXCLUSIVE lock, not hours.
BEGIN;
  ALTER TABLE huge_table DETACH PARTITION huge_table_2026_03;
  ALTER TABLE huge_table ATTACH PARTITION huge_table_2026_03_new
    FOR VALUES FROM (&#39;2026-03-01&#39;) TO (&#39;2026-04-01&#39;);
COMMIT;
-- 4. Keep the old one around until you&#39;re confident, then drop it.
-- DROP TABLE huge_table_2026_03;
</code></pre><p>Why this is so much better:</p>
<ul>
<li>The bulk INSERT into an empty, unindexed table is <strong>the fastest write path PostgreSQL has</strong>. No dead tuples are created at all.</li>
<li>Zero vacuum debt: you never made garbage, so there’s nothing to collect.</li>
<li>The lock window is milliseconds.</li>
<li>Rollback is trivial: swap the old partition back.</li>
<li>You can do partitions <strong>one at a time</strong>, over weeks, with a rollback point after each.</li>
</ul>
<p>There’s a caveat worth knowing: ATTACH PARTITION validates that existing rows satisfy the partition constraint, which requires a scan — unless you add a matching CHECK constraint to the new table first, in which case PostgreSQL skips the scan.</p>
<pre tabindex="0"><code>ALTER TABLE huge_table_2026_03_new
  ADD CONSTRAINT part_check
  CHECK (created_at &gt;= &#39;2026-03-01&#39; AND created_at &lt; &#39;2026-04-01&#39;) NOT VALID;
ALTER TABLE huge_table_2026_03_new VALIDATE CONSTRAINT part_check;
-- now ATTACH is instant
</code></pre><p><strong>If you have billions of rows and no partitioning, partitioning is probably the actual project</strong> — not the update itself. Everything downstream gets easier: DROP PARTITION replaces mass DELETE (instant, and creates zero dead tuples), vacuum works per-partition in parallel, and index sizes stay manageable.</p>
<h4 id="strategy-c-rebuild-and-swap-the-wholetable">Strategy C: Rebuild and swap the whole table</h4>
<p>When the change is uniform and you can’t partition, build a new table and cut over.</p>
<p>For a bloated table with no schema change, pg_repack does this online: it builds a fresh copy, captures concurrent changes via triggers, replays them, and swaps — taking the exclusive lock only for the final instant.</p>
<pre tabindex="0"><code>pg_repack -h host -d mydb -t huge_table --jobs 4
</code></pre><p>For a schema/data change, the same idea by hand with logical replication or triggers; tools like pgroll and Percona&rsquo;s approach automate it. The pattern is always: <strong>new table, bulk load, keep in sync, atomic rename.</strong></p>
<p>Budget for double the disk space. That’s the price of every swap-based strategy.</p>
<h3 id="strategy-d-dont-update-atall">Strategy D: Don’t update at all</h3>
<p>The strategy people skip past. Ask whether the update is necessary:</p>
<ul>
<li>Add a <strong>new column with a default</strong> instead of rewriting values — since PostgreSQL 11, adding a column with a non-volatile default is a metadata-only change that rewrites nothing, no matter how big the table.</li>
<li>Write a <strong>view</strong> that computes the new value from the old one, and migrate readers to it.</li>
<li>Use a <strong>generated column</strong> if the new value is derivable.</li>
<li>Let it happen <strong>lazily</strong>: update rows as they’re naturally touched by the application, with a background job mopping up the long tail over months.</li>
</ul>
<p>A billion-row update you don’t have to run is strictly better than one you run well.</p>
<h3 id="part-9-what-to-actuallywatch">Part 9: What to actually watch</h3>
<p>Four queries. Put them on a dashboard.</p>
<p><strong>1. Is vacuum keeping up?</strong></p>
<pre tabindex="0"><code>SELECT relname,
       n_live_tup,
       n_dead_tup,
       round(100.0 * n_dead_tup / NULLIF(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
       last_autovacuum,
       autovacuum_count,
       n_tup_upd,
       n_tup_hot_upd,
       round(100.0 * n_tup_hot_upd / NULLIF(n_tup_upd, 0), 1) AS hot_pct
FROM pg_stat_user_tables
WHERE n_dead_tup &gt; 10000
ORDER BY n_dead_tup DESC
LIMIT 20;
</code></pre><p>Sustained dead_pct above ~10% on a busy table means vacuum is losing. hot_pct near zero on an update-heavy table means you&rsquo;re paying full index cost on every write.</p>
<p><strong>2. What is vacuum doing right now?</strong></p>
<pre tabindex="0"><code>SELECT p.pid, p.relid::regclass AS table, p.phase,
       pg_size_pretty(p.heap_blks_total * 8192::bigint) AS total,
       round(100.0 * p.heap_blks_scanned / NULLIF(p.heap_blks_total,0), 1) AS pct_scanned,
       p.index_vacuum_count,
       now() - a.xact_start AS running_for
FROM pg_stat_progress_vacuum p
JOIN pg_stat_activity a USING (pid);
</code></pre><p>index_vacuum_count &gt; 1 is the smoking gun for maintenance_work_mem being too small — vacuum had to do the index pass more than once.</p>
<p><strong>3. Is anything pinning the xmin horizon?</strong></p>
<pre tabindex="0"><code>SELECT &#39;backend&#39; AS src, pid::text AS id, age(backend_xmin) AS xmin_age
FROM pg_stat_activity WHERE backend_xmin IS NOT NULL
UNION ALL
SELECT &#39;slot&#39;, slot_name, age(xmin) FROM pg_replication_slots WHERE xmin IS NOT NULL
UNION ALL
SELECT &#39;prepared_xact&#39;, gid, age(transaction) FROM pg_prepared_xacts
ORDER BY xmin_age DESC NULLS LAST
LIMIT 10;
</code></pre><p>Anything with a large age here is preventing vacuum from cleaning up <strong>database-wide.</strong></p>
<p><strong>4. How bloated is it, really?</strong></p>
<pre tabindex="0"><code>CREATE EXTENSION IF NOT EXISTS pgstattuple;
-- accurate but does a full scan; use the sampling version on huge tables
SELECT * FROM pgstattuple(&#39;user_state&#39;);
SELECT * FROM pgstattuple_approx(&#39;huge_table&#39;);
SELECT * FROM pgstatindex(&#39;user_state_user_id_idx&#39;);
</code></pre><p>For indexes, avg_leaf_density below ~50 means the index is worth rebuilding:</p>
<pre tabindex="0"><code>REINDEX INDEX CONCURRENTLY user_state_user_id_idx;
</code></pre><h3 id="the-checklist">The checklist</h3>
<p>If you’re in the situation this article opened with, do these in order:</p>
<ol>
<li><strong>Measure first.</strong> dead_pct, hot_pct, pg_stat_progress_vacuum, xmin horizon. Don&rsquo;t tune blind.</li>
<li><strong>Kill no-op writes</strong> with ON CONFLICT DO UPDATE &hellip; WHERE &hellip; IS DISTINCT FROM. Usually the biggest single win, and the cheapest to ship.</li>
<li><strong>Convert delete+insert to</strong> <strong>UPDATE</strong> — but only after step 4, or it buys you nothing.</li>
<li><strong>Make updates HOT-eligible.</strong> Drop or move indexes on frequently-changed columns (especially updated_at). Set fillfactor to 80. Verify hot_pct climbs above 80%.</li>
<li><strong>Tune autovacuum per-table</strong>: scale factor down to 0.01, cost delay to 0, cost limit way up, maintenance_work_mem to gigabytes.</li>
<li><strong>Drop unused indexes.</strong> Every one is a write tax and an extra full scan per vacuum.</li>
<li><strong>Partition</strong> if the table is large and the data has a natural key. This makes every future problem easier.</li>
<li><strong>For mass updates</strong>: batched keyset backfill, or partition rebuild-and-swap. Never one giant statement.</li>
</ol>
<p>The underlying idea is simple enough to hold in your head: <strong>PostgreSQL is a copy-on-write storage engine that trades write amplification for concurrency.</strong> Delete+insert opts into the maximum possible amplification — no HOT, full index maintenance, guaranteed dead tuples, guaranteed vacuum work. Understanding what happens on the 8 KB page is what turns “the database is slow” into a list of specific, fixable things.</p>
<p><em>If this was useful, the two commands worth internalizing are</em> <em>pageinspect&rsquo;s</em> <em>heap_page_items() and</em> <em>pgstattuple(). Point them at your hottest table. What you find will probably explain your CPU graph.</em></p>
<h3 id="references">References</h3>
<ul>
<li><a href="https://www.postgresql.org/docs/current/routine-vacuuming.html">PostgreSQL 18: Routine Vacuuming</a></li>
<li><a href="https://www.postgresql.org/docs/current/runtime-config-vacuum.html">PostgreSQL 18: Vacuuming configuration parameters</a></li>
<li><a href="https://www.postgresql.org/about/news/postgresql-18-released-3142/">PostgreSQL 18 Released</a></li>
<li><a href="https://techcommunity.microsoft.com/blog/adforpostgresql/postgresql-18-vacuuming-improvements-explained/4459484">PostgreSQL 18 Vacuuming Improvements Explained (Microsoft)</a></li>
<li><a href="https://www.dbi-services.com/blog/postgresql-18-introduce-autovacuum_vacuum_max_threshold/">autovacuum_vacuum_max_threshold in PostgreSQL 18 (dbi services)</a></li>
<li><a href="https://pganalyze.com/blog/5mins-postgres-17-faster-vacuum-adaptive-radix-trees">Faster VACUUM with Adaptive Radix Trees in PostgreSQL 17 (pganalyze)</a></li>
<li><a href="https://boringsql.com/posts/vacuum-at-the-page-level/">VACUUM at the Page Level (boringSQL)</a></li>
</ul>
]]></content:encoded></item><item><title>DateTime Handling in Java and PostgreSQL: A Practical Guide</title><link>https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/</link><pubDate>Sat, 15 Aug 2026 14:24:50 +0000</pubDate><guid>https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/</guid><description>&lt;p>&lt;em>Everything I wish someone had told me before I stored a meeting time in a&lt;/em> &lt;em>timestamp column.&lt;/em>&lt;/p>
&lt;p>&lt;img loading="lazy" src="https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-01.png">&lt;/p>
&lt;p>There is a specific kind of bug that only shows up in production, only in October, and only for users in one country. It gets closed as “not reproducible” twice before someone finally sits down with a coffee and reads the column definition.&lt;/p>
&lt;p>It is almost always a datetime bug.&lt;/p>
&lt;p>The frustrating part is that both Java and PostgreSQL have &lt;em>excellent&lt;/em> date/time support. java.time is one of the best-designed APIs in the JDK. PostgreSQL&amp;rsquo;s temporal types are precise and well documented. The bugs happen at the seam — in the assumptions each side makes about what the other means, and in the two or three words in the type names that mean the opposite of what you&amp;rsquo;d guess.&lt;/p></description><content:encoded><![CDATA[<p><em>Everything I wish someone had told me before I stored a meeting time in a</em> <em>timestamp column.</em></p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-01.png"></p>
<p>There is a specific kind of bug that only shows up in production, only in October, and only for users in one country. It gets closed as “not reproducible” twice before someone finally sits down with a coffee and reads the column definition.</p>
<p>It is almost always a datetime bug.</p>
<p>The frustrating part is that both Java and PostgreSQL have <em>excellent</em> date/time support. java.time is one of the best-designed APIs in the JDK. PostgreSQL&rsquo;s temporal types are precise and well documented. The bugs happen at the seam — in the assumptions each side makes about what the other means, and in the two or three words in the type names that mean the opposite of what you&rsquo;d guess.</p>
<p>This guide walks through both models, then spends real time on the part that actually causes outages: time zones.</p>
<p>Every code output and query result below was executed — Java 21 and PostgreSQL 16 — not written from memory.</p>
<h3 id="part-1-the-javaside">Part 1: The Java Side</h3>
<h4 id="ask-three-questions-get-onetype">Ask three questions, get one type</h4>
<p>Every java.time class is an answer to some combination of three questions:</p>
<ol>
<li>Do I know the <strong>date</strong>?</li>
<li>Do I know the <strong>time of day</strong>?</li>
<li>Do I know which <strong>point on the global timeline</strong> this is?</li>
</ol>
<p>That last one is the important one, and it’s the one people skip. Here’s the map:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-02.png"></p>
<p>The single most useful sentence in this article: <strong>LocalDateTime is not a point in time.</strong> 2026-08-15T12:00 is twelve o&rsquo;clock <em>somewhere</em>, and until you say where, it refers to a 26-hour-wide smear across the planet. If you have an Instant and you call .toLocalDateTime(), you have thrown away information and the compiler will not warn you.</p>
<h4 id="instant-is-the-boring-correctdefault">Instant is the boring, correct default</h4>
<pre tabindex="0"><code>Instant now = Instant.now();     // 2026-08-15T10:00:00.123456789Z
</code></pre><p>An Instant is nanoseconds since the epoch. No zone, no offset, no calendar. When it prints with a Z, that isn&rsquo;t a stored time zone — it&rsquo;s the ISO-8601 way of rendering an epoch value.</p>
<p>Anything that <em>happened</em> — a login, a payment, a row insert — is an Instant. Reach for something else only when you have a reason.</p>
<h4 id="zoneid-vs-zoneoffset-not-the-samething">ZoneId vs ZoneOffset: not the same thing</h4>
<p>This distinction matters more than any other in this article.</p>
<pre tabindex="0"><code>ZoneId.of(&#34;Europe/London&#34;)  // -&gt; java.time.ZoneRegion
ZoneId.of(&#34;+01:00&#34;)         // -&gt; java.time.ZoneOffset
</code></pre><p>A <strong>ZoneOffset</strong> is a fixed number of hours and minutes from UTC. +01:00. That&rsquo;s the whole thing. It&rsquo;s a fact about one moment.</p>
<p>A <strong>ZoneId</strong> (specifically a ZoneRegion, like Europe/London) is a <em>rulebook</em>. It knows that London was +00:00 in January, +01:00 in August, and it knows the exact instants at which it flips — going back decades and forward as far as legislators have declared.</p>
<p>ZoneOffset is a value. ZoneId is a function from instant to offset. Confusing them is the root cause of most scheduling bugs.</p>
<h4 id="offsetdatetime-vs-zoneddatetime">OffsetDateTime vs ZonedDateTime</h4>
<p>Both identify a real instant. The difference is what survives.</p>
<pre tabindex="0"><code>ZonedDateTime nyc = ZonedDateTime.of(2026, 7, 4, 12, 0, 0, 0,
                                     ZoneId.of(&#34;America/New_York&#34;));
nyc                    // 2026-07-04T12:00-04:00[America/New_York]
nyc.toOffsetDateTime() // 2026-07-04T12:00-04:00
nyc.toInstant()        // 2026-07-04T16:00:00Z
</code></pre><p>All three describe the same moment. But the ZonedDateTime still knows it&rsquo;s in New York, so if you add six months to it, it will correctly land on -05:00. The OffsetDateTime has been flattened — it remembers <em>that</em> the offset was -04:00, not <em>why</em>. Add six months and you get a wrong wall-clock time.</p>
<p>Rule of thumb: ZonedDateTime for calculating, OffsetDateTime for transporting, Instant for storing.</p>
<h4 id="duration-vsperiod">Duration vs Period</h4>
<pre tabindex="0"><code>LocalDate.of(2026, 1, 31).plusMonths(1)  // 2026-02-28
</code></pre><p>Duration is exact elapsed time — seconds and nanos. Period is calendar time — years, months, days. &ldquo;One month&rdquo; is not a fixed number of seconds, and plusMonths clamps rather than overflowing. Both behaviors are correct; mixing them up is not.</p>
<p>This distinction resurfaces the moment DST enters the picture, and we’ll come back to it.</p>
<h4 id="a-word-on-the-oldapi">A word on the old API</h4>
<p>java.util.Date is a mislabeled Instant (it has no time zone despite what toString() implies). Calendar is mutable and lenient. java.sql.Timestamp extends java.util.Date and adds nanoseconds while breaking the equals contract.</p>
<p>If you’re on a legacy codebase, convert at the boundary and never let them into your domain:</p>
<pre tabindex="0"><code>Instant i = oldDate.toInstant();
Date    d = Date.from(instant);
</code></pre><h3 id="part-2-the-postgresql-side">Part 2: The PostgreSQL Side</h3>
<h4 id="timestamptz-does-not-store-a-timezone">timestamptz does not store a time zone</h4>
<p>This is the single biggest source of confusion, and it comes straight from the type name. TIMESTAMP WITH TIME ZONE is an 8-byte value. So is TIMESTAMP WITHOUT TIME ZONE. There is no room in there for a zone, and PostgreSQL doesn&rsquo;t try:</p>
<pre tabindex="0"><code>SELECT pg_column_size(&#39;2026-08-15 12:00:00&#39;::timestamp)     AS ts_bytes,     
       --  8
       pg_column_size(&#39;2026-08-15 12:00:00+02&#39;::timestamptz) AS tstz_bytes,  
       --  8
       pg_column_size(&#39;12:00:00+02&#39;::timetz)                 AS timetz_bytes; 
       -- 12
</code></pre><p>What timestamptz actually does is <strong>convert to UTC on write and convert back on read</strong>. The offset you supplied on input is used and then discarded. From the <a href="https://www.postgresql.org/docs/current/datatype-datetime.html">official docs</a>:</p>
<blockquote>
<p><em>In either case, the value is stored internally as UTC, and the originally stated or assumed time zone is not retained.</em></p></blockquote>
<p>A better mental model: timestamptz is Instant. timestamp is LocalDateTime. The names lie; the semantics don&rsquo;t.</p>
<h4 id="the-demo-that-makes-itclick">The demo that makes it click</h4>
<p>Insert the same two literals into both column types, with the session zone set to London:</p>
<pre tabindex="0"><code>SET TimeZone = &#39;Europe/London&#39;;
CREATE TABLE t (id int, ts timestamp, tstz timestamptz);
INSERT INTO t VALUES (1, &#39;2026-08-15 12:00:00&#39;,    &#39;2026-08-15 12:00:00&#39;);
INSERT INTO t VALUES (2, &#39;2026-08-15 12:00:00+00&#39;, &#39;2026-08-15 12:00:00+00&#39;);
SELECT * FROM t;
</code></pre><pre tabindex="0"><code> id |         ts          |          tstz
----+---------------------+------------------------
  1 | 2026-08-15 12:00:00 | 2026-08-15 12:00:00+01
  2 | 2026-08-15 12:00:00 | 2026-08-15 13:00:00+01
</code></pre><p>Two things happened. Row 1’s tstz had no offset, so PostgreSQL <em>assumed</em> the session zone. Row 2&rsquo;s explicit +00 was honoured and re-rendered as 13:00+01. Meanwhile the ts column ignored the offset entirely and stored the digits.</p>
<p>Now reconnect from New York and run the identical SELECT:</p>
<pre tabindex="0"><code> id |         ts          |          tstz
----+---------------------+------------------------
  1 | 2026-08-15 12:00:00 | 2026-08-15 07:00:00-04
  2 | 2026-08-15 12:00:00 | 2026-08-15 08:00:00-04
</code></pre><p>The tstz column re-renders per client. The ts column is frozen. <strong>Neither row changed on disk.</strong></p>
<p>That last point deserves emphasis, because it’s where the panic starts: when a developer sees a different value than QA saw, the instinct is “the data is corrupted.” It isn’t. Two clients with different TimeZone settings are looking at the same UTC value through different windows.</p>
<h4 id="at-time-zone-runs-bothways">AT TIME ZONE runs both ways</h4>
<p>AT TIME ZONE is the one operator you need, and its behavior flips depending on the input type:</p>
<pre tabindex="0"><code>-- timestamptz -&gt; timestamp: &#34;what does the clock read there?&#34;
SELECT &#39;2026-08-15 12:00:00+00&#39;::timestamptz AT TIME ZONE &#39;Asia/Tokyo&#39;;
-- 2026-08-15 21:00:00   (type: timestamp without time zone)
</code></pre><pre tabindex="0"><code>-- timestamp -&gt; timestamptz: &#34;this wall clock reading WAS in Tokyo&#34;
SELECT &#39;2026-08-15 12:00:00&#39;::timestamp AT TIME ZONE &#39;Asia/Tokyo&#39;;
-- 2026-08-15 03:00:00+00   (type: timestamp with time zone)
</code></pre><p>One removes zone context, the other attaches it. Read it as “interpret this value in that zone.” The function form timezone(&lsquo;Asia/Tokyo&rsquo;, value) is identical.</p>
<h4 id="the-rest-of-thefamily">The rest of the family</h4>
<ul>
<li><strong>date</strong> — 4 bytes, no zone, no ambiguity. Great for things that are genuinely dates.</li>
<li><strong>time</strong> — time of day, no zone.</li>
<li><strong>timetz</strong> — 12 bytes, and <a href="https://www.postgresql.org/docs/current/datatype-datetime.html">PostgreSQL itself recommends against it</a>. A time of day with an offset but no date can&rsquo;t resolve DST, so it&rsquo;s not meaningfully anchored. It won&rsquo;t even accept a zone name:</li>
</ul>
<pre tabindex="0"><code>SELECT &#39;12:00:00 Europe/London&#39;::timetz; -- ERROR: invalid input syntax for type time with time zone
</code></pre><ul>
<li><strong>interval</strong> — 16 bytes storing months, days, and microseconds <em>separately</em>, which is exactly why it can be DST-aware.</li>
</ul>
<h4 id="now-vs-localtimestamp">now() vs localtimestamp</h4>
<pre tabindex="0"><code>SELECT pg_typeof(now()),            -- timestamp with time zone
       pg_typeof(current_timestamp), -- timestamp with time zone
       pg_typeof(localtimestamp),    -- timestamp without time zone
       pg_typeof(current_date);      -- date
</code></pre><p>now() and current_timestamp are timestamptz and are safe. localtimestamp and current_date are evaluated in the session zone, which means a DEFAULT current_date column produces different answers for different clients. Be deliberate.</p>
<h4 id="so-which-do-youpick">So which do you pick?</h4>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-03.png"></p>
<p>The short version: <strong>use</strong> <strong>timestamptz for essentially everything.</strong> Reach for timestamp only when you genuinely mean a floating wall-clock time that must not shift — &ldquo;the store opens at 09:00 local, whatever local means at that branch&rdquo; — and store the zone ID beside it.</p>
<h3 id="part-3-time-zonesproperly">Part 3: Time Zones, Properly</h3>
<p>This is where the real complexity lives, so let’s slow down.</p>
<h4 id="three-things-called-timezone">Three things called “time zone”</h4>
<ol>
<li><strong>An offset</strong> — +01:00. A number.</li>
<li><strong>An IANA zone ID</strong> — Europe/London. A named set of historical and future rules. PostgreSQL ships ~499 of these in pg_timezone_names; Java reads the same tzdata.</li>
<li><strong>An abbreviation</strong> — EST, BST, IST. Ambiguous, unversioned, and a trap.</li>
</ol>
<p>That third one is genuinely dangerous:</p>
<pre tabindex="0"><code>SELECT &#39;2026-08-15 12:00:00 EST&#39;::timestamptz,               -- 2026-08-15 17:00:00+00
       &#39;2026-08-15 12:00:00 America/New_York&#39;::timestamptz;  -- 2026-08-15 16:00:00+00
</code></pre><p>One hour apart. EST is a <em>fixed</em> -05:00 in PostgreSQL — it does not mean &ldquo;Eastern time,&rdquo; it means &ldquo;Eastern Standard Time, always.&rdquo; In August, New York is on -04:00. If you accept zone abbreviations from user input, you have already shipped a bug. (IST is worse — it can mean India, Ireland, or Israel depending on who you ask.)</p>
<p><strong>Always use full IANA names.</strong></p>
<p>Offsets aren’t zones either. Nepal is +05:45; the Chatham Islands are +12:45:</p>
<pre tabindex="0"><code>ZonedDateTime.of(2026,8,15,12,0,0,0, ZoneId.of(&#34;Asia/Kathmandu&#34;))
// 2026-08-15T12:00+05:45[Asia/Kathmandu]
</code></pre><p>If your API contract says “offset in whole hours,” it’s wrong.</p>
<h3 id="where-postgresqls-session-zone-comesfrom">Where PostgreSQL’s session zone comes from</h3>
<p>The TimeZone GUC is a <strong>per-connection setting</strong>. Its value is decided by, in order: an explicit SET TimeZone, the connection&rsquo;s startup parameters, the postgresql.conf value, and finally the server&rsquo;s OS zone.</p>
<p>That last fallback is why the same query returns different results on a developer laptop and in CI. And with connection pooling, SET TimeZone on a borrowed connection leaks to whoever gets it next.</p>
<p>Pin it. Set timezone = &lsquo;UTC&rsquo; in postgresql.conf, and add it to the JDBC URL so it survives pool reuse:</p>
<pre tabindex="0"><code>jdbc:postgresql://host:5432/db?options=-c%20TimeZone%3DUTC
</code></pre><p>Then make the client responsible for rendering. Formatting a timestamp for a human is a presentation concern, not a database one.</p>
<h4 id="the-spring-forward-gap">The spring-forward gap</h4>
<p>On 29 March 2026, London jumps from 01:00 to 02:00. The local times between them <strong>do not exist</strong>.</p>
<p>Ask Java for one anyway:</p>
<pre tabindex="0"><code>LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30);
gap.atZone(ZoneId.of(&#34;Europe/London&#34;));
// 2026-03-29T02:30+01:00[Europe/London]
ZoneId.of(&#34;Europe/London&#34;).getRules().getTransition(gap);
// Transition[Gap at 2026-03-29T01:00Z to +01:00]
</code></pre><p>Java doesn’t throw. It silently pushes the value forward by the length of the gap. PostgreSQL does the same thing:</p>
<pre tabindex="0"><code>SET TimeZone=&#39;Europe/London&#39;;
SELECT &#39;2026-03-29 01:30:00&#39;::timestamptz;
-- 2026-03-29 02:30:00+01
</code></pre><p>Both are “correct” — but neither told you. If a user sets a 01:30 alarm on that date, you should be validating with getTransition() and telling them, not letting the platform quietly reschedule.</p>
<h4 id="the-fall-back-overlapwhere-java-and-postgresql-disagree">The fall-back overlap — where Java and PostgreSQL disagree</h4>
<p>On 25 October 2026, London goes from 02:00 back to 01:00. Local times between 01:00 and 02:00 happen <strong>twice</strong>, one hour apart.</p>
<p>Java’s rules expose both:</p>
<pre tabindex="0"><code>ZoneId london = ZoneId.of(&#34;Europe/London&#34;);
LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30);
london.getRules().getValidOffsets(overlap);
// [+01:00, Z]
ZonedDateTime earlier = overlap.atZone(london);
// 2026-10-25T01:30+01:00[Europe/London]  instant = 2026-10-25T00:30:00Z
ZonedDateTime later = earlier.withLaterOffsetAtOverlap();
// 2026-10-25T01:30Z[Europe/London]       instant = 2026-10-25T01:30:00Z
</code></pre><p>Java’s default for an ambiguous local time is the <strong>earlier</strong> offset — 00:30Z.</p>
<p>Now ask PostgreSQL the same question:</p>
<pre tabindex="0"><code>SELECT &#39;2026-10-25 01:30:00&#39;::timestamp AT TIME ZONE &#39;Europe/London&#39;;
-- 2026-10-25 01:30:00+00
</code></pre><p>PostgreSQL picked 01:30Z — the <strong>later</strong> one.</p>
<p>Same input, same tzdata, one hour apart. If your application resolves an ambiguous local time in Java and your reporting job resolves it in SQL, they will disagree for one hour, twice a year, in every DST zone.</p>
<p>There is no clever fix. The lesson is: <strong>never let a</strong> <strong>LocalDateTime cross the boundary and get resolved on the other side.</strong> Resolve to an Instant in exactly one place, with explicit overlap handling, and send the instant.</p>
<h4 id="dst-arithmetic-plusdays-is-not-plushours24">DST arithmetic: plusDays is not plusHours(24)</h4>
<pre tabindex="0"><code>ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 12, 0, 0, 0, london);
// 2026-03-28T12:00Z[Europe/London]
start.plusDays(1);                 // 2026-03-29T12:00+01:00  (23 real hours)
start.plus(Duration.ofDays(1));    // 2026-03-29T13:00+01:00  (24 real hours)
start.plusHours(24);               // 2026-03-29T13:00+01:00
</code></pre><p>plusDays preserves the wall clock. Duration preserves elapsed time. PostgreSQL draws the same line, using the same value:</p>
<pre tabindex="0"><code>SET TimeZone=&#39;Europe/London&#39;;
SELECT &#39;2026-03-28 12:00:00+00&#39;::timestamptz + interval &#39;1 day&#39;,   -- 2026-03-29 12:00:00+01
       &#39;2026-03-28 12:00:00+00&#39;::timestamptz + interval &#39;24 hours&#39;; -- 2026-03-29 13:00:00+01
</code></pre><p>This is one place the two platforms agree beautifully — interval &lsquo;1 day&rsquo; and plusDays(1) mean the same thing, and interval &lsquo;24 hours&rsquo; and Duration.ofDays(1) mean the same thing. &ldquo;Every day at 9am&rdquo; is plusDays / interval &lsquo;1 day&rsquo;. &ldquo;Every 24 hours&rdquo; is Duration / interval &lsquo;24 hours&rsquo;. Pick the one the product actually means.</p>
<h4 id="future-events-an-instant-is-notenough">Future events: an instant is not enough</h4>
<pre tabindex="0"><code>ZoneId ny = ZoneId.of(&#34;America/New_York&#34;);
ZonedDateTime.of(2027, 1, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-01-15T14:00Z
ZonedDateTime.of(2027, 7, 15, 9, 0, 0, 0, ny).withZoneSameInstant(UTC); // 2027-07-15T13:00Z
</code></pre><p>“9am in New York” is a different UTC instant depending on the date. So if a user schedules a recurring 9am standup and a government later changes DST rules — which happens several times a year somewhere — the stored instant becomes wrong.</p>
<p>For <strong>past events</strong>, store the instant. That’s history; it can’t change.</p>
<p>For <strong>future user-scheduled events</strong>, store three things:</p>
<pre tabindex="0"><code>CREATE TABLE meetings (
  id            bigserial PRIMARY KEY,
  local_start   timestamp   NOT NULL,  -- 2027-01-15 09:00
  zone_id       text        NOT NULL,  -- &#39;America/New_York&#39;
  computed_utc  timestamptz NOT NULL   -- cache; recompute on tzdata update
);
</code></pre><p>The first two are the truth. The third is a derived index for querying, refreshed when tzdata changes.</p>
<h4 id="tzdata-is-a-movingtarget">tzdata is a moving target</h4>
<p>Java bundles a copy of the IANA database (2025b on the JDK I ran this on); PostgreSQL bundles its own; your OS has a third. They drift. When Java and PostgreSQL are on different tzdata versions, they will resolve recently-changed zones differently.</p>
<p>Check what you have:</p>
<pre tabindex="0"><code>ZoneRulesProvider.getVersions(&#34;Europe/London&#34;).lastKey();  // 2025b
</code></pre><pre tabindex="0"><code>SELECT name, utc_offset, is_dst FROM pg_timezone_names WHERE name = &#39;Europe/London&#39;;
</code></pre><p>Pin tzdata in your base images, and treat a tzdata bump as a deployment that requires recomputing cached future instants.</p>
<h3 id="part-4-wiring-java-to-postgresql">Part 4: Wiring Java to PostgreSQL</h3>
<h4 id="the-mappingtable">The mapping table</h4>
<p>Per the <a href="https://jdbc.postgresql.org/documentation/query/">pgJDBC documentation</a>:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-04.png"></p>
<p>Note the omission, in the driver’s own words:</p>
<blockquote>
<p><em>ZonedDateTime, Instant and OffsetTime / TIME WITH TIME ZONE are not supported.</em></p></blockquote>
<p>So don’t hand the driver an Instant and hope — depending on the driver version you&rsquo;ll either get a Can&rsquo;t infer the SQL type to use for an instance of java.time.Instant exception or undocumented behavior. Convert explicitly at the boundary instead:</p>
<pre tabindex="0"><code>// write
ps.setObject(1, instant.atOffset(ZoneOffset.UTC));
// read
Instant i = rs.getObject(&#34;occurred_at&#34;, OffsetDateTime.class).toInstant();
</code></pre><p>Note that reading a timestamptz gives you an OffsetDateTime normalized to UTC — the driver&rsquo;s offset is always Z, because that&rsquo;s what the backend holds. The offset carries no information. Convert straight to Instant and don&rsquo;t be tempted to read meaning into it.</p>
<h4 id="avoid-settimestamp--gettimestamp">Avoid setTimestamp / getTimestamp</h4>
<p>The java.sql.Timestamp overloads convert using the <strong>JVM default time zone</strong> unless you pass an explicit Calendar. That makes your persistence layer&rsquo;s correctness depend on the container&rsquo;s TZ environment variable. Use the java.time setObject/getObject methods instead — they&rsquo;re unambiguous.</p>
<p>If you must keep the JVM zone from mattering at all, set it explicitly at startup:</p>
<pre tabindex="0"><code>-Duser.timezone=UTC
</code></pre><p>Belt and braces, not a substitute for correct types.</p>
<h4 id="precision-java-has-nanos-postgresql-hasmicros">Precision: Java has nanos, PostgreSQL has micros</h4>
<pre tabindex="0"><code>Instant.parse(&#34;2026-08-15T10:00:00.123456789Z&#34;)           // .123456789
    .truncatedTo(ChronoUnit.MICROS)                        // .123456
</code></pre><p>PostgreSQL stores microseconds. Those last three digits are gone on the round trip, and PostgreSQL may <strong>round</strong> rather than truncate:</p>
<pre tabindex="0"><code>SELECT &#39;2026-08-15 10:00:00.123456789&#39;::timestamptz;  -- 10:00:00.123457
</code></pre><p>If you have an equality test or an optimistic-lock check comparing an in-memory Instant to one that&rsquo;s been through the database, truncate to micros <em>before</em> you persist so both sides agree.</p>
<h4 id="dont-wrap-the-column-in-afunction">Don’t wrap the column in a function</h4>
<p>Because timestamptz is stored as UTC, &ldquo;give me everything that happened on 1 Feb in New York&rdquo; tempts you into this:</p>
<pre tabindex="0"><code>EXPLAIN SELECT count(*) FROM events
WHERE (occurred_at AT TIME ZONE &#39;America/New_York&#39;)::date = DATE &#39;2026-02-01&#39;;
--  Seq Scan on events
</code></pre><p>Index gone. Compute the boundary instants instead and do a half-open range scan:</p>
<pre tabindex="0"><code>SELECT (DATE &#39;2026-02-01&#39; + time &#39;00:00&#39;) AT TIME ZONE &#39;America/New_York&#39;,  -- 2026-02-01 05:00+00
       (DATE &#39;2026-02-02&#39; + time &#39;00:00&#39;) AT TIME ZONE &#39;America/New_York&#39;;  -- 2026-02-02 05:00+00
EXPLAIN SELECT count(*) FROM events
WHERE occurred_at &gt;= &#39;2026-02-01 05:00:00+00&#39;
  AND occurred_at &lt;  &#39;2026-02-02 05:00:00+00&#39;;
--  Index Only Scan using events_occurred_at_idx
</code></pre><p>Note that this “day” is 24 hours only because February has no DST transition. On a transition day it’s 23 or 25 — which the boundary-instant approach handles correctly and a naive + interval &lsquo;24 hours&rsquo; does not. Half-open ranges (&gt;= and &lt;) also dodge the microsecond-boundary problem that BETWEEN creates.</p>
<h3 id="part-5-jpa-and-hibernate">Part 5: JPA and Hibernate</h3>
<p>Hibernate 6 made zoned types a first-class concern via TimeZoneStorageType:</p>
<p><img loading="lazy" src="https://vitaliisukach.dev/posts/datetime-handling-in-java-and-postgresql-a-practical-guide/image-05.png"></p>
<p>For PostgreSQL, the practical consequence is settled by the database, not the ORM: timestamptz discards the offset, so under DEFAULT your OffsetDateTime round-trips as UTC. Which is fine — as long as you expected it.</p>
<p>Sensible defaults:</p>
<pre tabindex="0"><code>spring:
  jpa:
    properties:
      hibernate:
        jdbc:
          time_zone: UTC
        timezone:
          default_storage: NORMALIZE_UTC
</code></pre><p>And in the entity, be explicit about intent:</p>
<pre tabindex="0"><code>@Entity
public class Booking {
    
    // A moment. Maps to timestamptz.
    @Column(nullable = false)
    private Instant createdAt;
    
    // A date. Not a moment.
    @Column(nullable = false)
    private LocalDate serviceDate;
    
    // A wall-clock time that must not shift. Maps to timestamp.
    @Column(nullable = false)
    private LocalDateTime localStart;
    
    @Column(nullable = false)
    private String zoneId;   // &#39;Europe/London&#39;
    
    public ZonedDateTime start() {
        return localStart.atZone(ZoneId.of(zoneId));
    }
}
</code></pre><p>If you genuinely need the user’s original offset preserved, @TimeZoneStorage(TimeZoneStorageType.COLUMN) with @TimeZoneColumn writes it to a companion column. Use it when the offset is <em>data</em> — an audit trail of what the user&rsquo;s device claimed — not as a default.</p>
<h3 id="part-6-testing-thisstuff">Part 6: Testing This Stuff</h3>
<p>Datetime bugs are only cheap if you catch them before the clocks change. Three habits do most of the work.</p>
<h4 id="1-inject-aclock">1. Inject a Clock</h4>
<p>Instant.now() scattered through business logic is untestable. Inject java.time.Clock and you can move time at will:</p>
<pre tabindex="0"><code>@Bean Clock clock() { return Clock.systemUTC(); }
// in a test
Clock fixed = Clock.fixed(Instant.parse(&#34;2026-10-25T00:30:00Z&#34;), ZoneOffset.UTC);
</code></pre><h4 id="2-run-ci-in-a-hostile-timezone">2. Run CI in a hostile time zone</h4>
<p>If your JVM, your database, and your CI runner are all UTC, every zone bug hides until production. Force one of them somewhere awkward:</p>
<pre tabindex="0"><code>-Duser.timezone=Pacific/Chatham
</code></pre><p>+12:45 and it observes DST. If your test suite survives Chatham, it will survive most things. Do the same for the PostgreSQL container:</p>
<pre tabindex="0"><code>@Container
static PostgreSQLContainer&lt;?&gt; pg = new PostgreSQLContainer&lt;&gt;(&#34;postgres:16&#34;)
        .withEnv(&#34;TZ&#34;, &#34;Asia/Kathmandu&#34;)
        .withUrlParam(&#34;options&#34;, &#34;-c TimeZone=Asia/Kathmandu&#34;);
</code></pre><p>The point isn’t that production runs there — it’s that a test which only passes in UTC is a test that has proven nothing about zone handling.</p>
<h4 id="3-write-the-four-transition-tests">3. Write the four transition tests</h4>
<p>For any code that converts between local and instant, pin these explicitly:</p>
<pre tabindex="0"><code>static final ZoneId LONDON = ZoneId.of(&#34;Europe/London&#34;);
@Test void gapTimesAreRejectedNotSilentlyShifted() {
    LocalDateTime gap = LocalDateTime.of(2026, 3, 29, 1, 30);
    assertThat(LONDON.getRules().getValidOffsets(gap)).isEmpty();
    assertThatThrownBy(() -&gt; scheduler.schedule(gap, LONDON))
        .isInstanceOf(NonExistentLocalTimeException.class);
}
@Test void overlapTimesResolveToTheDocumentedChoice() {
    LocalDateTime overlap = LocalDateTime.of(2026, 10, 25, 1, 30);
    assertThat(LONDON.getRules().getValidOffsets(overlap)).hasSize(2);
    assertThat(scheduler.resolve(overlap, LONDON))
        .isEqualTo(Instant.parse(&#34;2026-10-25T00:30:00Z&#34;)); // earlier, by policy
}
@Test void dailyRecurrenceKeepsWallClockAcrossDst() {
    ZonedDateTime start = ZonedDateTime.of(2026, 3, 28, 9, 0, 0, 0, LONDON);
    assertThat(recurrence.next(start).toLocalTime()).isEqualTo(LocalTime.of(9, 0));
}
@Test void instantSurvivesTheDatabaseRoundTrip() {
    Instant original = Instant.now().truncatedTo(ChronoUnit.MICROS);
    repository.save(new Event(original));
    assertThat(repository.findLatest().occurredAt()).isEqualTo(original);
}
</code></pre><p>The fourth one catches more real bugs than you’d expect — it’s the one that fails when someone quietly changes a column from timestamptz to timestamp.</p>
<h3 id="the-shortversion">The Short Version</h3>
<p>If you skim one section, make it this one.</p>
<p><strong>Schema</strong></p>
<ul>
<li>timestamptz by default. It&rsquo;s the same 8 bytes and it&rsquo;s unambiguous.</li>
<li>timestamp only for wall-clock times that must not shift — and store the zone ID beside it.</li>
<li>date for dates. timetz for nothing.</li>
<li>Pin timezone = &lsquo;UTC&rsquo; in postgresql.conf and in the JDBC URL.</li>
</ul>
<p><strong>Java</strong></p>
<ul>
<li>Instant for things that happened. LocalDate for dates. ZonedDateTime for scheduling. OffsetDateTime at the JDBC boundary.</li>
<li>LocalDateTime is not a point in time. Treat every use as a question to answer, not an answer.</li>
<li>Never accept zone abbreviations. Full IANA names only.</li>
<li>plusDays ≠ plus(Duration.ofDays(1)). Know which one your product means.</li>
</ul>
<p><strong>The seam</strong></p>
<ul>
<li>pgJDBC speaks LocalDate/LocalTime/LocalDateTime/OffsetDateTime. Convert Instant explicitly.</li>
<li>Avoid setTimestamp/getTimestamp — they use the JVM default zone.</li>
<li>Truncate to microseconds before persisting.</li>
<li>Don’t wrap indexed timestamp columns in AT TIME ZONE; compute boundary instants instead.</li>
</ul>
<p><strong>And the one that bites hardest</strong></p>
<p>Java resolves an ambiguous local time to the <em>earlier</em> offset. PostgreSQL picks the <em>later</em> one. Resolve ambiguity in exactly one place, deliberately, and pass instants across the boundary — never local times.</p>
<p>Store the instant, render the local time, and keep the zone rules in exactly one layer. Most datetime bugs are just that rule, broken somewhere.</p>
<p><em>All examples verified on OpenJDK 21.0.10 (tzdata 2025b) and PostgreSQL 16.13.</em></p>
<p><strong>Sources</strong></p>
<ul>
<li><a href="https://www.postgresql.org/docs/current/datatype-datetime.html">PostgreSQL: Date/Time Types</a></li>
<li><a href="https://jdbc.postgresql.org/documentation/query/">pgJDBC: Using Java 8 Date and Time classes</a></li>
<li><a href="https://docs.hibernate.org/orm/6.4/javadocs/org/hibernate/annotations/TimeZoneStorageType.html">Hibernate</a> <a href="https://docs.hibernate.org/orm/6.4/javadocs/org/hibernate/annotations/TimeZoneStorageType.html">TimeZoneStorageType Javadoc</a></li>
<li><a href="https://docs.hibernate.org/orm/6.2/migration-guide/">Hibernate 6 migration guide</a></li>
<li><a href="https://thorben-janssen.com/hibernate-6-offsetdatetime-and-zoneddatetime/">Thorben Janssen: TimezoneStorageType</a></li>
</ul>
]]></content:encoded></item><item><title>Debugging Postgres: The System Tables That Actually Tell You What’s Wrong</title><link>https://vitaliisukach.dev/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/</link><pubDate>Sun, 02 Aug 2026 08:37:40 +0000</pubDate><guid>https://vitaliisukach.dev/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/</guid><description>&lt;p>&lt;img loading="lazy" src="https://vitaliisukach.dev/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/image-01.png">&lt;/p>
&lt;p>When a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.&lt;/p>
&lt;p>This is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been &amp;ldquo;running&amp;rdquo; for 25 minutes with no sign of finishing.&lt;/p></description><content:encoded><![CDATA[<p><img loading="lazy" src="https://vitaliisukach.dev/posts/debugging-postgres-the-system-tables-that-actually-tell-you-whats-wrong/image-01.png"></p>
<p>When a migration hangs, a query slows to a crawl, or a deploy just sits there, most people reach for the same first move: kill it and try again. That works about as often as it doesn’t. Postgres already knows what’s wrong — it’s just keeping the answer in a handful of system catalogs and stats views that almost nobody looks at until things are already on fire.</p>
<p>This is a walkthrough of the tables I actually check, in the order I check them, when something breaks — with a running example: a migration that adds a NOT NULL column and backfills it on a 20-million-row orders table, which has been &ldquo;running&rdquo; for 25 minutes with no sign of finishing.</p>
<h3 id="start-with-is-it-evenrunning">Start with: is it even running?</h3>
<p>Before touching statistics or query plans, find out if the process is alive, blocked, or just slow.</p>
<pre tabindex="0"><code>SELECT pid, state, wait_event_type, wait_event, query, query_start
FROM pg_stat_activity
WHERE state != &#39;idle&#39;;
</code></pre><p>pg_stat_activity is a live snapshot of every connection to the database. The columns that matter most here are wait_event_type and wait_event. If a process shows wait_event_type = &lsquo;Lock&rsquo;, it isn&rsquo;t broken — it&rsquo;s queued behind something else. If it shows Client or IO, that&rsquo;s a different problem entirely (waiting on the client, or waiting on disk).</p>
<p>For the orders migration, this comes back with:</p>
<pre tabindex="0"><code>pid  | state  | wait_event_type | wait_event | query
-----+--------+-----------------+------------+----------------------------------
8842 | active | Lock            | relation   | ALTER TABLE orders ADD COLUMN ...
</code></pre><p>wait_event_type = Lock tells us immediately: the migration isn&rsquo;t slow, it&rsquo;s stuck behind something else that holds a conflicting lock on orders.</p>
<p>If you see a lock wait, the next stop is obvious:</p>
<pre tabindex="0"><code>SELECT blocked_locks.pid AS blocked_pid,
       blocking_locks.pid AS blocking_pid,
       blocked_activity.query AS blocked_query,
       blocking_activity.query AS blocking_query
FROM pg_locks blocked_locks
JOIN pg_stat_activity blocked_activity ON blocked_activity.pid = blocked_locks.pid
JOIN pg_locks blocking_locks
  ON blocking_locks.locktype = blocked_locks.locktype
  AND blocking_locks.database IS NOT DISTINCT FROM blocked_locks.database
  AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
  AND blocking_locks.pid != blocked_locks.pid
JOIN pg_stat_activity blocking_activity ON blocking_activity.pid = blocking_locks.pid
WHERE NOT blocked_locks.granted;
</code></pre><p>This join is the single most useful query in Postgres debugging. It tells you exactly which session is blocking which, and what each of them is running. Most “stuck migration” incidents end here: some other transaction — often a long-running read, an ORM that forgot to close a connection, or a previous deploy that never finished — is sitting on a lock the migration needs.</p>
<p>Running it for the orders case turns up the problem:</p>
<pre tabindex="0"><code>blocked_pid | blocking_pid | blocked_query                   | blocking_query
------------+--------------+---------------------------------+---------------------------------------------
 8842       | 6110         | ALTER TABLE orders ADD COLUMN.. | SELECT * FROM orders WHERE ... (idle in txn)
</code></pre><p>PID 6110 is an analytics job that opened a transaction, ran a read, and never committed — “idle in transaction” — leaving an ACCESS SHARE lock on orders that blocks the migration&rsquo;s ACCESS EXCLUSIVE lock. The fix here isn&rsquo;t in the migration at all: it&rsquo;s killing or fixing PID 6110 (SELECT pg_terminate_backend(6110); if you&rsquo;re sure it&rsquo;s safe to end it).</p>
<h3 id="if-its-a-long-running-operation-check-its-actualprogress">If it’s a long-running operation, check its actual progress</h3>
<p>Postgres exposes live progress for the operations that most commonly make migrations feel frozen:</p>
<pre tabindex="0"><code>SELECT * FROM pg_stat_progress_create_index;
SELECT * FROM pg_stat_progress_vacuum;
SELECT * FROM pg_stat_progress_analyze;
</code></pre><p>If you’re running CREATE INDEX CONCURRENTLY as part of a migration, pg_stat_progress_create_index shows you the current phase (building index, index validation, etc.) and how many tuples it has scanned versus the total. This turns &ldquo;is this hung or just big?&rdquo; from a guess into a fact.</p>
<p>Say the next step of the migration adds an index on the new column instead of blocking on a lock — the progress view might show:</p>
<pre tabindex="0"><code>pid  | phase                         | tuples_total | tuples_done
-----+-------------------------------+--------------+-------------
9021 | building index: scanning table| 20450112     | 14300556
</code></pre><p>That’s roughly 70% through a single scan phase, moving steadily — not stuck, just genuinely working through 20 million rows. Compare that against a snapshot taken a minute later: if tuples_done hasn&rsquo;t moved, now you know it actually is stalled, and it&rsquo;s worth going back to pg_locks.</p>
<h3 id="if-nothings-blocked-suspect-the-queryplan">If nothing’s blocked, suspect the query plan</h3>
<p>Not every broken migration is a lock problem. Sometimes the migration runs, but a backfill step or an added index makes the planner choose a catastrophically bad plan. That’s a statistics problem, and it lives in two places.</p>
<p><strong>pg_stat_user_tables</strong> tells you whether Postgres has been keeping its stats up to date on the table in question:</p>
<pre tabindex="0"><code>SELECT relname, n_live_tup, n_dead_tup, n_mod_since_analyze,
       last_vacuum, last_autovacuum, last_analyze, last_autoanalyze
FROM pg_stat_user_tables
WHERE relname = &#39;your_table&#39;;
</code></pre><p>If n_mod_since_analyze is large relative to n_live_tup, or last_autoanalyze is suspiciously old, the planner is working off outdated information. This happens constantly during migrations, because a migration that inserts, updates, or deletes a large chunk of a table changes its shape faster than autovacuum&rsquo;s default thresholds react to.</p>
<p>For orders, after the backfill step runs:</p>
<pre tabindex="0"><code>relname | n_live_tup | n_dead_tup | n_mod_since_analyze | last_autoanalyze
--------+------------+------------+---------------------+-------------------
orders  | 20450112   | 3800000    | 18900000            | 2026-07-28 03:14
</code></pre><p>18.9 million modified rows since the last analyze, against 20 million live rows — nearly the entire table has changed and autovacuum hasn’t caught up yet (last_autoanalyze is five days old). Any query the migration runs after this point is planning against numbers that are effectively meaningless.</p>
<p><strong>pg_stats</strong> (the readable view over the raw pg_statistic catalog) shows what the planner actually believes about each column:</p>
<pre tabindex="0"><code>SELECT attname, n_distinct, correlation, null_frac, most_common_vals
FROM pg_stats
WHERE tablename = &#39;your_table&#39;;
</code></pre><p>n_distinct and correlation drive index-vs-sequential-scan decisions. If a migration adds a new index and the planner still refuses to use it, stale stats here are the usual suspect.</p>
<p>On the new orders.status column right after the backfill, before an ANALYZE:</p>
<pre tabindex="0"><code> attname | n_distinct | correlation | null_frac
---------+------------+-------------+-----------
 status  | 1          | 1           | 1.0
</code></pre><p>The planner still believes every row is null (null_frac = 1.0) because that was true before the backfill ran. Any query filtering on status will get a wildly wrong row estimate and likely pick a sequential scan or a bad join order. The fix is almost always to run ANALYZE your_table; — either before the migration, or as an explicit step inside it, rather than waiting for autovacuum to get around to it. After ANALYZE orders;, the same query returns something like n_distinct = 4, null_frac = 0.0 — matching reality, and the planner&rsquo;s decisions follow accordingly.</p>
<p><strong>pg_class</strong> holds the planner&rsquo;s cached size estimates directly:</p>
<pre tabindex="0"><code>SELECT relname, reltuples, relpages
FROM pg_class
WHERE relname = &#39;your_table&#39;;
</code></pre><p>reltuples is an estimate, not a live count — it&rsquo;s only refreshed by ANALYZE or VACUUM. If this number is wildly off from the real row count, every cost estimate downstream is wrong too. For orders, reltuples might read 20450112 while a SELECT count(*) returns closer to 24000000 — a four-million-row gap that skews every cost estimate the planner builds on top of it.</p>
<h3 id="if-you-need-the-exact-query-and-its-cost-not-just-thetable">If you need the exact query and its cost, not just the table</h3>
<p>pg_stat_statements (an extension, so it needs to be enabled) aggregates every distinct query the server has run, with real timing:</p>
<pre tabindex="0"><code>SELECT query, calls, total_exec_time, mean_exec_time, rows
FROM pg_stat_statements
ORDER BY total_exec_time DESC
LIMIT 20;
</code></pre><p>This is the difference between “the migration is slow” and “this specific UPDATE inside the migration is slow, and it’s been called 40,000 times.” For the orders backfill, this might show:</p>
<pre tabindex="0"><code>query                                        | calls | mean_exec_time | rows
---------------------------------------------+-------+----------------+--------
 UPDATE orders SET status = $1 WHERE id = $2 | 40000 | 812.4 ms       | 40000
</code></pre><p>40,000 single-row updates averaging 800ms each explains a lot on its own — that’s batching the backfill row-by-row instead of in bulk, and it would still be slow even with perfect statistics. If it’s enabled in your environment, it’s usually the fastest way to go from vague suspicion to a concrete line of SQL to fix — in this case, rewriting the backfill as a single set-based UPDATE &hellip; WHERE status IS NULL in batches, rather than one row at a time.</p>
<h3 id="a-rough-order-of-operations">A rough order of operations</h3>
<p>Put together, a sensible debugging sequence looks like this: check pg_stat_activity to see if the process is alive and what it&rsquo;s waiting on; if it&rsquo;s waiting on a lock, join into pg_locks to find the blocker; if it&rsquo;s a long index build or vacuum, check the matching pg_stat_progress_* view for real progress; if nothing is blocked but it&rsquo;s still slow, check pg_stat_user_tables and pg_stats for stale statistics, and pg_class for stale row estimates; and if pg_stat_statements is available, use it to find the exact query eating the time.</p>
<p>None of these tables are exotic. They ship with every Postgres install (aside from pg_stat_statements, which just needs CREATE EXTENSION). The only real skill here is knowing which one answers which question — and reaching for pg_stat_activity and pg_locks before assuming it&rsquo;s a statistics problem, since lock contention is a far more common cause of a &ldquo;broken&rdquo; migration than a bad query plan.</p>
]]></content:encoded></item></channel></rss>