MIME-Version: 1.0
References: <19490-9c59c6a583513b99@postgresql.org>
 <46FE61C9-F273-45FD-BED7-0F8CDA6EB992@yandex-team.ru>
 <CAL9smLBMxKBCmsA9UGcmf93bT2_MsZ+POH-oHREuwKdmMU7jfQ@mail.gmail.com>
 <46DB3CAB-EA1C-41A5-9D6D-5F913A2AAF66@yandex-team.ru>
In-Reply-To: <46DB3CAB-EA1C-41A5-9D6D-5F913A2AAF66@yandex-team.ru>
From: Ayush Tiwari <ayushtiwari.slg01@gmail.com>
Date: Thu, 21 May 2026 13:15:07 +0530
Message-ID: 
 <CAJTYsWV6X4D7eyg=WWLwF+ucq1BvvF4nffgy0XGxiA59ty1C6w@mail.gmail.com>
Subject: Re: BUG #19490: Streaming standby on 16.14 stops applying WAL on
 MultiXactOffsetSLRU when primary is 16.8
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Marko Tiikkaja <marko@joh.to>, radim@boringsql.com,
	PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="0000000000000229e006524f1869"
Archived-At: 
 <https://www.postgresql.org/message-id/CAJTYsWV6X4D7eyg%3DWWLwF%2Bucq1BvvF4nffgy0XGxiA59ty1C6w%40mail.gmail.com>
Precedence: bulk

--0000000000000229e006524f1869
Content-Type: text/plain; charset="UTF-8"

Hi,

On Thu, 21 May 2026 at 12:55, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

>
>
> > On 21 May 2026, at 00:12, Marko Tiikkaja <marko@joh.to> wrote:
> >
> > #8  0x0000654c8ae2acba in SimpleLruWriteAll (ctl=0x654c8b63e400
>
> Thanks!
>
> This clearly points to SimpleLruWriteAll() added in 77dff5d937b1.
> If by chance you will have a backtrace of another deadlocking process -
> please post it.
>
> But it's not strictly necessary for analysis, I think we can figure out
> what
> happened from the backtrace you already posted.
>

I had a look at the code that Marko's backtrace pointed at and I
believe this is a straightforward self-deadlock introduced by
77dff5d937b.

In RecordNewMultiXact() on REL_16_STABLE:

  LWLockAcquire(MultiXactOffsetSLRULock, LW_EXCLUSIVE);

  ...

  if (InRecovery && next_pageno != pageno)
  {
      ...
      if (last_initialized_offsets_page == -1)
      {
          SimpleLruWriteAll(MultiXactOffsetCtl, false);  /* <-- here */
          init_needed = !SimpleLruDoesPhysicalPageExist(MultiXactOffsetCtl,
next_pageno);
      }
      else
          init_needed = (last_initialized_offsets_page == pageno);
      ...
  }

The outer LWLockAcquire takes MultiXactOffsetSLRULock EXCLUSIVE.
SimpleLruWriteAll() in REL_16_STABLE then does

  LWLockAcquire(shared->ControlLock, LW_EXCLUSIVE);

and for the MultiXactOffsetCtl SLRU, shared->ControlLock is
MultiXactOffsetSLRULock (set up by SimpleLruInit(...
MultiXactOffsetSLRULock ...)).
So it tries to take the very lock the same backend already holds.
LWLockAcquire does not detect that and parks the process on
LWLock:MultiXactOffsetSLRU forever.

That matches every datum in the report:

  -  wait_event = LWLock:MultiXactOffsetSLRU.
   - pg_stat_slru shows zero MultiXact activity, because the
    SimpleLruWriteAll loop never gets past LWLockAcquire to actually
    write a page.
  - Restart unwedges things briefly.
  - The deadlock only triggers when last_initialized_offsets_page is
    still -1, i.e. before any XLOG_MULTIXACT_ZERO_OFF_PAGE record has
    been replayed in this recovery session, which is at most once per
    startup and consistent with the "recurs after catch-up" behaviour.

The "safety flush" the comment justifies is it needed?
Every offsets page that this code path initializes is synchronously written
via SimpleLruWritePage() a few lines below the SimpleLruZeroPage(),
with an Assert that the page is clean afterwards.  So at the moment
we call SimpleLruDoesPhysicalPageExist(), there shouldn't be a relevant
dirty offsets page in the SLRU buffer cache that would lead to a
false negative.  Dropping the SimpleLruWriteAll() call therefore
removes the self-deadlock without changing correctness.

Maybe I'm missing something here. Thoughts?

Regards,
Ayush

--0000000000000229e006524f1869
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr">Hi,<div><br></div></div><div class=3D"gma=
il_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Th=
u, 21 May 2026 at 12:55, Andrey Borodin &lt;<a href=3D"mailto:x4mmm@yandex-=
team.ru">x4mmm@yandex-team.ru</a>&gt; wrote:<br></div><blockquote class=3D"=
gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(20=
4,204,204);padding-left:1ex"><br>
<br>
&gt; On 21 May 2026, at 00:12, Marko Tiikkaja &lt;<a href=3D"mailto:marko@j=
oh.to" target=3D"_blank">marko@joh.to</a>&gt; wrote:<br>
&gt; <br>
&gt; #8=C2=A0 0x0000654c8ae2acba in SimpleLruWriteAll (ctl=3D0x654c8b63e400=
<br>
<br>
Thanks!<br>
<br>
This clearly points to SimpleLruWriteAll() added in 77dff5d937b1.<br>
If by chance you will have a backtrace of another deadlocking process -<br>
please post it.<br>
<br>
But it&#39;s not strictly necessary for analysis, I think we can figure out=
 what<br>
happened from the backtrace you already posted.<br></blockquote><div><br></=
div>I had a look at the code that Marko&#39;s backtrace pointed at and I<br=
>believe this is a straightforward self-deadlock introduced by<br>77dff5d93=
7b.<br><br>In RecordNewMultiXact() on REL_16_STABLE:<br><br>=C2=A0 LWLockAc=
quire(MultiXactOffsetSLRULock, LW_EXCLUSIVE);<br><br>=C2=A0 ...<br><br>=C2=
=A0 if (InRecovery &amp;&amp; next_pageno !=3D pageno)<br>=C2=A0 {<br>=C2=
=A0 =C2=A0 =C2=A0 ...<br>=C2=A0 =C2=A0 =C2=A0 if (last_initialized_offsets_=
page =3D=3D -1)<br>=C2=A0 =C2=A0 =C2=A0 {<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 SimpleLruWriteAll(MultiXactOffsetCtl, false); =C2=A0/* &lt;-- here *=
/<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 init_needed =3D !SimpleLruDoesPhysi=
calPageExist(MultiXactOffsetCtl, next_pageno);<br>=C2=A0 =C2=A0 =C2=A0 }<br=
>=C2=A0 =C2=A0 =C2=A0 else<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 init_neede=
d =3D (last_initialized_offsets_page =3D=3D pageno);<br>=C2=A0 =C2=A0 =C2=
=A0 ...<br>=C2=A0 }<br><br>The outer LWLockAcquire takes MultiXactOffsetSLR=
ULock EXCLUSIVE.<br>SimpleLruWriteAll() in REL_16_STABLE then does<br><br>=
=C2=A0 LWLockAcquire(shared-&gt;ControlLock, LW_EXCLUSIVE);<br><br>and for =
the MultiXactOffsetCtl SLRU, shared-&gt;ControlLock is<br>MultiXactOffsetSL=
RULock (set up by SimpleLruInit(... MultiXactOffsetSLRULock ...)).<br>So it=
 tries to take the very lock the same backend already holds.<br>LWLockAcqui=
re does not detect that and parks the process on<br>LWLock:MultiXactOffsetS=
LRU forever.<br><br>That matches every datum in the report:<br><br>=C2=A0 -=
=C2=A0 wait_event =3D LWLock:MultiXactOffsetSLRU.<br>=C2=A0 =C2=A0- pg_stat=
_slru shows zero MultiXact activity, because the<br>=C2=A0 =C2=A0 SimpleLru=
WriteAll loop never gets past LWLockAcquire to actually<br>=C2=A0 =C2=A0 wr=
ite a page.<br>=C2=A0 - Restart unwedges things briefly.<br>=C2=A0 - The de=
adlock only triggers when last_initialized_offsets_page is<br>=C2=A0 =C2=A0=
 still -1, i.e. before any XLOG_MULTIXACT_ZERO_OFF_PAGE record has<br>=C2=
=A0 =C2=A0 been replayed in this recovery session, which is at most once pe=
r<br>=C2=A0 =C2=A0 startup and consistent with the &quot;recurs after catch=
-up&quot; behaviour.<br><br>The &quot;safety flush&quot; the comment justif=
ies is it needed?<br>Every offsets page that this code path initializes is =
synchronously written<br>via SimpleLruWritePage() a few lines below the Sim=
pleLruZeroPage(),<br>with an Assert that the page is clean afterwards.=C2=
=A0 So at the moment<br>we call SimpleLruDoesPhysicalPageExist(), there sho=
uldn&#39;t be a relevant<br>dirty offsets page in the SLRU buffer cache tha=
t would lead to a<br>false negative.=C2=A0 Dropping the SimpleLruWriteAll()=
 call therefore<br>removes the self-deadlock without changing correctness.<=
/div><div class=3D"gmail_quote gmail_quote_container"><br></div><div class=
=3D"gmail_quote gmail_quote_container">Maybe I&#39;m missing something here=
. Thoughts?<br><br>Regards,<br><div>Ayush=C2=A0</div></div></div>

--0000000000000229e006524f1869--