MIME-Version: 1.0
References: <19490-9c59c6a583513b99@postgresql.org>
 <46FE61C9-F273-45FD-BED7-0F8CDA6EB992@yandex-team.ru>
 <CAL9smLBMxKBCmsA9UGcmf93bT2_MsZ+POH-oHREuwKdmMU7jfQ@mail.gmail.com>
 <46DB3CAB-EA1C-41A5-9D6D-5F913A2AAF66@yandex-team.ru>
In-Reply-To: <46DB3CAB-EA1C-41A5-9D6D-5F913A2AAF66@yandex-team.ru>
From: Radim Marek <radim@boringsql.com>
Date: Thu, 21 May 2026 10:34:49 +0200
Message-ID: 
 <CAJgoLkJfFgL-V+pYB7=R81AbURTE6sMhzVHDQDhVGnfXRSJ9Wg@mail.gmail.com>
Subject: Re: BUG #19490: Streaming standby on 16.14 stops applying WAL on
 MultiXactOffsetSLRU when primary is 16.8
To: Andrey Borodin <x4mmm@yandex-team.ru>
Cc: Marko Tiikkaja <marko@joh.to>,
	PostgreSQL mailing lists <pgsql-bugs@lists.postgresql.org>
Content-Type: multipart/alternative; boundary="000000000000d0c20506524fc986"
Archived-At: 
 <https://www.postgresql.org/message-id/CAJgoLkJfFgL-V%2BpYB7%3DR81AbURTE6sMhzVHDQDhVGnfXRSJ9Wg%40mail.gmail.com>
Precedence: bulk

--000000000000d0c20506524fc986
Content-Type: text/plain; charset="UTF-8"

Thank you for the follow-up. In mean-time I can confirm the
commit 77dff5d937b1 might be the source of the original reported issue.

Unfortunately pinning version down to 16.12 only avoids the
MultiXactOffsetSLRU self-deadlock, but the standby then fails recovery
after 12+ hours.

FATAL: could not access status of transaction 24958976 DETAIL: Could not
read from file "pg_multixact/offsets/017C" at offset 221184: read too few
bytes. CONTEXT: WAL redo at 14770/873268E8 for MultiXact/CREATE_ID:
24958975 offset 61500431 nmembers 2: 3058927188 (fornokeyupd) 3058927189
(keysh)

We are going to try to pin 16.13 and try that before we can safely upgrade
of the primary/are confident we have working PITR recovery available should
we need it.

Radim

PS: Once I have some time I will try to setup a docker based harness to be
able to replicate original problem for later testing of the fix.

On Thu, 21 May 2026 at 09:25, Andrey Borodin <x4mmm@yandex-team.ru> wrote:

>
>
> > On 21 May 2026, at 00:12, Marko Tiikkaja <marko@joh.to> wrote:
> >
> > #8  0x0000654c8ae2acba in SimpleLruWriteAll (ctl=0x654c8b63e400
>
> Thanks!
>
> This clearly points to SimpleLruWriteAll() added in 77dff5d937b1.
> If by chance you will have a backtrace of another deadlocking process -
> please post it.
>
> But it's not strictly necessary for analysis, I think we can figure out
> what
> happened from the backtrace you already posted.
>
>
> Best regards, Andrey Borodin.
>

--000000000000d0c20506524fc986
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div>Thank you for the follow-up. In mean-time I can confi=
rm the commit=C2=A077dff5d937b1 might be the source of the original reporte=
d issue.</div><div><br></div><div>Unfortunately pinning version down to 16.=
12 only=C2=A0avoids the MultiXactOffsetSLRU self-deadlock, but the standby =
then fails recovery after 12+ hours.</div><div><br></div><div><div style=3D=
"font-family:monospace;white-space:pre">  FATAL:  could not access status o=
f transaction 24958976
  DETAIL:  Could not read from file &quot;pg_multixact/offsets/017C&quot; a=
t offset 221184: read too few bytes.
  CONTEXT:  WAL redo at 14770/873268E8 for MultiXact/CREATE_ID: 24958975 of=
fset 61500431 nmembers 2: 3058927188 (fornokeyupd) 3058927189 (keysh)</div>=
<div style=3D"font-family:monospace;white-space:pre"><br></div>We are going=
 to try to pin 16.13 and try that before we can safely upgrade of the prima=
ry/are confident we have working PITR recovery available should we need it.=
</div><div><br></div><div>Radim</div><div><br></div><div>PS: Once I have so=
me time I will try to setup a docker based harness to be able to replicate =
original problem for later testing of the fix.</div></div><br><div class=3D=
"gmail_quote gmail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">O=
n Thu, 21 May 2026 at 09:25, Andrey Borodin &lt;<a href=3D"mailto:x4mmm@yan=
dex-team.ru">x4mmm@yandex-team.ru</a>&gt; wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rg=
b(204,204,204);padding-left:1ex"><br>
<br>
&gt; On 21 May 2026, at 00:12, Marko Tiikkaja &lt;<a href=3D"mailto:marko@j=
oh.to" target=3D"_blank">marko@joh.to</a>&gt; wrote:<br>
&gt; <br>
&gt; #8=C2=A0 0x0000654c8ae2acba in SimpleLruWriteAll (ctl=3D0x654c8b63e400=
<br>
<br>
Thanks!<br>
<br>
This clearly points to SimpleLruWriteAll() added in 77dff5d937b1.<br>
If by chance you will have a backtrace of another deadlocking process -<br>
please post it.<br>
<br>
But it&#39;s not strictly necessary for analysis, I think we can figure out=
 what<br>
happened from the backtrace you already posted.<br>
<br>
<br>
Best regards, Andrey Borodin.<br>
</blockquote></div>

--000000000000d0c20506524fc986--