public inbox for pgsql-hackers@postgresql.org
help / color / mirror / Atom feedFrom: Amit Kapila <amit.kapila16@gmail.com>
To: Dilip Kumar <dilipbalaut@gmail.com>
Cc: shveta malik <shveta.malik@gmail.com>
Cc: Peter Smith <smithpb2250@gmail.com>
Cc: vignesh C <vignesh21@gmail.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>
Cc: Bharath Rupireddy <bharath.rupireddyforpostgres@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Proposal: Conflict log history table for Logical Replication
Date: Wed, 14 Jan 2026 15:59:31 +0530
Message-ID: <CAA4eK1Kf15UpNmpTTE2XyX=9PE_oTpOoy5xqg3rFWbxwwP4Rbg@mail.gmail.com> (raw)
In-Reply-To: <CAFiTN-tR8Rhs8uhfbck0Ac4dd1MopvvYgjK39nWyNXRp9Z3Qww@mail.gmail.com>
References: <CAFiTN-u5D5o_AGNbHRZHaOqAMWkxLf+hSk_r9X3gv6HbLOB5+g@mail.gmail.com>
<CALj2ACViThGQDYi-yeqUeHqG2Pozn2AiyvtDtjE6zhhbM0KsEA@mail.gmail.com>
<CAA4eK1+44b3vd_OWfiaVNtjf5Njb5cek09pmKRmttBByeg0NoA@mail.gmail.com>
<CAFiTN-v3L0WacCDx5dkOSonaZQbJfstXL4HrCPD1ahRdUsRnSg@mail.gmail.com>
<CALj2ACW63uuxh0fSoxEAF8OMWhz1dJKSkp268WJDzf5BUqCf5g@mail.gmail.com>
<CAFiTN-s9WWLOhW1TO27NtJwGf0bh2+MWyp3NEkZFeN_S5_p_rA@mail.gmail.com>
<CAA4eK1LxnsEx5sMbQkK5MHAgXKPROMQXQ0n=fKMwz+UsfKQaMQ@mail.gmail.com>
<CAD21AoDj+c4LXf2y4ESR-gVyv9d8V0G4R8R9pn-PcmT5zPzYcg@mail.gmail.com>
<CAA4eK1KokmAwNOL6bS-ip_E3F96PiQTjC4j-M+5vD1T6uUyi3Q@mail.gmail.com>
<CAFiTN-vFKE8E_N6h+peX9DP92mxCeFdm5A9Esn4DkLmNcZ-dOA@mail.gmail.com>
<CAFiTN-shLYf-fOTQ_dBf3Xfx05gxs_8d93MHZXyyz6w2Bg5geQ@mail.gmail.com>
<CAFiTN-tEgkKQHUikn6iBFCYf7XOObR7ncUq=OVh7WEk=6P4ymw@mail.gmail.com>
<CAFiTN-tQiakd8m+-d6WN6RpJXSv_JcropZ2oGzme4d1JudQhYg@mail.gmail.com>
<CAJpy0uDKbYWt+YPADj=4fHEvrGEWgnG1n_YsiGT_EZiZf0VSAw@mail.gmail.com>
<CAFiTN-t82BiXen+HfdR9jZyOpuSO92xonnUK=khXsiZWBfOxMA@mail.gmail.com>
<CAJpy0uAu2paxGAEffD=vaBTW9Jqbtxxawb8K8FgiASfeKPnGog@mail.gmail.com>
<CAJpy0uC0ZWgHOivJ102A1fMkppwK3RuSMafRPKyjwkmJrjhVUw@mail.gmail.com>
<CAFiTN-vFV9-zajrwjYHYyFnyQsooOAXW4CpxB5f-iT3APjOtoQ@mail.gmail.com>
<CAJpy0uBeU1dZgaqsSVKc=P=EVUKxRgVuHR8jDXFL-HLibbE-kQ@mail.gmail.com>
<CAA4eK1+FOkOxhzVLAnDymoNjp4i98H-L1+ZsWDgJEv-ndnTzTA@mail.gmail.com>
<CAFiTN-sVK6Bp+BawCJU_WpAXQSTX4OkKmce5EE4YNBgD-XSjZw@mail.gmail.com>
<CAA4eK1LbjV0bctib9wUnBpEkC+2rZFPnGuRtrKuc5AtUAzum+A@mail.gmail.com>
<CAFiTN-vq50N3QP9p3_SH+tJ8Pn=uRDb0X4qEcQZYcGW9AX88rQ@mail.gmail.com>
<CAFiTN-u3+zRGPESP5kUUfa6NxaWh1HL-gd1225KJ0Uvzi1urow@mail.gmail.com>
<CAA4eK1L4iNk6mNTC83PbYrRfUdtivH4U961PkdFfOO7mvc=USg@mail.gmail.com>
<CAFiTN-v+Mh64UfR5zb5rwgyGm6HS80XRSZ_XeaWkg8=+s9o3Kg@mail.gmail.com>
<CAFiTN-s3ZFHteQsiC3H4=AjTWxuwN-w69XQ3xL5X6YOMTua4pA@mail.gmail.com>
<CAJpy0uDe724nY59j-8hMapZ_Fru1Wo-NucF4Ea1B3Jrw=+J+UQ@mail.gmail.com>
<CAFiTN-uR=86L_5tyiA7n73EXCSCuDfQKfL5O=c8n7zZom8_ONQ@mail.gmail.com>
<CAD21AoDfOS-J0M9WbM3D20eGbSPzbfLQ-9XoYkxO4AZ9twqyvg@mail.gmail.com>
<CAFiTN-vMTg2X7vwfHLr5Gvy8ViV63_iaEcpHmM8V5GpA9-u8cg@mail.gmail.com>
<CAA4eK1+b2Ws0e_ZYJsgZAPn7VWndxAK_YM_QMKcfXst3e7F6Jg@mail.gmail.com>
<CAFiTN-v6hFKMPrSyTBsz=AtEETYMbOxrqvhZJsPQqKgQc4WCLw@mail.gmail.com>
<CAA4eK1KV3rYkaxys5fh-PtE9kq5xrFbiaRpOSPoRgQG494ek+g@mail.gmail.com>
<CAFiTN-utvu=QjY1QQ1a_TvkpkpvesMWo9M8wTFYLaOTPdpOJvw@mail.gmail.com>
<CAA4eK1+HoSOEqNwT3twArPNx4_D7hSUoEg2LnYhX8n9iUwhXgQ@mail.gmail.com>
<CAFiTN-tqmsfW0Sk=1RhzuduxqLrf9KEc8VOvBae+4aYxWTJwuA@mail.gmail.com>
<CAA4eK1JmCQ=DHe3HsqpX+P3mGDUd_Z7E7oAxdstK6822W6tuCw@mail.gmail.com>
<CAFiTN-uE4eAUYewuq3c5deAt3TtVork+H6rkUHRv68cOGr5rmQ@mail.gmail.com>
<CAFiTN-sJbhPX+LbA8YuQeYJpfGA2XA+OKXf8jCm04RoJOyzLvw@mail.gmail.com>
<CAJpy0uBPOyWj9itFjHzGXfrUuYS8KGmAvgdcV_9FPjWZ0EZz_w@mail.gmail.com>
<CAFiTN-s=iLE4qM4qmw9yXKqW09R_c_HqaSGeZXJ2EaTVfXss+g@mail.gmail.com>
<CAA4eK1KYo0vZpPSRc_4gVpa06-J39gxjs3tHFyckgkBfYJSfFA@mail.gmail.com>
<CAFiTN-vrKc6OWzrg6yvpwYcj79k=zkrDp3uwiZzjwrWLJAq6tw@mail.gmail.com>
<CAA4eK1LmvrfEgn1NUZZ=E3yMCjQdNZ5=_SBEry73-EmF6jM_PQ@mail.gmail.com>
<CAFiTN-vjfub5b3PqPQzfOw9BSjm8jt28ott+Hoz9CrRxJHzYkg@mail.gmail.com>
<CAFiTN-v=ANapYvRK+SOy2wJb4CSuD6Vb6_bTGuReM9Dv+3tucA@mail.gmail.com>
<CALDaNm1zEYoSdf2Ns-=UJRw95E5sbfpB0oaNUWtRJN27Q1Knhw@mail.gmail.com>
<CALDaNm3USsXVNBsfdpkp60HVgrTV4taWMk1xZYNBa7QUF=V0jg@mail.gmail.com>
<CAFiTN-sNg9ghLNkB2Kn0SwBGOub9acc99XZZU_d5NAcyW-yrEg@mail.gmail.com>
<CAJpy0uAF3EYcYdpTHdKMeXfvaPbNvnWrZUATrSLL1hqjao=33A@mail.gmail.com>
<CAFiTN-uikggCKp2LscTorKY5d3KF9j93DW0xebDcRX86G+ZsSw@mail.gmail.com>
<CAJpy0uDaOoVK8S3_xxTAcTDpfK1AY7tApw7nPOZG_gUz+DMi=Q@mail.gmail.com>
<CAA4eK1+AdeC5B9xrAXSKWGtTh-0d8xdD=fZttmOBm+c8o8thAQ@mail.gmail.com>
<CAFiTN-skBQAeuzuUd+PDK0Gqc8g+4x9ypBMwJhOrmW8ZCFKGSA@mail.gmail.com>
<CAJpy0uCdrsW5T+okq7xTOVxagje7FW3DOeY5B0CGKYa5VqF_tQ@mail.gmail.com>
<CAFiTN-u+_mFj9caYYFO7=_YHFXk5y=vvOm2H2=5hctYktmAVGA@mail.gmail.com>
<CALDaNm1aivk9KgQ5daeF6YZzuE+0wWc2yb7wb6qikNyvfPN0Sg@mail.gmail.com>
<CAJpy0uD6fTEUYJx3+yDbvB=VW7c5AaGoeSd7iwHdYYO=kYGn3g@mail.gmail.com>
<CALDaNm2YOOdJ25X1sJ+DYz37K6Qi4g0ZNFHb_pQMF9UqancnEA@mail.gmail.com>
<CAHut+PtMS5bENS0DVtBj+s3kUEOq61+hSkqLODjFB78egB0imQ@mail.gmail.com>
<CAFiTN-s_M83sfs+MHHbUrMesjsCPN4JWxY5MChCEiY1U-u7=9g@mail.gmail.com>
<CAFiTN-vj8NTm9w_L2XdhxJCub_RZw__YVUgfXa1B1kJzJctRNw@mail.gmail.com>
<CAJpy0uBDLnfhuSiev8W9ZMFNTzUmqhds2dKayUpLoN-z1dtsLA@mail.gmail.com>
<CAFiTN-uL9f0X+=Ep4BbAPvaTJA7S4XHM--G4BsnPJw4uJW7EGQ@mail.gmail.com>
<CAJpy0uDG=t-y_m8t1zpBzfz9viP3K8dyQgkruaraVT85UtTkrg@mail.gmail.com>
<CAFiTN-tR8Rhs8uhfbck0Ac4dd1MopvvYgjK39nWyNXRp9Z3Qww@mail.gmail.com>
On Tue, Jan 13, 2026 at 6:15 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
>
> On Tue, Jan 13, 2026 at 5:02 PM shveta malik <shveta.malik@gmail.com> wrote:
> >
> > On Tue, Jan 13, 2026 at 4:09 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > >
> > > On Tue, Jan 13, 2026 at 3:59 PM shveta malik <shveta.malik@gmail.com> wrote:
> > > >
> > > > On Sat, Jan 10, 2026 at 6:45 PM Dilip Kumar <dilipbalaut@gmail.com> wrote:
> > > > >
> > > > >>
> > > > > Here is the updated patch
> > > > >
> > > >
> > > > Thanks, I will review the code. Few concerns while experimenting with 001 alone:
> > > >
> > > > 1)
> > > > I am able to insert and update rows in pg_conflict.pg_conflict_16394.
> > > > It should be restricted.
> > > >
> > > > 2)
> > > > I think we need to block 'SELECT for UPDATE' and similar operations on
> > > > CLT. Currently it is allowed on CLT.
> > > > See this:
> > > >
> > > > postgres=# SELECT * FROM pg_toast.pg_toast_3466 for UPDATE;
> > > > ERROR: cannot lock rows in TOAST relation "pg_toast_3466"
> > > > postgres=# SELECT * FROM pg_conflict.pg_conflict_16394 for UPDATE;
> > > > ....
> > >
> > > I sent an analysis on this a few days ago but realized it only went to
> > > Amit. Resending to the full list:
> > >
> > > So the summary is, currently, all INSERT/UPDATE/DELETE operations are
> > > blocked for TOAST tables for safety. However, I have allowed these
> > > operations for the pg-conflict table. Unlike TOAST, the system does
> > > not internally reference conflict data, so user manipulation doesn't
> > > pose a safety risk to the system. If a user modifies or deletes this
> > > data, they simply lose their logs without impacting system stability.
> > > What are your thoughts on this approach?
> > >
> >
> > I don’t see a strong reason for a user to manually perform INSERT or
> > UPDATE here. But on rethinking, I also agree that allowing it does no
> > harm. It simply gives the user flexibility to modify data they “own”,
> > i.e., data that the system does not internally reference. Personally,
> > I’m okay with leaving it unrestricted, but it will be good to document
> > it as a NOTE/CAUTION, stating that these DML operations are allowed
> > and it is the user’s responsibility to manage any changes they make
> > manually.
>
> To clarify, I’m allowing INSERT and UPDATE alongside DELETE not
> because I anticipate a specific use case, but to avoid adding
> unnecessary code complexity. Since restricting these operations isn't
> a safety requirement, I felt it was better to keep the implementation
> simple rather than adding extra checks that don't provide real value.
>
What kind of complexity you are anticipating, can you show by top-up
patch? I think allowing just DELETE and TRUNCATE to table owners would
be ideal.
> So let's get consensus on this decision and then we can change accordingly.
>
> > >
> > > >> I Wrote
> > > > I have been exploring the enforcement of these restrictions.
> > > > Currently, DML is validated via CheckValidResultRel(). If we do not
> > > > explicitly check for the conflict log table in that function, DML
> > > > operations, including DELETE, will be permitted. While I am not overly
> > > > concerned about users attempting manual INSERT or UPDATE operations.
> > >
> > > > Now for TRUNCATE, the truncate_check_rel() currently relies on
> > > > IsSystemClass() to restrict truncations. Since the conflict namespace
> > > > is included in IsSystemClass() to prevent unauthorized DDL, TRUNCATE
> > > > is blocked by default. We could allow it by adding an exception in
> > > > truncate_check_rel() (e.g., IsSystemClass() &&
> > > > !IsConflictNamespace()), but I am unsure if this is necessary. Do we
> > > > really need to permit TRUNCATE, or is allowing DELETE sufficient for
> > > > user-driven cleanup?
> > >
> >
> > I am okay if we allow DELETE alone.
>
> Thanks for your opinion.
>
> > > >
> > > >
> > > > 3)
> > > > We currently skip ANALYZE on TOAST tables, but I’m not sure whether
> > > > the same restriction should apply to CLT. Since users are expected to
> > > > query CLT frequently, collecting statistics could be beneficial. Even
> > > > though there are currently no indexes or other structures to enable
> > > > more optimal plans, having statistics should not harm. Thoughts?
> > > >
> > > > postgres=# analyze pg_toast.pg_toast_16399;
> > > > WARNING: skipping "pg_toast_16399" --- cannot analyze non-tables or
> > > > special system tables
> > > >
> > > > postgres=# analyze pg_conflict.pg_conflict_16394;
> > > > ANALYZE
> > >
> > > I think its good to ANALYZE CLT data because user logically never need
> > > to query the toast data, but they would have to query the conflict
> > > data.
> >
> > Right. Do you think we shall allow index creation as well on this
> > table? It may help if the table is huge. As an example, a user can
> > create an index on relid to query it faster. Currently it is not
> > allowed.
>
> That’s an interesting point. I initially considered creating internal
> indexes to simplify management and avoid requiring users to have DDL
> permissions on the pg_conflict schema. However, I realized that
> predefined indexes might be too restrictive for diverse search
> requirements. Perhaps we should omit indexes from the initial
> version? We can then decide whether to add predefined indexes or allow
> external ones based on actual usage patterns and feedback. Thoughts?
>
I agree that the indexes could be considered later unless required by
internal code to search the conflict table.
--
With Regards,
Amit Kapila.
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: amit.kapila16@gmail.com, dilipbalaut@gmail.com, shveta.malik@gmail.com, smithpb2250@gmail.com, vignesh21@gmail.com, sawada.mshk@gmail.com, bharath.rupireddyforpostgres@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Proposal: Conflict log history table for Logical Replication
In-Reply-To: <CAA4eK1Kf15UpNmpTTE2XyX=9PE_oTpOoy5xqg3rFWbxwwP4Rbg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox