MIME-Version: 1.0
References: 
 <CAH5HC97p4kkpikar+swuC0Lx4YTVkE30sTsFX94tyzih7Cc_=w@mail.gmail.com>
 <CAMT0RQRPJUVsZbv6B=G6ccoZ+OM2vPOG85B6xDd+vKR6-0wt0g@mail.gmail.com>
 <CA+hUKGLL5o6hc4ZPvy=JMuFSKksQiqo0PfSsT5Kr2e30Pag8Hw@mail.gmail.com>
 <CAH5HC94_Miqg9wnu=wFBrDR0q97aE6-VO_A3ObPHTMQ5Xjki2g@mail.gmail.com>
 <CAH5HC95q5Omkh4kD1CmUC0FYrgt1Ck0eVRSGewoA9LB4=yWUgQ@mail.gmail.com>
 <CAFiTN-u7jcUkGpLybhGvMFXT6u=EPxRfE9qvEHDvUmJHL3Zs=w@mail.gmail.com>
 <CAH5HC95i8zcg5smHB=X3dmHt1Edi4uNSdDuH1y=kCNex5PV0_g@mail.gmail.com>
 <CAH5HC94CfrCMM-k4Dk8GD9NvBkV-Y8WwcmXV1HCyt4nz3C8EJA@mail.gmail.com>
 <CAH5HC95AkaAxLm-=FfCGY94QaUUrTwL1cB5JZUrD8Fn1v8KKaQ@mail.gmail.com>
 <CAH5HC95_6S4G_e5pHy=9DqSUNd9WR9e=_L1rQ7DSaeL8_+HXDQ@mail.gmail.com>
 <CAH5HC951bRwVKAwEvoef_d1ES_0atEsFqQNPXPiNOMmjcZ-7+A@mail.gmail.com>
 <CAKYtNApb_aL=psF+=VwjppTd717F4dGk_MVG9XW06OOW8LHRUQ@mail.gmail.com>
 <CAMT0RQRbO5tkubbX8OiSt9PY8wwKaAfT10FFp6SF4Od3Xhi18Q@mail.gmail.com>
 <CAH5HC94C+2jM5dikDzv-KkhbhOUz3Zm-Bd62Xycuh9XCPG2x8g@mail.gmail.com>
 <CAH5HC97skKgTkOBUipZjpOCXYc27txdzFL8UmVmBKaRvc4LO5A@mail.gmail.com>
 <CAH5HC95hD0tS2FKi8B6=o4g_Pf9WWbg8ZfD0Mu3m-h7ecfiK8A@mail.gmail.com>
 <CAH5HC97VKPYVbfbidyTPiiOZnxYmBKWm7EpWCFYkJvyiGjLepQ@mail.gmail.com>
 <CAH5HC94w4XA87UiKhgw9KKhCfVgU_p+ayT7jJF2gVEGvUibA_A@mail.gmail.com>
 <CAH5HC96kAB-HU-Nz0TMgD_V=+P7Y2f0drb1gYNCO9fidqyLhCA@mail.gmail.com>
 <CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com>
In-Reply-To: 
 <CAH5HC97_9JeDmCNL1im=ehuXFUghZpaXTy1pHo26J0J9LwzN6A@mail.gmail.com>
From: solai v <solai.cdac@gmail.com>
Date: Fri, 22 May 2026 16:04:23 +0530
Message-ID: 
 <CAF0whuc-zU-H4zPxERH7onw00xGUYaj6ZmUbOfvZWNusb9EtNg@mail.gmail.com>
Subject: Re: Adding pg_dump flag for parallel export to pipes
To: Nitin Motiani <nitinmotiani@google.com>
Cc: Hannu Krosing <hannuk@google.com>,
 Mahendra Singh Thalor <mahi6run@gmail.com>,
	Dilip Kumar <dilipbalaut@gmail.com>, Thomas Munro <thomas.munro@gmail.com>,
	pgsql-hackers@postgresql.org
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
Archived-At: 
 <https://www.postgresql.org/message-id/CAF0whuc-zU-H4zPxERH7onw00xGUYaj6ZmUbOfvZWNusb9EtNg%40mail.gmail.com>
Precedence: bulk

Hi all,

Thank you for the updated patch.

On Fri, May 22, 2026 at 1:03=E2=80=AFPM Nitin Motiani <nitinmotiani@google.=
com> wrote:
>
> Changed how pipe commands are quoted in the Windows test. The latest
> versions are attached.

I worked on reproducing the current limitation around parallel dumps
and then tested the latest v16 patch adding --pipe support for
pg_dump. To begin with, I verified the existing behavior.
For example:
pg_dump postgres | gzip > dump.sql.gz works, but does not support paralleli=
sm,
whereas:
pg_dump -Fd -j 4 -f dumpdir postgres
du -sh dumpdir
21M dumpdir
requires intermediate disk storage. This demonstrates the current
limitation where users must choose between parallelism and streaming
pipelines.
I then tested the patch introducing --pipe support. The feature is
quite useful for modern workflows where users want to stream dump
output directly to compression or upload pipelines without relying on
intermediate storage. Basic functionality worked as expected.
For example:
pg_dump -p 55432 -Fd -j 4 --pipe=3D"cat > dump.out" postgres, produced a
~38MB output file,
and:
pg_dump -p 55432 -Fd -j 4 --pipe=3D"gzip > dump.gz" postgres produced, a
compressed file (~11MB).
The initial contents appeared valid:
gunzip -c dump.gz | head
1
2
3
...
Also, no intermediate directory was created, confirming that the patch
enables streaming without filesystem-backed staging. Error handling
also behaved correctly.
For example:
--pipe=3D"invalid_cmd"
resulted in:
pg_dump: error: pipe command failed: command not found
and:
--pipe=3D"gzip | false"
resulted in:
pg_dump: error: pipe command failed: child process exited with exit code 1
However, I observed an important issue when using the feature with
multiple parallel workers. Since the pipe command is executed per
output file, using: --pipe=3D"gzip > dump.gz", it results in multiple
workers invoking independent gzip processes that all write to the same
output file. This leads to corrupted or truncated output.
In my testing:
gunzip -c dump.gz > dump.sql
failed with:
gzip: dump.gz: unexpected end of file
This suggests that concurrent writes to a shared output target are not
coordinated and can result in invalid dumps. It would be helpful to
clarify expected usage patterns here. For example: whether users are
expected to generate distinct outputs per worker, or whether
safeguards should be implemented to prevent multiple workers from
writing to the same destination. Additionally, during failure
scenarios I observed backend logs such as:
FATAL: connection to client lost
Broken pipe
While this is expected when the pipe terminates prematurely, it may be
worth considering whether error messaging or cleanup behavior can be
made clearer from the user perspective.
Overall, the feature is valuable and aligns well with modern backup
workflows. However, behavior in multi-worker scenarios with shared
pipe targets may need further clarification or safeguards to avoid
data corruption. Looking forward to more feedback.


Regards.
Solai