MIME-Version: 1.0
References: <19354-eefe6d8b3e84f9f2@postgresql.org> <CA+TgmoaRGSezRaA7x00X495Qho8WGTzggbDSUt-JsruXceZWug@mail.gmail.com>
 <CA+zULE4L4rA2DLAcfy=eQL7w_ZexV4P5zpQRbP=_qrhJBEOzjg@mail.gmail.com>
 <2292889.1765846569@sss.pgh.pa.us> <CAE2r8H5vaSyaC_t1FcpHBo-BB_=SrFj7GFnOC-SxC6WDf5c9VA@mail.gmail.com>
 <CA+zULE47EXZOp7qKYODd+mjSgDiR-WX5ZNBkwdKnj-Zc0FT58w@mail.gmail.com>
 <CA+TgmoZaoc37ohnhF5inoPxWzfoznV483xQw8Fmw+ELFScv47g@mail.gmail.com>
 <2393116.1765899706@sss.pgh.pa.us> <CA+TgmoaoW4F2rRzYcQQim9ddT4-6H3oi0UYV9Ucw-rRQ5MdHsg@mail.gmail.com>
 <CA+hUKGKy-ViGBXdOjcPownBM=OdWiULO8H1RyH1r_8qNp=U4CA@mail.gmail.com>
 <6a8122ac-123d-4e93-9269-0b3be1e4a5a4@iki.fi> <CAAAe_zCLVunjt1u+2E86shwc3hk1x4bzUyU86nY1fq-nAVYN0Q@mail.gmail.com>
 <CA+hUKGJMrcS=hBkqVk=5pjM4w8edG=_ArASC82RqB6HQro-v-g@mail.gmail.com>
In-Reply-To: <CA+hUKGJMrcS=hBkqVk=5pjM4w8edG=_ArASC82RqB6HQro-v-g@mail.gmail.com>
Reply-To: assam258@gmail.com
From: Henson Choi <assam258@gmail.com>
Date: Wed, 15 Apr 2026 13:25:04 +0900
Message-ID: <CAAAe_zCwaccH7h+GOtHbo_docCY-o0c5NMRuYkdz15f=KL4f0g@mail.gmail.com>
Subject: Re: BUG #19354: JOHAB rejects valid byte sequences
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>, Robert Haas <robertmhaas@gmail.com>, Tom Lane <tgl@sss.pgh.pa.us>, 
	Jeroen Vermeulen <jtvjtv@gmail.com>, VASUKI M <vasukianand0119@gmail.com>, 
	pgsql-bugs@lists.postgresql.org
Content-Type: multipart/alternative; boundary="0000000000004d06df064f781a4e"
Archived-At: <https://www.postgresql.org/message-id/CAAAe_zCwaccH7h%2BGOtHbo_docCY-o0c5NMRuYkdz15f%3DKL4f0g%40mail.gmail.com>
Precedence: bulk

--0000000000004d06df064f781a4e
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

>
> 3.  UHC (=3D "Unified Hangul Code", invented by Microsoft): used EUR-KR
> as a base but supplied all possible pre-composed Hangul and 8,222
> Hanja (complete CJK as of Unicode 2.0).


Small correction: UHC's additions over EUC-KR are on the Hangul side,
not Hanja.  UHC adds 8,822 pre-composed Hangul (taking Hangul coverage
from EUC-KR's 2,350 up to the full 11,172) and leaves Hanja unchanged
at KS X 1001's 4,888.  I enumerated all three encodings against
PostgreSQL's current conversion tables to double-check:


    Encoding   Hangul   Hanja
    EUC_KR      2,350    4,888
    UHC        11,172    4,888
    JOHAB      11,172    4,888   (after this patch)


"Complete CJK as of Unicode 2.0" is off too -- Unicode 2.0's CJK
Unified Ideographs block had roughly 20,900 characters, so UHC and
JOHAB both carry only the KS X 1001 Hanja subset.  The 8,222 figure
looks like it got swapped with the 8,822 Hangul number.


>  Realpolitik that fed back into standards:

1.  The Hancom "Hangul" word processor used de facto standard JOHAB
> encoding, and dominated.
> 2.  KS X 1001 recognised this and added that annex.


Minor nit on the sequence: KS C 5601 already had a combinational annex
in its 1982 revision, but with a different bit layout from the one
Hancom's word processor used.  The 1992 revision swapped the annex's
bit layout to the commercial combinational form (=EC=83=81=EC=9A=A9 =EC=A1=
=B0=ED=95=A9=ED=98=95) that
the industry -- Hancom included -- had popularised.  The KS X
1001:2004 commentary documents this transition explicitly ("=EB=B9=84=ED=8A=
=B8
=EC=A1=B0=ED=95=A9=EC=9D=84 =EB=84=90=EB=A6=AC =EC=93=B0=EA=B3=A0 =EC=9E=88=
=EB=8A=94 =EC=9D=B4=EB=A5=B8=EB=B0=94 =EC=83=81=EC=9A=A9 =EC=A1=B0=ED=95=A9=
=ED=98=95=EC=9C=BC=EB=A1=9C =EB=B0=94=EA=BF=88").  So "KS
recognised the de facto standard" applies to 1992, not to the annex's
first appearance.


Worth mentioning for atmosphere: that period was the tail end of the
Apple II clone / MSX era and the rise of IBM PC compatibles in Korea,
and contemporary Korean computer magazines ran running debates over
Wansung vs Johab on three axes at once -- the encoding, the keyboard
layout (=EB=91=90=EB=B2=8C=EC=8B=9D vs =EC=84=B8=EB=B2=8C=EC=8B=9D, the Kor=
ean QWERTY-vs-Dvorak argument), and
the font rendering strategy (per-syllable bitmap tables for Wansung
vs jamo-composition for Johab) -- right alongside their game reviews.
The 1992 annex revision landed in the middle of that churn, not
ahead of it.


One further observation that fits your KS X 1002 note.  EUC-KR isn't
really a single standard but a layered stack -- KS X 1001 (the
character set) + ISO/IEC 2022 (the code-extension skeleton) + the
AT&T-era EUC convention of pinning G0 to ASCII and G1 to the 8-bit
region, later formalised in Korea as KS X 2901.  That informal
layering is precisely what let UHC land so easily: Microsoft extended
the same 8-bit region with additional Hangul, and every EUC-KR
decoder silently kept working for the covered subset.

KS X 1002 tried the opposite approach -- a formally separated
supplementary set, designated via a distinct ISO-2022 escape
sequence.  The design was cleaner on paper but required every
consumer to implement set-switching for a supplementary character
range that nobody was motivated to support.  UHC sidestepped this
entirely by just filling in the unused 8-bit slots.  So the
structural reason 1002 lost to UHC isn't just market power; it is
that UHC matched EUC-KR's informal extensibility while 1002 demanded
strict ISO-2022 compliance.  JOHAB (Annex 3) sits at the other end of
that spectrum -- a self-contained spec where a single document nails
down character set, byte layout, and composition algorithm, which is
what makes the verifier fix tractable.

A small downstream consequence of UHC's slot-filling approach is that
byte-wise comparison no longer matches Korean dictionary order: the
8,822 added Hangul land in the low 0x81-0xA0 range, ahead of the
gananada-ordered EUC-KR region.  Unicode's Hangul Syllables block
(U+AC00-U+D7A3) later restored that by assigning all 11,172 syllables
algorithmically in gananada order, so UTF-8 memcmp once again
produces Korean lexicographic order -- one of the quieter practical
drivers of Korea's Unicode migration.

Credit where it's due on that outcome: getting all 11,172 precomposed
Hangul into the BMP in algorithmic gananada order (the "Korean
Hangul Mess" cleanup in Unicode 2.0, 1996) wasn't inevitable.
Engineers at Microsoft's Korean office were notable advocates for
that arrangement alongside Korean standards-body contributors and
other vendors, and the Korean computing world has been quietly
benefiting from it ever since.  It's a nice detail given who's
reading this thread.


Everything else in the summary matches what I had -- thanks for the
independent write-up, and for taking another look at the patch.

> > The counter argument would be that you could use iconv
> > --from-code=3DJOHAB ..., or libiconv, or the codecs available in Python=
,
> > Java, etc for dealing with historical archived data, something that
> > data archivists must be very aware of.
>
> Sure.  But it's not comfortable to remove a user-visible feature
> we've had for decades.  My own primary concern about it was that a
> correct fix could require non-backwards-compatible behavior changes.
> Henson's analysis says that that's not a problem.  So assuming this
> patch withstands review, I'd be much happier to see it applied than
> to remove JOHAB.

Thank you -- the backward-compat angle was the hinge I was hoping
would carry, and I'm glad the analysis held up.  On the size of the
remaining audience: niche Korean standards have a small but stubborn
user base, much the way Dvorak users persist in the West.  There are
still =EC=84=B8=EB=B2=8C=EC=8B=9D (Sebeolsik) keyboard users in Korea who k=
eep hand-cut
stickers over their QWERTY-printed keycaps rather than switch back;
the JOHAB data holdouts are that kind of tail -- vanishingly small in
absolute numbers, but without a graceful alternative if we close the
door.  A correctly-working JOHAB serves that tail at near-zero
ongoing cost, which is ultimately what the patch is arguing for.

> No opinion at the moment about whether to back-patch.

Happy to defer on back-patching.  The behaviour change is strictly
additive (previously-rejected sequences start accepting, nothing is
reinterpreted), so the back-branches are technically safe, but v19-
only is a perfectly reasonable policy call if the project prefers
minimum surface area on the first cycle.

If you do want back-patches, I'm happy to produce per-branch
versions.  Given how long the JOHAB code has been stable (as noted
earlier in the thread), my feeling is that the same patch should
apply cleanly down to PG 14 without modification.  Happy to verify
that and post the set if it would help.


One personal aside: reading KS X 1001 Annex 3 end-to-end for this fix
turned out to be an unexpectedly cheerful detour -- it felt a bit
like cracking open a 6502 assembly reference from roughly the same
era.  Back then I also had a popular neural-networks book that
convinced teenage-me computers would never approach human cognition
because they could never match the brain's memory scale -- a
prediction that, looking around in 2026, has aged about as well as
you'd expect.  Thanks to everyone on the thread for making that
side-quest worthwhile.


Regards,
Henson

--0000000000004d06df064f781a4e
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr"><div dir=3D"ltr"><div class=3D"gmail_quote"><blockquote cl=
ass=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid=
 rgb(204,204,204);padding-left:1ex">3.=C2=A0 UHC (=3D &quot;Unified Hangul =
Code&quot;, invented by Microsoft): used EUR-KR<br>
as a base but supplied all possible pre-composed Hangul and 8,222<br>
Hanja (complete CJK as of Unicode 2.0).</blockquote><div><br></div><div>Sma=
ll correction: UHC&#39;s additions over EUC-KR are on the Hangul side,<br>n=
ot Hanja.=C2=A0 UHC adds 8,822 pre-composed Hangul (taking Hangul coverage<=
br>from EUC-KR&#39;s 2,350 up to the full 11,172) and leaves Hanja unchange=
d<br>at KS X 1001&#39;s 4,888.=C2=A0 I enumerated all three encodings again=
st<br>PostgreSQL&#39;s current conversion tables to double-check:<br><br><b=
r>=C2=A0 =C2=A0 Encoding =C2=A0 Hangul =C2=A0 Hanja<br>=C2=A0 =C2=A0 EUC_KR=
 =C2=A0 =C2=A0 =C2=A02,350 =C2=A0 =C2=A04,888<br>=C2=A0 =C2=A0 UHC =C2=A0 =
=C2=A0 =C2=A0 =C2=A011,172 =C2=A0 =C2=A04,888<br>=C2=A0 =C2=A0 JOHAB =C2=A0=
 =C2=A0 =C2=A011,172 =C2=A0 =C2=A04,888 =C2=A0 (after this patch)<br><br><b=
r>&quot;Complete CJK as of Unicode 2.0&quot; is off too -- Unicode 2.0&#39;=
s CJK<br>Unified Ideographs block had roughly 20,900 characters, so UHC and=
<br>JOHAB both carry only the KS X 1001 Hanja subset.=C2=A0 The 8,222 figur=
e<br>looks like it got swapped with the 8,822 Hangul number.<br></div><div>=
=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0=
.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">=C2=A0Realpol=
itik that fed back into standards:</blockquote><blockquote class=3D"gmail_q=
uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2=
04);padding-left:1ex">
1.=C2=A0 The Hancom &quot;Hangul&quot; word processor used de facto standar=
d JOHAB<br>
encoding, and dominated.<br>
2.=C2=A0 KS X 1001 recognised this and added that annex.</blockquote><div><=
br></div><div>Minor nit on the sequence: KS C 5601 already had a combinatio=
nal annex<br>in its 1982 revision, but with a different bit layout from the=
 one<br>Hancom&#39;s word processor used.=C2=A0 The 1992 revision swapped t=
he annex&#39;s<br>bit layout to the commercial combinational form (=EC=83=
=81=EC=9A=A9 =EC=A1=B0=ED=95=A9=ED=98=95) that<br>the industry -- Hancom in=
cluded -- had popularised.=C2=A0 The KS X<br>1001:2004 commentary documents=
 this transition explicitly (&quot;=EB=B9=84=ED=8A=B8<br>=EC=A1=B0=ED=95=A9=
=EC=9D=84 =EB=84=90=EB=A6=AC =EC=93=B0=EA=B3=A0 =EC=9E=88=EB=8A=94 =EC=9D=
=B4=EB=A5=B8=EB=B0=94 =EC=83=81=EC=9A=A9 =EC=A1=B0=ED=95=A9=ED=98=95=EC=9C=
=BC=EB=A1=9C =EB=B0=94=EA=BF=88&quot;).=C2=A0 So &quot;KS<br>recognised the=
 de facto standard&quot; applies to 1992, not to the annex&#39;s<br>first a=
ppearance.<br><br><br>Worth mentioning for atmosphere: that period was the =
tail end of the<br>Apple II clone / MSX era and the rise of IBM PC compatib=
les in Korea,<br>and contemporary Korean computer magazines ran running deb=
ates over<br>Wansung vs Johab on three axes at once -- the encoding, the ke=
yboard<br>layout (=EB=91=90=EB=B2=8C=EC=8B=9D vs =EC=84=B8=EB=B2=8C=EC=8B=
=9D, the Korean QWERTY-vs-Dvorak argument), and<br>the font rendering strat=
egy (per-syllable bitmap tables for Wansung<br>vs jamo-composition for Joha=
b) -- right alongside their game reviews.<br>The 1992 annex revision landed=
 in the middle of that churn, not<br>ahead of it.<br><br><br>One further ob=
servation that fits your KS X 1002 note.=C2=A0 EUC-KR isn&#39;t<br>really a=
 single standard but a layered stack -- KS X 1001 (the<br>character set) + =
ISO/IEC 2022 (the code-extension skeleton) + the<br>AT&amp;T-era EUC conven=
tion of pinning G0 to ASCII and G1 to the 8-bit<br>region, later formalised=
 in Korea as KS X 2901.=C2=A0 That informal<br>layering is precisely what l=
et UHC land so easily: Microsoft extended<br>the same 8-bit region with add=
itional Hangul, and every EUC-KR<br>decoder silently kept working for the c=
overed subset.<br><br>KS X 1002 tried the opposite approach -- a formally s=
eparated<br>supplementary set, designated via a distinct ISO-2022 escape<br=
>sequence.=C2=A0 The design was cleaner on paper but required every<br>cons=
umer to implement set-switching for a supplementary character<br>range that=
 nobody was motivated to support.=C2=A0 UHC sidestepped this<br>entirely by=
 just filling in the unused 8-bit slots.=C2=A0 So the<br>structural reason =
1002 lost to UHC isn&#39;t just market power; it is<br>that UHC matched EUC=
-KR&#39;s informal extensibility while 1002 demanded<br>strict ISO-2022 com=
pliance.=C2=A0 JOHAB (Annex 3) sits at the other end of<br>that spectrum --=
 a self-contained spec where a single document nails<br>down character set,=
 byte layout, and composition algorithm, which is<br>what makes the verifie=
r fix tractable.<br><br>A small downstream consequence of UHC&#39;s slot-fi=
lling approach is that<br>byte-wise comparison no longer matches Korean dic=
tionary order: the<br>8,822 added Hangul land in the low 0x81-0xA0 range, a=
head of the<br>gananada-ordered EUC-KR region.=C2=A0 Unicode&#39;s Hangul S=
yllables block<br>(U+AC00-U+D7A3) later restored that by assigning all 11,1=
72 syllables<br>algorithmically in gananada order, so UTF-8 memcmp once aga=
in<br>produces Korean lexicographic order -- one of the quieter practical<b=
r>drivers of Korea&#39;s Unicode migration.<br><br>Credit where it&#39;s du=
e on that outcome: getting all 11,172 precomposed<br>Hangul into the BMP in=
 algorithmic gananada order (the &quot;Korean<br>Hangul Mess&quot; cleanup =
in Unicode 2.0, 1996) wasn&#39;t inevitable.<br>Engineers at Microsoft&#39;=
s Korean office were notable advocates for<br>that arrangement alongside Ko=
rean standards-body contributors and<br>other vendors, and the Korean compu=
ting world has been quietly<br>benefiting from it ever since.=C2=A0 It&#39;=
s a nice detail given who&#39;s<br>reading this thread.<br><br><br>Everythi=
ng else in the summary matches what I had -- thanks for the<br>independent =
write-up, and for taking another look at the patch.<br></div><div><br></div=
><div>&gt; &gt; The counter argument would be that you could use iconv<br><=
/div><span>&gt;<span>=C2=A0</span>&gt; --from-code=3DJOHAB ..., or libiconv=
, or the codecs available in Python,<br>&gt;<span>=C2=A0</span>&gt; Java, e=
tc for dealing with historical archived data, something that<br>&gt;<span>=
=C2=A0</span>&gt; data archivists must be very aware of.<br>&gt;<span>=C2=
=A0</span><br></span>&gt;<span>=C2=A0</span>Sure.=C2=A0 But it&#39;s not co=
mfortable to remove a user-visible feature<br>&gt;<span>=C2=A0</span>we&#39=
;ve had for decades.=C2=A0 My own primary concern about it was that a<br>&g=
t;<span>=C2=A0</span>correct fix could require non-backwards-compatible beh=
avior changes.<br>&gt;<span>=C2=A0</span>Henson&#39;s analysis says that th=
at&#39;s not a problem.=C2=A0 So assuming this<br>&gt;<span>=C2=A0</span>pa=
tch withstands review, I&#39;d be much happier to see it applied than<br>&g=
t;<span>=C2=A0</span>to remove JOHAB.</div><div class=3D"gmail_quote"><br><=
/div><div class=3D"gmail_quote">Thank you -- the backward-compat angle was =
the hinge I was hoping<br>would carry, and I&#39;m glad the analysis held u=
p.=C2=A0 On the size of the<br>remaining audience: niche Korean standards h=
ave a small but stubborn<br>user base, much the way Dvorak users persist in=
 the West.=C2=A0 There are<br>still =EC=84=B8=EB=B2=8C=EC=8B=9D (Sebeolsik)=
 keyboard users in Korea who keep hand-cut<br>stickers over their QWERTY-pr=
inted keycaps rather than switch back;<br>the JOHAB data holdouts are that =
kind of tail -- vanishingly small in<br>absolute numbers, but without a gra=
ceful alternative if we close the<br>door.=C2=A0 A correctly-working JOHAB =
serves that tail at near-zero<br>ongoing cost, which is ultimately what the=
 patch is arguing for.<br><br>&gt; No opinion at the moment about whether t=
o back-patch.<br><br>Happy to defer on back-patching.=C2=A0 The behaviour c=
hange is strictly<br>additive (previously-rejected sequences start acceptin=
g, nothing is<br>reinterpreted), so the back-branches are technically safe,=
 but v19-<br>only is a perfectly reasonable policy call if the project pref=
ers<br>minimum surface area on the first cycle.<br><br>If you do want back-=
patches, I&#39;m happy to produce per-branch<br>versions.=C2=A0 Given how l=
ong the JOHAB code has been stable (as noted<br>earlier in the thread), my =
feeling is that the same patch should<br>apply cleanly down to PG 14 withou=
t modification.=C2=A0 Happy to verify<br>that and post the set if it would =
help.<br><br><br>One personal aside: reading KS X 1001 Annex 3 end-to-end f=
or this fix<br>turned out to be an unexpectedly cheerful detour -- it felt =
a bit<br>like cracking open a 6502 assembly reference from roughly the same=
<br>era.=C2=A0 Back then I also had a popular neural-networks book that<br>=
convinced teenage-me computers would never approach human cognition<br>beca=
use they could never match the brain&#39;s memory scale -- a<br>prediction =
that, looking around in 2026, has aged about as well as<br>you&#39;d expect=
.=C2=A0 Thanks to everyone on the thread for making that<br>side-quest wort=
hwhile.<br><br><br>Regards,<br>Henson<br></div></div>
</div>

--0000000000004d06df064f781a4e--