MIME-Version: 1.0
Content-Type: multipart/mixed;
 boundary="=_15be5a0026d1f02d26af8269971d5661"
Content-Transfer-Encoding: 8bit
Date: Mon, 06 Jan 2025 18:49:06 +0900
From: torikoshia <torikoshia@oss.nttdata.com>
To: Jelte Fennema-Nio <postgres@jeltef.nl>, tgl@sss.pgh.pa.us
Cc: Bruce Momjian <bruce@momjian.us>, Pgsql Hackers
 <pgsql-hackers@postgresql.org>
Subject: Re: RFC: Allow EXPLAIN to Output Page Fault Information
In-Reply-To: <D6PENGW1PL79.266GQNPSMRIGA@jeltef.nl>
References: <c20f6340eb26f3b736abc59471bfada8@oss.nttdata.com>
 <1614321.1735055528@sss.pgh.pa.us> <D6MJOHS7HZ80.3B17NDGUV6T47@jeltef.nl>
 <Z3LII9YEgp3kAZG-@momjian.us> <3861403.1735576764@sss.pgh.pa.us>
 <D6PENGW1PL79.266GQNPSMRIGA@jeltef.nl>
User-Agent: Roundcube Webmail/1.4.11
Message-ID: <1f22794321b745549d54359d343e37b8@oss.nttdata.com>
Archived-At: <https://www.postgresql.org/message-id/1f22794321b745549d54359d343e37b8%40oss.nttdata.com>
Precedence: bulk

--=_15be5a0026d1f02d26af8269971d5661
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset=UTF-8;
 format=flowed


On Tue, Dec 31, 2024 at 1:39 AM Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Bruce Momjian <bruce@momjian.us> writes:
> 
>> I certainly would love to see storage I/O numbers as distinct from
>> kernel read I/O numbers.
> 
> Me too, but I think it is 100% wishful thinking to imagine that
> page fault counts match up with that.  Maybe there are filesystems
> where a read that we request maps one-to-one with a subsequent
> page fault, but it hardly seems likely to me that that's
> universal.  Also, you can't tell page faults for reading program
> code apart from those for data, and you won't get any information
> at all about writes.

Thanks for the explanation.


On Tue, Dec 31, 2024 at 7:57 AM Jelte Fennema-Nio <postgres@jeltef.nl> 
wrote:
> On Mon Dec 30, 2024 at 5:39 PM CET, Tom Lane wrote:
>> Bruce Momjian <bruce@momjian.us> writes:
>>> I certainly would love to see storage I/O numbers as distinct from
>>> kernel read I/O numbers.
>> 
>> Me too, but I think it is 100% wishful thinking to imagine that
>> page fault counts match up with that.
> 
> Okay I played around with this patch a bit, in hopes of proving you
> wrong. But I now agree with you. I cannot seem to get any numbers out 
> of
> this that make sense.
> 
> The major page fault numbers are always zero, even after running:
> 
> echo 1 > /proc/sys/vm/drop_caches
> 
> If Takahori has a way to get some more useful insights from this patch,
> I'm quite interested in the steps he took (I might very well have 
> missed
> something obvious).

Thanks for testing.

I also did pg_ctl restart to clear buffercache in addition to your step 
and saw many major faults again.
However, when I replaced the restart with pg_buffercache_evict(), I also 
observed too few number of major fault.
I now feel majflt from getrusage() is not appropriate metrics for 
measuring storage I/O.


> **However, I think the general direction has merit**: Changing this 
> patch to
> use `ru_inblock`/`ru_oublock` gives very useful insights. `ru_inblock`
> is 0 when everything is in page cache, and it is very high when stuff 
> is
> not. I was only hacking around and basically did this:
> 
> s/ru_minflt/ru_inblock/g
> s/ru_majflt/ru_oublock/g

Great!
I misunderstood these metrics contain page cached I/O.

As far as I inspected, they come from read_bytes/write_bytes of 
task_io_accounting and the comment seems they are what we want, i.e. 
storage I/O:

    -- 
/usr/src/linux-headers-5.15.0-127/include/linux/task_io_accounting.h
    struct task_io_accounting {
     ..(snip)..
    #ifdef CONFIG_TASK_IO_ACCOUNTING
            /*
             * The number of bytes which this task has caused to be read 
from
             * storage.
             */
            u64 read_bytes;

            /*
             * The number of bytes which this task has caused, or shall 
cause to be
             * written to disk.
             */
            u64 write_bytes;

> Obviously more is needed. We'd probably want to show these numbers in
> useful units like MB or something. Also, maybe there's some better way
> of getting read/write numbers for the current process than
> ru_inblock/ru_oublock (but this one seems to work at least reasonably
> well).

Updated the PoC patch to calculate them by KB:

   =# EXPLAIN (ANALYZE, STORAGEIO) SELECT * FROM pgbench_accounts;
                                                              QUERY PLAN
   
---------------------------------------------------------------------------------------------------------------------------------
    Seq Scan on pgbench_accounts  (cost=0.00..263935.35 rows=10000035 
width=97) (actual time=1.447..3900.279 rows=10000000 loops=1)
      Buffers: shared hit=2587 read=161348
    Planning Time: 0.367 ms
    Execution:
      Storage I/O: read=1291856 KB write=0 KB
    Execution Time: 4353.253 ms
   (6 rows)


>  Also, maybe there's some better way
> of getting read/write numbers for the current process than
> ru_inblock/ru_oublock (but this one seems to work at least reasonably
> well).

Maybe, but as far as using getrusage(), ru_inblock and ru_outblock seem 
the best.

> One other thing that I noticed when playing around with this, which
> would need to be addressed: Parallel workers need to pass these values
> to the main process somehow, otherwise the IO from those processes gets 
> lost.

Yes.
I haven't developed it yet but I believe we can pass them like 
buffer/WAL usage.


-- 
Regards,

--
Atsushi Torikoshi
Seconded from NTT DATA GROUP CORPORATION to SRA OSS K.K.
--=_15be5a0026d1f02d26af8269971d5661
Content-Transfer-Encoding: base64
Content-Type: text/x-diff;
 name=v1-0001-PoC-Allow-EXPLAIN-to-output-storage-I-O-informati.patch
Content-Disposition: attachment;
 filename=v1-0001-PoC-Allow-EXPLAIN-to-output-storage-I-O-informati.patch;
 size=10748

RnJvbSA3Yzc2ZDYzODExMzgxZWU0MjUyZTAyMjg1ZjU4ZTA4MTZkY2YyNzNkIE1vbiBTZXAgMTcg
MDA6MDA6MDAgMjAwMQpGcm9tOiBBdHN1c2hpIFRvcmlrb3NoaSA8dG9yaWtvc2hpYUBvc3MubnR0
ZGF0YS5jb20+CkRhdGU6IE1vbiwgNiBKYW4gMjAyNSAxODozMjo0MCArMDkwMApTdWJqZWN0OiBb
UEFUQ0ggdjFdIFBvQzogQWxsb3cgRVhQTEFJTiB0byBvdXRwdXQgc3RvcmFnZSBJL08gaW5mb3Jt
YXRpb24KClRoaXMgcGF0Y2ggYWRkcyBTVE9SQUdFSU8gb3B0aW9uIHRvIEVYUExBSU4gYW5kIGl0
IHRyYWNrcyB0aGUKYW1vdW50IG9mIHJlYWQvd3JpdGUgZnJvbSBzdG9yYWdlIGR1cmluZyBwbGFu
bmluZyBhbmQgZXhlY3V0aW9uCnBoYXNlcyB1c2luZyBydV9pbmJsb2NrL3J1X291dGJsb2NrIGZy
b20gZ2V0cnVzYWdlKDIpLgoKVGhpcyBpcyBhIFBvQyBwYXRjaCBhbmQgb25seSBzaG93cyB0aGVt
IHdoZW4gdGhlIEZPUk1BVCBpcyBURVhULgpBY2N1bXVsYXRpbmcgUGFyYWxsZWwgd29ya2Vycycg
SS9PIGhhcyBub3QgaW1wbGVtZW50ZWQgeWV0LgpCdWlsZGluZyBvbiBXaW5kb3dzIGNhdXNlcyBl
cnJvci4KLS0tCiBzcmMvYmFja2VuZC9jb21tYW5kcy9leHBsYWluLmMgfCAxMDMgKysrKysrKysr
KysrKysrKysrKysrKysrKysrKysrKystCiBzcmMvYmFja2VuZC9jb21tYW5kcy9wcmVwYXJlLmMg
fCAgMjMgKysrKysrKysKIHNyYy9pbmNsdWRlL2NvbW1hbmRzL2V4cGxhaW4uaCB8ICAxMCArKysr
CiAzIGZpbGVzIGNoYW5nZWQsIDEzNCBpbnNlcnRpb25zKCspLCAyIGRlbGV0aW9ucygtKQoKZGlm
ZiAtLWdpdCBhL3NyYy9iYWNrZW5kL2NvbW1hbmRzL2V4cGxhaW4uYyBiL3NyYy9iYWNrZW5kL2Nv
bW1hbmRzL2V4cGxhaW4uYwppbmRleCBjMjRlNjZmODIuLjNjM2I2OGIzMSAxMDA2NDQKLS0tIGEv
c3JjL2JhY2tlbmQvY29tbWFuZHMvZXhwbGFpbi5jCisrKyBiL3NyYy9iYWNrZW5kL2NvbW1hbmRz
L2V4cGxhaW4uYwpAQCAtMTMsNiArMTMsOCBAQAogICovCiAjaW5jbHVkZSAicG9zdGdyZXMuaCIK
IAorI2luY2x1ZGUgPHN5cy9yZXNvdXJjZS5oPgorCiAjaW5jbHVkZSAiYWNjZXNzL3hhY3QuaCIK
ICNpbmNsdWRlICJjYXRhbG9nL3BnX3R5cGUuaCIKICNpbmNsdWRlICJjb21tYW5kcy9jcmVhdGVh
cy5oIgpAQCAtMTQ1LDYgKzE0Nyw4IEBAIHN0YXRpYyB2b2lkIHNob3dfZm9yZWlnbnNjYW5faW5m
byhGb3JlaWduU2NhblN0YXRlICpmc3N0YXRlLCBFeHBsYWluU3RhdGUgKmVzKTsKIHN0YXRpYyBj
b25zdCBjaGFyICpleHBsYWluX2dldF9pbmRleF9uYW1lKE9pZCBpbmRleElkKTsKIHN0YXRpYyBi
b29sIHBlZWtfYnVmZmVyX3VzYWdlKEV4cGxhaW5TdGF0ZSAqZXMsIGNvbnN0IEJ1ZmZlclVzYWdl
ICp1c2FnZSk7CiBzdGF0aWMgdm9pZCBzaG93X2J1ZmZlcl91c2FnZShFeHBsYWluU3RhdGUgKmVz
LCBjb25zdCBCdWZmZXJVc2FnZSAqdXNhZ2UpOworc3RhdGljIGJvb2wgcGVla19zdG9yYWdlaW8o
RXhwbGFpblN0YXRlICplcywgY29uc3QgU3RvcmFnZUlPICp1c2FnZSk7CitzdGF0aWMgdm9pZCBz
aG93X3N0b3JhZ2VpbyhFeHBsYWluU3RhdGUgKmVzLCBjb25zdCBTdG9yYWdlSU8gKnVzYWdlKTsK
IHN0YXRpYyB2b2lkIHNob3dfd2FsX3VzYWdlKEV4cGxhaW5TdGF0ZSAqZXMsIGNvbnN0IFdhbFVz
YWdlICp1c2FnZSk7CiBzdGF0aWMgdm9pZCBzaG93X21lbW9yeV9jb3VudGVycyhFeHBsYWluU3Rh
dGUgKmVzLAogCQkJCQkJCQkgY29uc3QgTWVtb3J5Q29udGV4dENvdW50ZXJzICptZW1fY291bnRl
cnMpOwpAQCAtMjE3LDYgKzIyMSw4IEBAIEV4cGxhaW5RdWVyeShQYXJzZVN0YXRlICpwc3RhdGUs
IEV4cGxhaW5TdG10ICpzdG10LAogCQkJYnVmZmVyc19zZXQgPSB0cnVlOwogCQkJZXMtPmJ1ZmZl
cnMgPSBkZWZHZXRCb29sZWFuKG9wdCk7CiAJCX0KKwkJZWxzZSBpZiAoc3RyY21wKG9wdC0+ZGVm
bmFtZSwgInN0b3JhZ2VpbyIpID09IDApCisJCQllcy0+c3RvcmFnZWlvID0gZGVmR2V0Qm9vbGVh
bihvcHQpOwogCQllbHNlIGlmIChzdHJjbXAob3B0LT5kZWZuYW1lLCAid2FsIikgPT0gMCkKIAkJ
CWVzLT53YWwgPSBkZWZHZXRCb29sZWFuKG9wdCk7CiAJCWVsc2UgaWYgKHN0cmNtcChvcHQtPmRl
Zm5hbWUsICJzZXR0aW5ncyIpID09IDApCkBAIC00NzUsNiArNDgxLDggQEAgc3RhbmRhcmRfRXhw
bGFpbk9uZVF1ZXJ5KFF1ZXJ5ICpxdWVyeSwgaW50IGN1cnNvck9wdGlvbnMsCiAJCQkJcGxhbmR1
cmF0aW9uOwogCUJ1ZmZlclVzYWdlIGJ1ZnVzYWdlX3N0YXJ0LAogCQkJCWJ1ZnVzYWdlOworCVN0
b3JhZ2VJTwlzdG9yYWdlaW8gPSB7MH07CisJU3RvcmFnZUlPCXN0b3JhZ2Vpb19zdGFydCA9IHsw
fTsKIAlNZW1vcnlDb250ZXh0Q291bnRlcnMgbWVtX2NvdW50ZXJzOwogCU1lbW9yeUNvbnRleHQg
cGxhbm5lcl9jdHggPSBOVUxMOwogCU1lbW9yeUNvbnRleHQgc2F2ZWRfY3R4ID0gTlVMTDsKQEAg
LTQ5OSw2ICs1MDcsMTUgQEAgc3RhbmRhcmRfRXhwbGFpbk9uZVF1ZXJ5KFF1ZXJ5ICpxdWVyeSwg
aW50IGN1cnNvck9wdGlvbnMsCiAJCWJ1ZnVzYWdlX3N0YXJ0ID0gcGdCdWZmZXJVc2FnZTsKIAlJ
TlNUUl9USU1FX1NFVF9DVVJSRU5UKHBsYW5zdGFydCk7CiAKKwlpZiAoZXMtPnN0b3JhZ2VpbykK
Kwl7CisJCXN0cnVjdCBydXNhZ2UgcnVzYWdlOworCisJCWdldHJ1c2FnZShSVVNBR0VfU0VMRiwg
JnJ1c2FnZSk7CisJCXN0b3JhZ2Vpb19zdGFydC5pbmJsb2NrID0gcnVzYWdlLnJ1X2luYmxvY2s7
CisJCXN0b3JhZ2Vpb19zdGFydC5vdXRibG9jayA9IHJ1c2FnZS5ydV9vdWJsb2NrOworCX0KKwog
CS8qIHBsYW4gdGhlIHF1ZXJ5ICovCiAJcGxhbiA9IHBnX3BsYW5fcXVlcnkocXVlcnksIHF1ZXJ5
U3RyaW5nLCBjdXJzb3JPcHRpb25zLCBwYXJhbXMpOwogCkBAIC01MTgsOSArNTM1LDE5IEBAIHN0
YW5kYXJkX0V4cGxhaW5PbmVRdWVyeShRdWVyeSAqcXVlcnksIGludCBjdXJzb3JPcHRpb25zLAog
CQlCdWZmZXJVc2FnZUFjY3VtRGlmZigmYnVmdXNhZ2UsICZwZ0J1ZmZlclVzYWdlLCAmYnVmdXNh
Z2Vfc3RhcnQpOwogCX0KIAorCWlmIChlcy0+c3RvcmFnZWlvKQorCXsKKwkJc3RydWN0IHJ1c2Fn
ZSBydXNhZ2U7CisKKwkJZ2V0cnVzYWdlKFJVU0FHRV9TRUxGLCAmcnVzYWdlKTsKKwkJc3RvcmFn
ZWlvLmluYmxvY2sgPSBydXNhZ2UucnVfaW5ibG9jayAtIHN0b3JhZ2Vpb19zdGFydC5pbmJsb2Nr
OworCQlzdG9yYWdlaW8ub3V0YmxvY2sgPSBydXNhZ2UucnVfb3VibG9jayAtIHN0b3JhZ2Vpb19z
dGFydC5vdXRibG9jazsKKwl9CisKIAkvKiBydW4gaXQgKGlmIG5lZWRlZCkgYW5kIHByb2R1Y2Ug
b3V0cHV0ICovCiAJRXhwbGFpbk9uZVBsYW4ocGxhbiwgaW50bywgZXMsIHF1ZXJ5U3RyaW5nLCBw
YXJhbXMsIHF1ZXJ5RW52LAogCQkJCSAgICZwbGFuZHVyYXRpb24sIChlcy0+YnVmZmVycyA/ICZi
dWZ1c2FnZSA6IE5VTEwpLAorCQkJCSAgIChlcy0+c3RvcmFnZWlvID8gJnN0b3JhZ2VpbyA6IE5V
TEwpLAogCQkJCSAgIGVzLT5tZW1vcnkgPyAmbWVtX2NvdW50ZXJzIDogTlVMTCk7CiB9CiAKQEAg
LTY0NCw3ICs2NzEsNyBAQCB2b2lkCiBFeHBsYWluT25lUGxhbihQbGFubmVkU3RtdCAqcGxhbm5l
ZHN0bXQsIEludG9DbGF1c2UgKmludG8sIEV4cGxhaW5TdGF0ZSAqZXMsCiAJCQkgICBjb25zdCBj
aGFyICpxdWVyeVN0cmluZywgUGFyYW1MaXN0SW5mbyBwYXJhbXMsCiAJCQkgICBRdWVyeUVudmly
b25tZW50ICpxdWVyeUVudiwgY29uc3QgaW5zdHJfdGltZSAqcGxhbmR1cmF0aW9uLAotCQkJICAg
Y29uc3QgQnVmZmVyVXNhZ2UgKmJ1ZnVzYWdlLAorCQkJICAgY29uc3QgQnVmZmVyVXNhZ2UgKmJ1
ZnVzYWdlLCBjb25zdCBTdG9yYWdlSU8gKnBsYW5zdG9yYWdlaW8sCiAJCQkgICBjb25zdCBNZW1v
cnlDb250ZXh0Q291bnRlcnMgKm1lbV9jb3VudGVycykKIHsKIAlEZXN0UmVjZWl2ZXIgKmRlc3Q7
CkBAIC02NTQsNiArNjgxLDkgQEAgRXhwbGFpbk9uZVBsYW4oUGxhbm5lZFN0bXQgKnBsYW5uZWRz
dG10LCBJbnRvQ2xhdXNlICppbnRvLCBFeHBsYWluU3RhdGUgKmVzLAogCWludAkJCWVmbGFnczsK
IAlpbnQJCQlpbnN0cnVtZW50X29wdGlvbiA9IDA7CiAJU2VyaWFsaXplTWV0cmljcyBzZXJpYWxp
emVNZXRyaWNzID0gezB9OworCVN0b3JhZ2VJTwlzdG9yYWdlaW8gPSB7MH07CisJU3RvcmFnZUlP
CXN0b3JhZ2Vpb19zdGFydCA9IHswfTsKKwlzdHJ1Y3QgcnVzYWdlIHJ1c2FnZTsKIAogCUFzc2Vy
dChwbGFubmVkc3RtdC0+Y29tbWFuZFR5cGUgIT0gQ01EX1VUSUxJVFkpOwogCkBAIC02NzQsNiAr
NzA0LDE0IEBAIEV4cGxhaW5PbmVQbGFuKFBsYW5uZWRTdG10ICpwbGFubmVkc3RtdCwgSW50b0Ns
YXVzZSAqaW50bywgRXhwbGFpblN0YXRlICplcywKIAkgKi8KIAlJTlNUUl9USU1FX1NFVF9DVVJS
RU5UKHN0YXJ0dGltZSk7CiAKKwlpZiAoZXMtPnN0b3JhZ2VpbykKKwl7CisJCWdldHJ1c2FnZShS
VVNBR0VfU0VMRiwgJnJ1c2FnZSk7CisKKwkJc3RvcmFnZWlvX3N0YXJ0LmluYmxvY2sgPSBydXNh
Z2UucnVfaW5ibG9jazsKKwkJc3RvcmFnZWlvX3N0YXJ0Lm91dGJsb2NrID0gcnVzYWdlLnJ1X291
YmxvY2s7CisJfQorCiAJLyoKIAkgKiBVc2UgYSBzbmFwc2hvdCB3aXRoIGFuIHVwZGF0ZWQgY29t
bWFuZCBJRCB0byBlbnN1cmUgdGhpcyBxdWVyeSBzZWVzCiAJICogcmVzdWx0cyBvZiBhbnkgcHJl
dmlvdXNseSBleGVjdXRlZCBxdWVyaWVzLgpAQCAtNzQ4LDcgKzc4Niw4IEBAIEV4cGxhaW5PbmVQ
bGFuKFBsYW5uZWRTdG10ICpwbGFubmVkc3RtdCwgSW50b0NsYXVzZSAqaW50bywgRXhwbGFpblN0
YXRlICplcywKIAlFeHBsYWluUHJpbnRQbGFuKGVzLCBxdWVyeURlc2MpOwogCiAJLyogU2hvdyBi
dWZmZXIgYW5kL29yIG1lbW9yeSB1c2FnZSBpbiBwbGFubmluZyAqLwotCWlmIChwZWVrX2J1ZmZl
cl91c2FnZShlcywgYnVmdXNhZ2UpIHx8IG1lbV9jb3VudGVycykKKwlpZiAocGVla19idWZmZXJf
dXNhZ2UoZXMsIGJ1ZnVzYWdlKSB8fCBwZWVrX3N0b3JhZ2VpbyhlcywgcGxhbnN0b3JhZ2Vpbykg
fHwKKwkJbWVtX2NvdW50ZXJzKQogCXsKIAkJRXhwbGFpbk9wZW5Hcm91cCgiUGxhbm5pbmciLCAi
UGxhbm5pbmciLCB0cnVlLCBlcyk7CiAKQEAgLTc2Miw2ICs4MDEsOSBAQCBFeHBsYWluT25lUGxh
bihQbGFubmVkU3RtdCAqcGxhbm5lZHN0bXQsIEludG9DbGF1c2UgKmludG8sIEV4cGxhaW5TdGF0
ZSAqZXMsCiAJCWlmIChidWZ1c2FnZSkKIAkJCXNob3dfYnVmZmVyX3VzYWdlKGVzLCBidWZ1c2Fn
ZSk7CiAKKwkJaWYgKGVzLT5zdG9yYWdlaW8pCisJCQlzaG93X3N0b3JhZ2VpbyhlcywgcGxhbnN0
b3JhZ2Vpbyk7CisKIAkJaWYgKG1lbV9jb3VudGVycykKIAkJCXNob3dfbWVtb3J5X2NvdW50ZXJz
KGVzLCBtZW1fY291bnRlcnMpOwogCkBAIC04MTMsNiArODU1LDI2IEBAIEV4cGxhaW5PbmVQbGFu
KFBsYW5uZWRTdG10ICpwbGFubmVkc3RtdCwgSW50b0NsYXVzZSAqaW50bywgRXhwbGFpblN0YXRl
ICplcywKIAogCXRvdGFsdGltZSArPSBlbGFwc2VkX3RpbWUoJnN0YXJ0dGltZSk7CiAKKwlpZiAo
ZXMtPnN0b3JhZ2VpbykKKwl7CisJCWdldHJ1c2FnZShSVVNBR0VfU0VMRiwgJnJ1c2FnZSk7CisK
KwkJc3RvcmFnZWlvLmluYmxvY2sgPSBydXNhZ2UucnVfaW5ibG9jayAtIHN0b3JhZ2Vpb19zdGFy
dC5pbmJsb2NrOworCQlzdG9yYWdlaW8ub3V0YmxvY2sgPSBydXNhZ2UucnVfb3VibG9jayAtIHN0
b3JhZ2Vpb19zdGFydC5vdXRibG9jazsKKworCQlpZiAoZXMtPmZvcm1hdCA9PSBFWFBMQUlOX0ZP
Uk1BVF9URVhUKQorCQl7CisJCQlFeHBsYWluSW5kZW50VGV4dChlcyk7CisJCQlhcHBlbmRTdHJp
bmdJbmZvU3RyaW5nKGVzLT5zdHIsICJFeGVjdXRpb246XG4iKTsKKwkJCWVzLT5pbmRlbnQrKzsK
KwkJfQorCQlzaG93X3N0b3JhZ2VpbyhlcywgJnN0b3JhZ2Vpbyk7CisKKwkJaWYgKGVzLT5mb3Jt
YXQgPT0gRVhQTEFJTl9GT1JNQVRfVEVYVCkKKwkJCWVzLT5pbmRlbnQtLTsKKwkJRXhwbGFpbkNs
b3NlR3JvdXAoIkV4ZWN1dGlvbiIsICJFeGVjdXRpb24iLCB0cnVlLCBlcyk7CisJfQorCiAJLyoK
IAkgKiBXZSBvbmx5IHJlcG9ydCBleGVjdXRpb24gdGltZSBpZiB3ZSBhY3R1YWxseSByYW4gdGhl
IHF1ZXJ5ICh0aGF0IGlzLAogCSAqIHRoZSB1c2VyIHNwZWNpZmllZCBBTkFMWVpFKSwgYW5kIGlm
IHN1bW1hcnkgcmVwb3J0aW5nIGlzIGVuYWJsZWQgKHRoZQpAQCAtNDIzMiw2ICs0Mjk0LDQzIEBA
IHNob3dfYnVmZmVyX3VzYWdlKEV4cGxhaW5TdGF0ZSAqZXMsIGNvbnN0IEJ1ZmZlclVzYWdlICp1
c2FnZSkKIAl9CiB9CiAKK3N0YXRpYyBib29sCitwZWVrX3N0b3JhZ2VpbyhFeHBsYWluU3RhdGUg
KmVzLCBjb25zdCBTdG9yYWdlSU8gKnVzYWdlKQoreworCWlmICh1c2FnZSA9PSBOVUxMKQorCQly
ZXR1cm4gZmFsc2U7CisKKwlpZiAodXNhZ2UtPmluYmxvY2sgPD0gMCAmJiB1c2FnZS0+b3V0Ymxv
Y2sgPD0gMCkKKwkJcmV0dXJuIGZhbHNlOworCisJZWxzZQorCQlyZXR1cm4gdHJ1ZTsKK30KKwor
LyoKKyAqIFNob3cgc3RvcmFnZSBJL08uCisgKgorICogU2luY2UgdGhlIHVuaXQgb2YgaW5ibG9j
ay9vdXRibG9jayBpcyA1MTIgYnl0ZXMsIGNoYW5nZSB0aGVtIHRvIEtCIGJ5IGRpdmlkaW5nIGJ5
IHR3by4KKyAqLworc3RhdGljIHZvaWQKK3Nob3dfc3RvcmFnZWlvKEV4cGxhaW5TdGF0ZSAqZXMs
IGNvbnN0IFN0b3JhZ2VJTyAqdXNhZ2UpCit7CisJLyogU2hvdyBvbmx5IHBvc2l0aXZlIGNvdW50
ZXIgdmFsdWVzLiAqLworCWlmICh1c2FnZS0+aW5ibG9jayA8PSAwICYmIHVzYWdlLT5vdXRibG9j
ayA8PSAwKQorCQlyZXR1cm47CisKKwlpZiAoZXMtPmZvcm1hdCA9PSBFWFBMQUlOX0ZPUk1BVF9U
RVhUKQorCXsKKworCQlFeHBsYWluSW5kZW50VGV4dChlcyk7CisJCWFwcGVuZFN0cmluZ0luZm9T
dHJpbmcoZXMtPnN0ciwgIlN0b3JhZ2UgSS9POiIpOworCQlhcHBlbmRTdHJpbmdJbmZvKGVzLT5z
dHIsICIgcmVhZD0lbGQgS0IiLCAobG9uZykgdXNhZ2UtPmluYmxvY2sgLyAyKTsKKwkJYXBwZW5k
U3RyaW5nSW5mbyhlcy0+c3RyLCAiIHdyaXRlPSVsZCBLQiIsIChsb25nKSB1c2FnZS0+b3V0Ymxv
Y2sgLyAyKTsKKworCQlhcHBlbmRTdHJpbmdJbmZvQ2hhcihlcy0+c3RyLCAnXG4nKTsKKwl9Cit9
CisKIC8qCiAgKiBTaG93IFdBTCB1c2FnZSBkZXRhaWxzLgogICovCmRpZmYgLS1naXQgYS9zcmMv
YmFja2VuZC9jb21tYW5kcy9wcmVwYXJlLmMgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9wcmVwYXJl
LmMKaW5kZXggODk4OWMwYzg4Li40NzQzYmFjZjQgMTAwNjQ0Ci0tLSBhL3NyYy9iYWNrZW5kL2Nv
bW1hbmRzL3ByZXBhcmUuYworKysgYi9zcmMvYmFja2VuZC9jb21tYW5kcy9wcmVwYXJlLmMKQEAg
LTE3LDYgKzE3LDcgQEAKICNpbmNsdWRlICJwb3N0Z3Jlcy5oIgogCiAjaW5jbHVkZSA8bGltaXRz
Lmg+CisjaW5jbHVkZSA8c3lzL3Jlc291cmNlLmg+CiAKICNpbmNsdWRlICJhY2Nlc3MveGFjdC5o
IgogI2luY2x1ZGUgImNhdGFsb2cvcGdfdHlwZS5oIgpAQCAtNTc5LDYgKzU4MCw4IEBAIEV4cGxh
aW5FeGVjdXRlUXVlcnkoRXhlY3V0ZVN0bXQgKmV4ZWNzdG10LCBJbnRvQ2xhdXNlICppbnRvLCBF
eHBsYWluU3RhdGUgKmVzLAogCWluc3RyX3RpbWUJcGxhbmR1cmF0aW9uOwogCUJ1ZmZlclVzYWdl
IGJ1ZnVzYWdlX3N0YXJ0LAogCQkJCWJ1ZnVzYWdlOworCVN0b3JhZ2VJTwlzdG9yYWdlaW9fc3Rh
cnQgPSB7MH07CisJU3RvcmFnZUlPCXN0b3JhZ2VpbyA9IHswfTsKIAlNZW1vcnlDb250ZXh0Q291
bnRlcnMgbWVtX2NvdW50ZXJzOwogCU1lbW9yeUNvbnRleHQgcGxhbm5lcl9jdHggPSBOVUxMOwog
CU1lbW9yeUNvbnRleHQgc2F2ZWRfY3R4ID0gTlVMTDsKQEAgLTU5NSw2ICs1OTgsMTYgQEAgRXhw
bGFpbkV4ZWN1dGVRdWVyeShFeGVjdXRlU3RtdCAqZXhlY3N0bXQsIEludG9DbGF1c2UgKmludG8s
IEV4cGxhaW5TdGF0ZSAqZXMsCiAKIAlpZiAoZXMtPmJ1ZmZlcnMpCiAJCWJ1ZnVzYWdlX3N0YXJ0
ID0gcGdCdWZmZXJVc2FnZTsKKworCWlmIChlcy0+c3RvcmFnZWlvKQorCXsKKwkJc3RydWN0IHJ1
c2FnZSBydXNhZ2U7CisKKwkJZ2V0cnVzYWdlKFJVU0FHRV9TRUxGLCAmcnVzYWdlKTsKKwkJc3Rv
cmFnZWlvX3N0YXJ0LmluYmxvY2sgPSBydXNhZ2UucnVfaW5ibG9jazsKKwkJc3RvcmFnZWlvX3N0
YXJ0Lm91dGJsb2NrID0gcnVzYWdlLnJ1X291YmxvY2s7CisJfQorCiAJSU5TVFJfVElNRV9TRVRf
Q1VSUkVOVChwbGFuc3RhcnQpOwogCiAJLyogTG9vayBpdCB1cCBpbiB0aGUgaGFzaCB0YWJsZSAq
LwpAQCAtNjQ2LDYgKzY1OSwxNSBAQCBFeHBsYWluRXhlY3V0ZVF1ZXJ5KEV4ZWN1dGVTdG10ICpl
eGVjc3RtdCwgSW50b0NsYXVzZSAqaW50bywgRXhwbGFpblN0YXRlICplcywKIAkJQnVmZmVyVXNh
Z2VBY2N1bURpZmYoJmJ1ZnVzYWdlLCAmcGdCdWZmZXJVc2FnZSwgJmJ1ZnVzYWdlX3N0YXJ0KTsK
IAl9CiAKKwlpZiAoZXMtPnN0b3JhZ2VpbykKKwl7CisJCXN0cnVjdCBydXNhZ2UgcnVzYWdlOwor
CisJCWdldHJ1c2FnZShSVVNBR0VfU0VMRiwgJnJ1c2FnZSk7CisJCXN0b3JhZ2Vpby5pbmJsb2Nr
ID0gcnVzYWdlLnJ1X2luYmxvY2sgLSBzdG9yYWdlaW9fc3RhcnQuaW5ibG9jazsKKwkJc3RvcmFn
ZWlvLm91dGJsb2NrID0gcnVzYWdlLnJ1X291YmxvY2sgLSBzdG9yYWdlaW9fc3RhcnQub3V0Ymxv
Y2s7CisJfQorCiAJcGxhbl9saXN0ID0gY3BsYW4tPnN0bXRfbGlzdDsKIAogCS8qIEV4cGxhaW4g
ZWFjaCBxdWVyeSAqLwpAQCAtNjU2LDYgKzY3OCw3IEBAIEV4cGxhaW5FeGVjdXRlUXVlcnkoRXhl
Y3V0ZVN0bXQgKmV4ZWNzdG10LCBJbnRvQ2xhdXNlICppbnRvLCBFeHBsYWluU3RhdGUgKmVzLAog
CQlpZiAocHN0bXQtPmNvbW1hbmRUeXBlICE9IENNRF9VVElMSVRZKQogCQkJRXhwbGFpbk9uZVBs
YW4ocHN0bXQsIGludG8sIGVzLCBxdWVyeV9zdHJpbmcsIHBhcmFtTEksIHBzdGF0ZS0+cF9xdWVy
eUVudiwKIAkJCQkJCSAgICZwbGFuZHVyYXRpb24sIChlcy0+YnVmZmVycyA/ICZidWZ1c2FnZSA6
IE5VTEwpLAorCQkJCQkJICAgKGVzLT5zdG9yYWdlaW8gPyAmc3RvcmFnZWlvIDogTlVMTCksCiAJ
CQkJCQkgICBlcy0+bWVtb3J5ID8gJm1lbV9jb3VudGVycyA6IE5VTEwpOwogCQllbHNlCiAJCQlF
eHBsYWluT25lVXRpbGl0eShwc3RtdC0+dXRpbGl0eVN0bXQsIGludG8sIGVzLCBwc3RhdGUsIHBh
cmFtTEkpOwpkaWZmIC0tZ2l0IGEvc3JjL2luY2x1ZGUvY29tbWFuZHMvZXhwbGFpbi5oIGIvc3Jj
L2luY2x1ZGUvY29tbWFuZHMvZXhwbGFpbi5oCmluZGV4IGVhNzQxOTk1MS4uN2I0NTJiMzA2IDEw
MDY0NAotLS0gYS9zcmMvaW5jbHVkZS9jb21tYW5kcy9leHBsYWluLmgKKysrIGIvc3JjL2luY2x1
ZGUvY29tbWFuZHMvZXhwbGFpbi5oCkBAIC00OSw2ICs0OSw3IEBAIHR5cGVkZWYgc3RydWN0IEV4
cGxhaW5TdGF0ZQogCWJvb2wJCWFuYWx5emU7CQkvKiBwcmludCBhY3R1YWwgdGltZXMgKi8KIAli
b29sCQljb3N0czsJCQkvKiBwcmludCBlc3RpbWF0ZWQgY29zdHMgKi8KIAlib29sCQlidWZmZXJz
OwkJLyogcHJpbnQgYnVmZmVyIHVzYWdlICovCisJYm9vbAkJc3RvcmFnZWlvOwkJLyogcHJpbnQg
c3RvcmFnZWlvICovCiAJYm9vbAkJd2FsOwkJCS8qIHByaW50IFdBTCB1c2FnZSAqLwogCWJvb2wJ
CXRpbWluZzsJCQkvKiBwcmludCBkZXRhaWxlZCBub2RlIHRpbWluZyAqLwogCWJvb2wJCXN1bW1h
cnk7CQkvKiBwcmludCB0b3RhbCBwbGFubmluZyBhbmQgZXhlY3V0aW9uIHRpbWluZyAqLwpAQCAt
NzMsNiArNzQsMTQgQEAgdHlwZWRlZiBzdHJ1Y3QgRXhwbGFpblN0YXRlCiAJRXhwbGFpbldvcmtl
cnNTdGF0ZSAqd29ya2Vyc19zdGF0ZTsgLyogbmVlZGVkIGlmIHBhcmFsbGVsIHBsYW4gKi8KIH0g
RXhwbGFpblN0YXRlOwogCit0eXBlZGVmIHN0cnVjdCBTdG9yYWdlSU8KK3sKKwkvKiBOb3RlIHRo
YXQgcGFnZSBzaXplIGhlcmUgaXMgNTEyIGJ5dGUgKi8KKwlsb25nCQlpbmJsb2NrOwkJCS8qICMg
b2YgcGFnZXMgdGhhdCBoYXMgYmVlbiByZWFkIGZyb20gc3RvcmFnZSAqLworCWxvbmcJCW91dGJs
b2NrOwkJCS8qICMgb2YgcGFnZXMgdGhhdCBoYXMgYmVlbiB3cml0dGVuIG9yIHNoYWxsIGJlIHdy
aXR0ZW4gdG8gc3RvcmFnZSAqLworfQkJCVN0b3JhZ2VJTzsKKworCiAvKiBIb29rIGZvciBwbHVn
aW5zIHRvIGdldCBjb250cm9sIGluIEV4cGxhaW5PbmVRdWVyeSgpICovCiB0eXBlZGVmIHZvaWQg
KCpFeHBsYWluT25lUXVlcnlfaG9va190eXBlKSAoUXVlcnkgKnF1ZXJ5LAogCQkJCQkJCQkJCSAg
IGludCBjdXJzb3JPcHRpb25zLApAQCAtMTA4LDYgKzExNyw3IEBAIGV4dGVybiB2b2lkIEV4cGxh
aW5PbmVQbGFuKFBsYW5uZWRTdG10ICpwbGFubmVkc3RtdCwgSW50b0NsYXVzZSAqaW50bywKIAkJ
CQkJCSAgIFBhcmFtTGlzdEluZm8gcGFyYW1zLCBRdWVyeUVudmlyb25tZW50ICpxdWVyeUVudiwK
IAkJCQkJCSAgIGNvbnN0IGluc3RyX3RpbWUgKnBsYW5kdXJhdGlvbiwKIAkJCQkJCSAgIGNvbnN0
IEJ1ZmZlclVzYWdlICpidWZ1c2FnZSwKKwkJCQkJCSAgIGNvbnN0IFN0b3JhZ2VJTyAqcGxhbnN0
b3JhZ2VpbywKIAkJCQkJCSAgIGNvbnN0IE1lbW9yeUNvbnRleHRDb3VudGVycyAqbWVtX2NvdW50
ZXJzKTsKIAogZXh0ZXJuIHZvaWQgRXhwbGFpblByaW50UGxhbihFeHBsYWluU3RhdGUgKmVzLCBR
dWVyeURlc2MgKnF1ZXJ5RGVzYyk7CgpiYXNlLWNvbW1pdDogYjFlZjQ4OTgwZGRkMDgyYTkwZWQz
OWFhNTkxNGFmNDVlNTMwNTllNgotLSAKMi4zNC4xCgo=
--=_15be5a0026d1f02d26af8269971d5661--