TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

ISHIKAWA,chiaki
(This is a repost of
https://bugzilla.mozilla.org/show_bug.cgi?id=1242030#c234

I think it merits wider discussion, thus posting it here. I know that
the mailer may mungle the text formatting. Please see the above URL in
that case.)

Performance analysis when we remove the caching of open stream.

Many moons ago, there was a question of whether we should rip out the
code for caching open file stream to make the source easier for
maintenance.

Of course, we need to check that the performance does not suffer from
doing this.

The suggestion from Jorg was that I check the operation of copying,
say, 100 messages from a folder to the other. (I think it was mentioned in
a mail post, or somewhere, but I am a little hazy and could not find
the original reference.)

Well, family matters prevented me from doing this quickly.

But now I have produced the performance data for copying 1001 messages
from a folder to the other for the version created from TB hg source
tree with the buffered output feature WITHOUT caching of open file
stream and for the version WITHOUT buffered output but WITH caching.
Since the only method available for me to measure the timing is the
wristwatch, I copied enough messages so that I can measure the time
meaningfully.

The following is the summary of what I found.

Summary: Based on the timing data of copying of 1001 messages which
takes about 3-6 seconds with my patched TB (no caching of open file
stream, but buffering output stream) and about 215 seconds (!)  with
the unpatched TB (caching of open file stream, but unbuffered output
stream) in one typical test case, the open/close of message folder per
message copy s not a performance bottleneck of TB at all.
Rather it was found that unbuffered output is a killer overhead.
The observation is backed up by the analysis of the number of system
calls involved.

It is hard to believe the timing difference of 3-6 seconds vs 215
seconds.
However, you might want to check how long it takes for you to copy
hundred messages from a folder to the other. I think it will be a
dozen seconds or so. (It was on my Windows10 released TB version, too.)
I didn't realize the speed was so slow since I don't do bulk copying
often in real world, and I have used the local buffered version for
local testing quite a while last year and the before.

Background:

Shall we rip out the code for caching open file stream from TB source
code to simplify the code for easy maintenance?

For example, using buffered output everywhere in TB requires the
caching of buffered open file stream. This caching turns out not quite
easy or straightforward with the changes of some buffered file
creation functions since the fall of 2017.

As a matter of fact, I could not cope with this caching when I tried to
adapt my patches for buffered output with the later hg source tree
after I could not program for a while in the fall of 2017 and early
2018 due to family matters.

So I decided to rip out the caching of open file stream in order to
enable buffered output in the Spring of 2018. The patches were
modified eventually to work with mid-April source tree.

Of course the big question pops up.
Does this doing away with caching of open file stream affect performance
significantly?

ANSWER:

Definitely NO is the answer based on performance data and
system call statistics.
(And actually, the buffered output turns out to be a huge win for
message copying case.)

Here is the report of detailed performance analysis.

Conclusion: we can safely rip out the code for caching of open file
stream to make the source tree easier to maintain.
We can introduce the buffered output for much better performance of
TB even after ripping the caching of open file streams.

My point here is that the maintaining caching of open file stream has
become rather difficult and made the code maintenance difficult if we
wanted to introduce new features such as using buffered stream as
often as possible. And as the performance data and system call
analysis shows, the caching actually didn't buy much in terms of
performance gain, and the gain obtained by using buffered output far
outweighs the merit of caching, if any.

Detailed statistics.

I obtained the elapsed time and system call statistics for the following
combinations of
source folder and test folder.

Note 215 seconds (marked with leading *) for Binary-2 in the case of
copy 1001 messages.
That clearly shows that the current TB is very slow, and buffered output
can bring about significant win.

Source folder                     Target Folder
--------------------------------+-----+------+-------+------------
                                 |  0  | gdb  |TeXLive| Libc-alpha
--------------------------------+-----+------+-------+------------
1|emacs-org-mode-DIR            | 1A  | 1B   | 1C    | 1D
  |copy 1001 msgs (sec)Binary-1  | 4   |  3   | 3.5-4 | 3.5-4
  |copy 2000 msgs (sec)Binary-1  |5-5.5|      |  6    |
1'                              |     |      |       |
  |copy 1001 msgs (sec)Binary-2  |*215 |      |
--------------------------------+-----+------+-------+------------
2|libc-alpha-dir      387MB     | 2A  | 2B   | 2C    | n/a
  |copy 1001 msgs (sec)Binary-1  | 2-3 | 2-3  |  5    |
--------------------------------+-----+------+-------+------------
4| Fix-compiler-warnings 8MB    | 4A  | 4B   | 4C    | 4D
  |copy 1001 msgs (sec)Binary-1  | 3.5 |3.5-4 | 3-4   | 3
  |copy 2000 msgs (sec)Binary-1  | 6   |      | 6-7   |
--------------------------------+-----+------+-------+------------

For example, the tag 1A in the table refers to the case of
source folder: emacs-org-mode-DIR
target folder: newly-created empty zero-folder

Regarding syscall statistics

(First) This is for the case of copying 1001 messages from
emacs-org-mode-DIR
to zero-folder (newly created empty folder) using the Binary-1
(buffered output but without caching open file stream).

The wall elapsed time was approximately 4 sec.

I notice that there is one fsync, but 24 fdatasync calls..

There are large number openat/close.

                 # of calls  time spent  avg. time spent (microseconds)

clone                1        198       198.000000
exit                 1          0         0.000000
fsync                1     118148    118148.000000 <- fsync once
getpriority          1         84        84.000000
mkdir                1         12        12.000000
prctl                1         87        87.000000
quotactl             1        101       101.000000
set_robust_list      1         50        50.000000
setpriority          1         85        85.000000
statfs               1        104       104.000000
rename               2        166        83.000000
uname                2        152        76.000000
epoll_wait           3          0         0.000000
ftruncate            4        292        73.000000
shmat                4        339        84.750000
shmget               4        548       137.000000
geteuid              6        241        40.166667
shmdt                6        543        90.500000
shmctl               8        593        74.125000
restart_syscall     10    4848284    484828.400000
unlink              11       2776       252.363636
fdatasync           24    2373405     98891.875000 <- 24 fdatasync calls?
brk                 60      11547       192.450000
getrusage           86       8377        97.406977
dup                123      11352        92.292683
munmap             149      12001        80.543624
sendmsg            180      15179        84.327778
getpid             286      20527        71.772727
mmap               489      81128       165.905930
getuid             711      47481        66.780591
fstat              928      68061        73.341595
stat              2039     171133        83.929868
lstat             2159     223204       103.383048
times             2772     291825       105.275974
fcntl             2781     221920        79.798634
close             3574     212760        59.529938 <- close
openat            4159     434382       104.443857 <- openat
writev            4774     517728       108.447424
access            5978     467610        78.221813
mprotect          6322     533449        84.379785
lseek             8883     602492        67.825284
read             10544     851908        80.795524
madvise          10636     454616        42.743137
gettid           11268     666666        59.164537
write            11945    1145337        95.884219 <- write
poll             23582   96770841      4103.589221
recvmsg          38291    3232226        84.412160
futex            43569  703213974     16140.236728

(Second)
This is for the case of copying 1001 messages from emacs-org-mode-DIR
to zero-folder (newly created empty folder) using the Binary-2
(unbuffered output but with caching open file stream).

The elapsed time was 3 minutes 35 seconds, i.e.. 215 sec.

Why is this so slow compared with the above?
Obviously the sheer number of context switches of write is the killing
factor.

 >write           168947   14689604        86.948001    <- write is the
killer
 >                                                      Too many context
switches.w

We notice 24 fdatasync calls.

We see that the # of openat and close are called less often due to
caching..
 >openat            2224     375128       168.672662      yes open and close
 >close             2233     107400        48.096731      is called less
often.

Without caching in the previous case, it was
 >close             3574     212760        59.529938 <- close
 >openat            4159     434382       104.443857 <- openat


                 # of calls  time spent  avg. time spent (microseconds)

clone                1        560       560.000000
getpriority          1         60        60.000000
prctl                1         98        98.000000
readlink             1         68        68.000000
set_robust_list      1        119       119.000000
setpriority          1         95        95.000000
exit                 2          0         0.000000
quotactl             2        142        71.000000
rt_sigreturn         2        131        65.500000
shmdt                2        229       114.500000
statfs               2        131        65.500000
tgkill               2        183        91.500000
epoll_wait           3          0         0.000000
mkdir                4        198        49.500000
rename               4        483       120.750000
uname                4        388        97.000000
shmat                6         80        13.333333
shmget               6         96        16.000000
ftruncate            7        535        76.428571
getdents             8        350        43.750000
restart_syscall     10    8756919    875691.900000
shmctl              12         64         5.333333
geteuid             20       1037        51.850000
unlink              32     186171      5817.843750
brk                 48      13646       284.291667
fdatasync           80   10857559    135719.487500      <--- Wow
sendmsg             84      10231       121.797619
getrusage           90       9931       110.344444
getuid             350      23665        67.614286
dup                401      54583       136.117207
munmap             413      34947        84.617433
mmap               738      66095        89.559621
getpid             860      81509        94.777907
fstat             1001      71634        71.562438
fsync             1001  146870325    146723.601399
lstat             1155     116723       101.058874
madvise           1335      34313        25.702622
fcntl             1638     187943       114.739316
openat            2224     375128       168.672662      yes open and close
close             2233     107400        48.096731      is called less
often.
stat              2591     280157       108.126978      Less by 1001.
writev            3348     373862       111.667264
times             3392     370686       109.282429
access            3521     347793        98.776768
lseek             4858     378373        77.886579
mprotect          5551     611719       110.199784
read              8276     668608        80.788787
poll             19455  478942793     24617.979594
recvmsg          28682    2631934        91.762569
gettid           54866    3151040        57.431561
futex           119854 2759027240     23019.901213
write           168947   14689604        86.948001    <- write is the killer
                                                       Too many context
switches.


The text file that contains detail is posted to buzilla 1242030.

( The post is https://bugzilla.mozilla.org/attachment.cgi?id=8993972 )

TIA

Chiaki

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

ISHIKAWA,chiaki
On 2018/07/22 8:07, ISHIKAWA,chiaki wrote:
> (This is a repost of
> https://bugzilla.mozilla.org/show_bug.cgi?id=1242030#c234
>
> I think it merits wider discussion, thus posting it here. I know that
> the mailer may mungle the text formatting. Please see the above URL in
> that case.)
>
> Performance analysis when we remove the caching of open stream.
>
 > ... [ omission ] ...

If it is not  clear, :-)
my intention is to propose, we rip out the code for caching open file
stream, so that the code base is simpler.

I can include the patch to do so in the patch set to use buffered output.

In an ideal world, someone could produce the patch to remove the caching
first independent of buffered output patch set.

Unfortunately, in my case, I only hit this problem and was forced to
remove the caching code after I tried to adapt my patches to use
buffered output for the current (mid-April) tree, and so the removable
is mixed in the patch set.

TIA

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Wayne Mery
In reply to this post by ISHIKAWA,chiaki
On 7/23/2018 10:39 PM, ISHIKAWA,chiaki wrote:

> On 2018/07/22 8:07, ISHIKAWA,chiaki wrote:
>> (This is a repost of
>> https://bugzilla.mozilla.org/show_bug.cgi?id=1242030#c234
>>
>> I think it merits wider discussion, thus posting it here. I know that
>> the mailer may mungle the text formatting. Please see the above URL in
>> that case.)
>>
>> Performance analysis when we remove the caching of open stream.
>>
>  > ... [ omission ] ...
>
> If it is not  clear, :-)
> my intention is to propose, we rip out the code for caching open file
> stream, so that the code base is simpler.
>
> I can include the patch to do so in the patch set to use buffered output.
>
> In an ideal world, someone could produce the patch to remove the caching
> first independent of buffered output patch set.
>
> Unfortunately, in my case, I only hit this problem and was forced to
> remove the caching code after I tried to adapt my patches to use
> buffered output for the current (mid-April) tree, and so the removable
> is mixed in the patch set.
>
> TIA

Looks like solid analysis to me, and that the transformation should proceed!

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Disaster Master
On 7/24/2018, 8:05:42 AM, Wayne <[hidden email]> wrote:

> On 7/23/2018 10:39 PM, ISHIKAWA,chiaki wrote:
>> On 2018/07/22 8:07, ISHIKAWA,chiaki wrote:
>>> (This is a repost of
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=1242030#c234
>>>
>>> I think it merits wider discussion, thus posting it here. I know that
>>> the mailer may mungle the text formatting. Please see the above URL in
>>> that case.)
>>>
>>> Performance analysis when we remove the caching of open stream.
>>>
>>  > ... [ omission ] ...
>>
>> If it is not  clear, :-)
>> my intention is to propose, we rip out the code for caching open file
>> stream, so that the code base is simpler.
>>
>> I can include the patch to do so in the patch set to use buffered output.
>>
>> In an ideal world, someone could produce the patch to remove the caching
>> first independent of buffered output patch set.
>>
>> Unfortunately, in my case, I only hit this problem and was forced to
>> remove the caching code after I tried to adapt my patches to use
>> buffered output for the current (mid-April) tree, and so the removable
>> is mixed in the patch set.

> Looks like solid analysis to me, and that the transformation should proceed!

Agreed - and from the sounds of the performance improvements, I can only
hope this makes it into the next RELEASE version, even if it is a major
change. I'd hate to be forced to choose between running a nightly and
missing out on the performance gains for a year or more.
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Disaster Master
In reply to this post by ISHIKAWA,chiaki
On 7/21/2018, 7:07:37 PM, ISHIKAWA,chiaki <[hidden email]> wrote:
> Summary: Based on the timing data of copying of 1001 messages which
> takes about 3-6 seconds with my patched TB (no caching of open file
> stream, but buffering output stream) and about 215 seconds (!)  with
> the unpatched TB (caching of open file stream, but unbuffered output
> stream) in one typical test case, the open/close of message folder per
> message copy s not a performance bottleneck of TB at all.
> Rather it was found that unbuffered output is a killer overhead.
> The observation is backed up by the analysis of the number of system
> calls involved.

Question: is this issue more or less dependent on POP vs IMAP?

Meaning, since I only use IMAP, should I also expect to see similar
(huge) performance gains?
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Jörg Knobloch
In reply to this post by Disaster Master
On 24/07/2018 14:42, Disaster Master wrote:
> Agreed - and from the sounds of the performance improvements, I can only
> hope this makes it into the next RELEASE version, even if it is a major
> change. I'd hate to be forced to choose between running a nightly and
> missing out on the performance gains for a year or more.

No hope. Will most certainly not backport drastic changes like this.

The improvements have been planned for years, but no one ever got around
to providing the patches.

Jörg.

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Disaster Master
On 7/25/2018, 4:48:04 AM, Jörg Knobloch <[hidden email]> wrote:

> On 24/07/2018 14:42, Disaster Master wrote:
>> Agreed - and from the sounds of the performance improvements, I can only
>> hope this makes it into the next RELEASE version, even if it is a major
>> change. I'd hate to be forced to choose between running a nightly and
>> missing out on the performance gains for a year or more.
>
> No hope. Will most certainly not backport drastic changes like this.
>
> The improvements have been planned for years, but no one ever got around
> to providing the patches.

Bummer, but I understand...

Can you address my other question bout whether this will benefit both
POP and IMAP, or is this just for one or the other?

Thanks - guess I'll be using a non release build once these patches land
and bake for a while...
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Wayne Mery
In reply to this post by Jörg Knobloch
On 7/25/2018 10:31 AM, Disaster Master wrote:

> On 7/25/2018, 4:48:04 AM, Jörg Knobloch <[hidden email]> wrote:
>> On 24/07/2018 14:42, Disaster Master wrote:
>>> Agreed - and from the sounds of the performance improvements, I can only
>>> hope this makes it into the next RELEASE version, even if it is a major
>>> change. I'd hate to be forced to choose between running a nightly and
>>> missing out on the performance gains for a year or more.
>>
>> No hope. Will most certainly not backport drastic changes like this.
>>
>> The improvements have been planned for years, but no one ever got around
>> to providing the patches.
>
> Bummer, but I understand...
>
> Can you address my other question bout whether this will benefit both
> POP and IMAP, or is this just for one or the other?

If you look at the bug cited by OP, it addresses multiple issues.
https://bugzilla.mozilla.org/show_bug.cgi?id=1242030#attachments

> Thanks - guess I'll be using a non release build once these patches land
> and bake for a while...

Beta is where you will want to be.

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Disaster Master
On 7/25/2018, 10:54:31 AM, Wayne <[hidden email]> wrote:
> On 7/25/2018 10:31 AM, Disaster Master wrote:
>> Bummer, but I understand...
>>
>> Can you address my other question bout whether this will benefit both
>> POP and IMAP, or is this just for one or the other?

> If you look at the bug cited by OP, it addresses multiple issues.
> https://bugzilla.mozilla.org/show_bug.cgi?id=1242030#attachments

Thanks, I know this, I was referring specifically to ISHIKAWAs comment
about "the patch regarding ripping out the code for caching open file
stream, and including the patch to do so in the patch set to use
buffered output".

>> Thanks - guess I'll be using a non release build once these patches land
>> and bake for a while...

> Beta is where you will want to be.
Gotcha, thanks...
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

Jörg Knobloch
In reply to this post by Disaster Master
On 25/07/2018 16:31, Disaster Master wrote:
> Can you address my other question bout whether this will benefit both
> POP and IMAP, or is this just for one or the other?

It improves access to all mbox files, not maildir. So if you use POP
with mbox or IMAP with local mbox storage, then you'll get the benefit
... one day.

The truth is, if you're downloading or moving two messages, you won't
notice the difference, and typically you don't move message in bulk,
well I don't. So while getting the house clean is a good thing, it's
nothing to get too excited about. Just my humble opinion.

Jörg.

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

EricNo.SpamValette
In reply to this post by Disaster Master
On 7/25/18 8:34 PM, Jörg Knobloch wrote:

> On 25/07/2018 16:31, Disaster Master wrote:
>> Can you address my other question bout whether this will benefit both
>> POP and IMAP, or is this just for one or the other?
>
> It improves access to all mbox files, not maildir. So if you use POP
> with mbox or IMAP with local mbox storage, then you'll get the benefit
> ... one day.
>
> The truth is, if you're downloading or moving two messages, you won't
> notice the difference, and typically you don't move message in bulk,
> well I don't. So while getting the house clean is a good thing, it's
> nothing to get too excited about. Just my humble opinion.


Well actually, on Linux, using a filer (read remote fs acessed using SMB
protocol) as home directory (for backup reasons) is unusable because of
both error handling and 79 char lenght write...

-- eric




_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

ISHIKAWA,chiaki
On 2018/07/26 5:02, EricNo.SpamValette wrote:

> On 7/25/18 8:34 PM, Jörg Knobloch wrote:
>> On 25/07/2018 16:31, Disaster Master wrote:
>>> Can you address my other question bout whether this will benefit both
>>> POP and IMAP, or is this just for one or the other?
>>
>> It improves access to all mbox files, not maildir. So if you use POP
>> with mbox or IMAP with local mbox storage, then you'll get the benefit
>> ... one day.
>>
>> The truth is, if you're downloading or moving two messages, you won't
>> notice the difference, and typically you don't move message in bulk,
>> well I don't. So while getting the house clean is a good thing, it's
>> nothing to get too excited about. Just my humble opinion.
>
>
> Well actually, on Linux, using a filer (read remote fs acessed using SMB
> protocol) as home directory (for backup reasons) is unusable because of
> both error handling and 79 char lenght write...
>
> -- eric
>
>

I believe Jorg has answered the question regarding the
local copying improvment.

I know the issue for remote write. The buffered output patch improves
the write to remote file system .

Due to family matters, I could not do much programming in my spare time,
for half a year or so.
But now that I am back, I am hoping to get the patch into the tree in
the NEXT major release cycle.
(Famous last words.)

Or maybe TB's source base gets rewritten completely (???), whichever
comes first.

TIA



>
>
> _______________________________________________
> dev-apps-thunderbird mailing list
> [hidden email]
> https://lists.mozilla.org/listinfo/dev-apps-thunderbird

_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird
Reply | Threaded
Open this post in threaded view
|

Re: TB performance and maintenance issue: Can we remove the caching of open file stream without impacting the performance much?

ISHIKAWA,chiaki
In reply to this post by Jörg Knobloch
On 2018/07/26 3:34, Jörg Knobloch wrote:

> On 25/07/2018 16:31, Disaster Master wrote:
>> Can you address my other question bout whether this will benefit both
>> POP and IMAP, or is this just for one or the other?
>
> It improves access to all mbox files, not maildir. So if you use POP
> with mbox or IMAP with local mbox storage, then you'll get the benefit
> ... one day.
>
> The truth is, if you're downloading or moving two messages, you won't
> notice the difference, and typically you don't move message in bulk,
> well I don't. So while getting the house clean is a good thing, it's
> nothing to get too excited about. Just my humble opinion.
>
> Jörg.
>

Dear  Jörg.

You are not opposed to ripping out the code for caching, are you?

I am going to tidy up the patch set in that direction if I understand
your position correctly.

TIA

Chiaki
_______________________________________________
dev-apps-thunderbird mailing list
[hidden email]
https://lists.mozilla.org/listinfo/dev-apps-thunderbird