Apache2::SizeLimit httpd process too big, exiting

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Apache2::SizeLimit httpd process too big, exiting

Ruslan Dautkhanov
Hello,

After several unsuccessful attempts to run bugzilla 4.0 in fastcgi
environment, I decided to run under apache+mod_perl.

It seems that for every request httpd is dying:
Apache2::SizeLimit httpd process too big, exiting at SIZE=*296592 *KB
SHARE=3808 KB UNSHARED=292784 REQUESTS=2 LIFETIME=0 seconds
Apache2::SizeLimit httpd process too big, exiting at SIZE=291428 KB
SHARE=3764 KB UNSHARED=287664 REQUESTS=2 LIFETIME=0 seconds
Apache2::SizeLimit httpd process too big, exiting at SIZE=321772 KB
SHARE=4220 KB UNSHARED=317552 REQUESTS=2 LIFETIME=0 seconds
Apache2::SizeLimit httpd process too big, exiting at SIZE=291428 KB
SHARE=3764 KB UNSHARED=287664 REQUESTS=2 LIFETIME=0 seconds
...

Meanwhile bugzilla works *normal*, as needed on the user side -- all actions
are completed.

top snapshot:
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
30601 apache    15   0  *258m  *63m 2256 S  0.0  9.0   0:00.00 httpd
30605 apache    15   0  258m  63m 2256 S  0.0  9.0   0:00.00 httpd
30614 apache    18   0  258m  62m 1812 S  0.0  9.0   0:00.00 httpd
30621 apache    20   0  258m  62m 1812 S  0.0  9.0   0:00.00 httpd
30625 apache    17   0  258m  62m 1812 S  0.0  9.0   0:00.00 httpd

Woof, isn't 258 Mb a lot for just one httpd? How to stop bugzilla's httpd
grow so much so it is killed after every request.

Bugzilla 4.0
Apache/2.2.3 (CentOS) DAV/2 mod_perl/2.0.4 Perl/v5.8.8 configured
perl, v5.8.8 built for x86_64-linux-thread-multi
CentOS 5.5 with latest updates
kernel 2.6.18-238.5.1.el5.028stab085.2

Any help will be greatly appreciated!

PS. I am starting disliking Perl... The setup of numerous Perl modules for
Bugzilla wasn't always easy, e.g. some external library
 dependencies require software that were conflicting with my "standard"
packages I setup through yum. I was okay with downloading
 and compiling fresher modules since I have dedicated virtual box just for
Bugzilla. Though it was still time-consuming.

Thank you,
Ruslan Dautkhanov
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Peter Vereshagin
We can live forever looking to my eyes, support-bugzilla!
2011/03/20 00:27:02 -0600 Ruslan Dautkhanov <[hidden email]> => To [hidden email] :

RD> Apache2::SizeLimit httpd process too big, exiting at SIZE=*296592 *KB
RD> SHARE=3808 KB UNSHARED=292784 REQUESTS=2 LIFETIME=0 seconds

Did yuo try MaxRedquests limit?

73! Peter pgp: A0E26627 (4A42 6841 2871 5EA7 52AB  12F8 0CE1 4AAC A0E2 6627)
--
http://vereshagin.org
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Max Kanat-Alexander
In reply to this post by Ruslan Dautkhanov
On 03/19/2011 11:27 PM, Ruslan Dautkhanov wrote:
> Apache2::SizeLimit httpd process too big, exiting at SIZE=*296592 *KB
> SHARE=3808 KB UNSHARED=292784 REQUESTS=2 LIFETIME=0 seconds

        This is not a huge problem unless your load is very high. It is a bug
and we will be fixing it soon.

> Woof, isn't 258 Mb a lot for just one httpd? How to stop bugzilla's httpd
> grow so much so it is killed after every request.

        Ignore the "VIRT" number. Also ignore the "RSS" number, actually. What
you really want to know is how much private (non-shared) RAM each
process is using, which unfortunately top cannot tell you. I'm not sure
of any good tool that will tell you, in fact.

> PS. I am starting disliking Perl... The setup of numerous Perl modules for
> Bugzilla wasn't always easy, e.g. some external library
>  dependencies require software that were conflicting with my "standard"
> packages I setup through yum. I was okay with downloading
>  and compiling fresher modules since I have dedicated virtual box just for
> Bugzilla. Though it was still time-consuming.

        Well, this page may have helped:

        https://wiki.mozilla.org/Bugzilla:Prerequisites

        If you only use that information and then only use install-module.pl to
install the modules, it should actually be very easy to get all the
prerequisites on RHEL5 or newer.

        -Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Ruslan Dautkhanov
In reply to this post by Ruslan Dautkhanov
>
> > Apache2::SizeLimit httpd process too big, exiting at SIZE=*296592 *KB
> > SHARE=3808 KB UNSHARED=292784 REQUESTS=2 LIFETIME=0 seconds
>
>        This is not a huge problem unless your load is very high. It is a
> bug
> and we will be fixing it soon.
>

No load at all. I just installed it. httpd/bugzilla dies after *every*
invocation.

I changed that hard coded limit in mod_perl.pl from 70Mb to 320 Mb and it's
better
now, but still bailing out every 7-9 executions.
It doesn't affect bugzilla from user's perspective, since Apache2::SizeLimit
is killing
itself after request was processed.


>
> > Woof, isn't 258 Mb a lot for just one httpd? How to stop bugzilla's httpd
> > grow so much so it is killed after every request.
>
>        Ignore the "VIRT" number. Also ignore the "RSS" number, actually.
> What
> you really want to know is how much private (non-shared) RAM each
> process is using, which unfortunately top cannot tell you. I'm not sure
> of any good tool that will tell you, in fact.
>
/proc/PID/smaps ?


Overall, the memory consumption looks very bad. I don't know why bugzilla
cosumes 300Mb memory? It's a lot.

I tried to comment loop where it was loading all *.cgi files in mod_perl.pl.
It's turned to be a little bit better, without affecting performance, but it
is still a lot.

Best regards,
Ruslan Dautkhanov
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Max Kanat-Alexander
On 03/21/2011 11:55 AM, Ruslan Dautkhanov wrote:
> No load at all. I just installed it. httpd/bugzilla dies after *every*
> invocation.

        Yes, that is the bug that is known:

        https://bugzilla.mozilla.org/show_bug.cgi?id=633061

        With no load at all, your users will not notice a problem.

> I changed that hard coded limit in mod_perl.pl from 70Mb to 320 Mb and it's
> better
> now, but still bailing out every 7-9 executions.

        That's fine, that prevents Bugzilla from using up too much RAM.

> /proc/PID/smaps ?

        If you want to process it and add up all of the Private_Clean and
Private_Dirty numbers together yourself, sure. Seems like a lot of work,
though.

> Overall, the memory consumption looks very bad. I don't know why bugzilla
> cosumes 300Mb memory? It's a lot.

        It doesn't. For a very detailed description of how memory consumption
and reporting works on Linux, see this email:

        http://mail-archives.apache.org/mod_mbox/perl-modperl/201102.mbox/browser

> I tried to comment loop where it was loading all *.cgi files in mod_perl.pl.
> It's turned to be a little bit better, without affecting performance, but it
> is still a lot.

        That would actually make performance much worse and actually
significantly increase memory consumption in a live installation,
because you are eliminating the copy-on-write behavior of Apache.

        -Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Ruslan Dautkhanov
In reply to this post by Ruslan Dautkhanov
---------- Forwarded message ----------
From: Ruslan Dautkhanov <[hidden email]>
Date: Mon, Mar 21, 2011 at 11:19 PM
Subject: Re: Apache2::SizeLimit httpd process too big, exiting
To: Max Kanat-Alexander <[hidden email]>


Hello,

> I tried to comment loop where it was loading all *.cgi files in
mod_perl.pl.
> It's turned to be a little bit better, without affecting performance, but
it
> is still a lot.

       That would actually make performance much worse and actually
significantly increase memory consumption in a live installation,
because you are eliminating the copy-on-write behavior of Apache.


Thank you for the link. It explained a lot.
I tested with and without loop to preload all *.cgi files.
Before (no preloading):

> Private_Dirty: 29.73 Mb
> Private_Clean: 0.00 Mb
> Size: 290.98 Mb
> Shared_Clean: 3.98 Mb
> Swap: 0.00 Mb
> Shared_Dirty: 30.14 Mb
> Rss: 63.86 Mb
> Pss: 35.41 Mb
>

After (with preloading, as it comes in original mod_perl.pl):

> Private_Dirty: 3.61 Mb
> Private_Clean: 0.00 Mb
> Size: 268.24 Mb
> Shared_Clean: 1.76 Mb
> Swap: 0.00 Mb
> Shared_Dirty: 57.34 Mb
> Rss: 62.71 Mb
> Pss: 11.91 Mb
>

Perl script I used:

> my %mem;
> open(FILE, "< /proc/$pid/smaps") or die "process pid=$pid not found: $!";
> while (<FILE>)
> {       /^(\w+):\s+(\d+) kB/ || next;
>         $mem{$1}+=$2;
> }
> close FILE;
> foreach my $type (sort {$a<=>$b} keys %mem)
> {       printf("$type: %.02f Mb\n", $mem{$type}/1024);
> }
>

Private dirty is much less in the second case. So, it makes sense to
preload.
I still don't understand why Size ~ 268..290Mb?
What does it mean? Does it count the same shared memory of all httpd
mod_perl processes
multiplied by number of processes?

PS. Thanks, your one-line patch to Apache::SizeLimit solved the problem...

Best regards,
Ruslan Dautkhanov
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Max Kanat-Alexander
On 03/22/2011 07:07 AM, Ruslan Dautkhanov wrote:
> I still don't understand why Size ~ 268..290Mb?

        Size is the entire "virtual memory space" of the process, it has
nothing to do with actual RAM being used. Basically you should ignore
it, it is a useless number.

> PS. Thanks, your one-line patch to Apache::SizeLimit solved the problem...

        Ah great! :-)

        -Max
--
http://www.bugzillasource.com/
Competent, Friendly Bugzilla, Perl, and IT Services
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

RE: Apache2::SizeLimit httpd process too big, exiting

Tosh, Michael J
Max Kanat-Alexander wrote:
>> PS. Thanks, your one-line patch to Apache::SizeLimit solved the problem...
>
> Ah great! :-)

Which one liner for SizeLimit?  I'd like to apply that here too. Would I find that in the Apache2::SizeLimit version 0.96?  
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.
Reply | Threaded
Open this post in threaded view
|

Re: Apache2::SizeLimit httpd process too big, exiting

Peter Vereshagin
I've seen the future support-bugzilla <= it is motto.
2011/03/23 09:04:59 -0400 "Tosh, Michael J" <[hidden email]> => To Max Kanat-Alexander :

TMJ> >> PS. Thanks, your one-line patch to Apache::SizeLimit solved the problem...
TMJ> Which one liner for SizeLimit?  I'd like to apply that here too. Would I find that in the Apache2::SizeLimit version 0.96?  

https://bug633061.bugzilla.mozilla.org/attachment.cgi?id=511268

73! Peter pgp: A0E26627 (4A42 6841 2871 5EA7 52AB  12F8 0CE1 4AAC A0E2 6627)
--
http://vereshagin.org
_______________________________________________
support-bugzilla mailing list
[hidden email]
https://lists.mozilla.org/listinfo/support-bugzilla
PLEASE put [hidden email] in the To: field when you reply.