OT apache troubleshoot

[Barry R Cisna]

Hello All,

# off topic

FYI. I did search the Apache forums and couldn't really find a definite
solution.

On my home CentOS 6.x / K12Linux server I do run Apache server with
about 6 sites running for various people I know that they maintain
simply as a hobby type thing for themselves. I run Joomla as the CMS for
convenience which has worked very well since the get-go.

Problem:
Without having done any updates to the server for sometime and no
monkeying around with configuration files on the Apache part anyway.
Some times now when running top I seen many instances of http in the mix
and each process about 25% memory,,,which of course does make the server
itself get eventually slow.

This is about an 4 month old brand new medium grade hardware,but nothing
'enterprise grade" FWIW.

In the past I always did see about 10-12 instances of httpd running when
I run ' ps aux|grep httpd' but never have seen 6-8 instances running
when I run top.
I guess due to the fact I have never seen this in top before,(and
consequently the server getting somewhat slow),I have never really
looked into this or have read up any on the http 'forking process'.

If I simply shut off the httpd service for 10 mins then restart
later,,these of course disappear. I am not sure what to think in regards
to troubleshooting and tracing what causes this behaviour. All 6
websites have run (on my previous server hardware) and I never did have
this happen.
Sorry for the long post

Thank You,
Barry

[Todd Hackett]

Hi Barry

PID USER PR NI VIRT RES SHR S %cpu %Mem Time+ COMMAND
2789 daemon 30 10 261m 12m 3428 S 0.0 0.3 0:01.77httpd
2923 daemon 30 10 258m 9120 3440 S 0.0 0.2 0:00.97 httpd
3106 daemon 30 10 258m 9048 3396 S 0.0 0.2 0:01.15 httpd
3501 daemon 30 10 258m 8992 3328 S 0.0 0.2 0:00.57 httpd
3891 daemon 30 10 258m 9036 3388 S 0.0 0.2 0:00.79 httpd
3930 daemon 30 10 258m 8764 3132 S 0.0 0.2 0:00.05 httpd
4021 daemon 30 10 257m 8764 3328 S 0.0 0.2 0:00.28 httpd
4717 daemon 30 10 257m 8704 3312 S 0.0 0.2 0:01.01 httpd
4719 daemon 30 10 258m 9012 3364 S 0.0 0.2 0:00.89 httpd
4726 daemon 30 10 258m 8372 3028 S 0.0 0.2 0:00.01 httpd

uptime is 965 days

The above is a production server with many websites ... Are you sure it
is 25% and not 0.2 or 0.3%? You can limit the number of processes
to spawn; on my test server at home .. I typically use 2 to 4
depending. If it is a BIG memory concern, drop that to 1 or 2 and see
how it goes. I have seen up to 25 children running at busy times and it
doesn't seem to slow things down. ( dual 64Bit 3.6Gig Xeon's
hyper-threading; with 8Gig of memory, scsi raid )

You didn't mention your server: Memory, Disk system; and the apache
version.

todh

Post by Barry R Cisna
Hello All,
# off topic
FYI. I did search the Apache forums and couldn't really find a definite
solution.
On my home CentOS 6.x / K12Linux server I do run Apache server with
about 6 sites running for various people I know that they maintain
simply as a hobby type thing for themselves. I run Joomla as the CMS for
convenience which has worked very well since the get-go.
Without having done any updates to the server for sometime and no
monkeying around with configuration files on the Apache part anyway.
Some times now when running top I seen many instances of http in the mix
and each process about 25% memory,,,which of course does make the server
itself get eventually slow.
This is about an 4 month old brand new medium grade hardware,but nothing
'enterprise grade" FWIW.
In the past I always did see about 10-12 instances of httpd running when
I run ' ps aux|grep httpd' but never have seen 6-8 instances running
when I run top.
I guess due to the fact I have never seen this in top before,(and
consequently the server getting somewhat slow),I have never really
looked into this or have read up any on the http 'forking process'.
If I simply shut off the httpd service for 10 mins then restart
later,,these of course disappear. I am not sure what to think in regards
to troubleshooting and tracing what causes this behaviour. All 6
websites have run (on my previous server hardware) and I never did have
this happen.
Sorry for the long post
Thank You,
Barry
_______________________________________________
K12OSN mailing list
https://www.redhat.com/mailman/listinfo/k12osn
For more info see<http://www.k12os.org>

--
Todd Hackett System Administrator
POBox 1168
Libby, Montana 59923 406.293.3843

http://www.todhackett.com

Network and Server Administration.
Secure Firewall management.
Antispam/antivirus eMail.
Managed Web Hosting.

Like trains? http://www.soundrail.com
Got Stuff? http://www.libbystuff.com

[Barry R Cisna]

Tod,

Apache 2.2.15
32-bit CentOS 6.3
AMD E-350 Zacate APU
8 GB ram
Sata-3

Here is a paste from 'top -u apache'

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2013 apache 20 0 56296 25m 6040 S 0.0 0.3 0:02.90
httpd
2014 apache 20 0 57320 26m 6096 S 0.0 0.3 0:03.95
httpd
2015 apache 20 0 56260 25m 5484 S 0.0 0.3 0:02.83
httpd
28504 apache 20 0 45600 11m 872 S 0.0 0.2 0:00.01
httpd
28505 apache 20 0 56268 25m 5504 S 0.0 0.3 0:08.05
httpd
28506 apache 20 0 56284 26m 6116 S 0.0 0.3 0:07.55
httpd
28507 apache 20 0 56268 25m 5540 S 0.0 0.3 0:08.47
httpd
28508 apache 20 0 56416 25m 5536 S 0.0 0.3 0:07.06
httpd
28509 apache 20 0 56296 25m 5532 S 0.0 0.3 0:04.99
httpd
28510 apache 20 0 57292 25m 5540 S 0.0 0.3 0:08.66
httpd
28511 apache 20 0 56268 25m 5536 S 0.0 0.3 0:07.11
httpd
28512 apache 20 0 57320 26m 5536 S 0.0 0.3 0:05.82 httpd

The server is running fine like this.
,,,

When I noticed the server starting to get sluggish and the load average
was around 8.00- 9.00,,when I run 'top' with no parameters I would see
httpd entries about 6-8 times and yes it showed around 25% mem usage
each. I have never even seen httpd listed when running top before(with
no extra parameters). This has only happened about 3 times,and it seems
like it happened after a power outage at my house.Probably just a
coincedence?

Since I posted this and shut off httpd for 10 mins then restarted I have
never again seen httpd even listed in top ( this seems to be 'normal")
with no parameters.
Just wonder what makes httpd load the server in this fashion and what
spawns it?

Thanks,
Barry

[Jim Kinney]

You need to run other performance tools to get a better picture than just
top. Iostat will show drive access. Check apache logs to look for web
crawlers. A bad link that causes a process loop can eat your system. A
power blink that causes a small file corruption can generate similar
symptoms.

Post by Barry R Cisna
Tod,
Apache 2.2.15
32-bit CentOS 6.3
AMD E-350 Zacate APU
8 GB ram
Sata-3
Here is a paste from 'top -u apache'
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
2013 apache 20 0 56296 25m 6040 S 0.0 0.3 0:02.90
httpd
2014 apache 20 0 57320 26m 6096 S 0.0 0.3 0:03.95
httpd
2015 apache 20 0 56260 25m 5484 S 0.0 0.3 0:02.83
httpd
28504 apache 20 0 45600 11m 872 S 0.0 0.2 0:00.01
httpd
28505 apache 20 0 56268 25m 5504 S 0.0 0.3 0:08.05
httpd
28506 apache 20 0 56284 26m 6116 S 0.0 0.3 0:07.55
httpd
28507 apache 20 0 56268 25m 5540 S 0.0 0.3 0:08.47
httpd
28508 apache 20 0 56416 25m 5536 S 0.0 0.3 0:07.06
httpd
28509 apache 20 0 56296 25m 5532 S 0.0 0.3 0:04.99
httpd
28510 apache 20 0 57292 25m 5540 S 0.0 0.3 0:08.66
httpd
28511 apache 20 0 56268 25m 5536 S 0.0 0.3 0:07.11
httpd
28512 apache 20 0 57320 26m 5536 S 0.0 0.3 0:05.82 httpd
The server is running fine like this.
,,,
When I noticed the server starting to get sluggish and the load average
was around 8.00- 9.00,,when I run 'top' with no parameters I would see
httpd entries about 6-8 times and yes it showed around 25% mem usage
each. I have never even seen httpd listed when running top before(with
no extra parameters). This has only happened about 3 times,and it seems
like it happened after a power outage at my house.Probably just a
coincedence?
Since I posted this and shut off httpd for 10 mins then restarted I have
never again seen httpd even listed in top ( this seems to be 'normal")
with no parameters.
Just wonder what makes httpd load the server in this fashion and what
spawns it?
Thanks,
Barry
_______________________________________________
K12OSN mailing list
https://www.redhat.com/mailman/listinfo/k12osn
For more info see <http://www.k12os.org>

[Les Mikesell]

Post by Jim Kinney
You need to run other performance tools to get a better picture than just
top. Iostat will show drive access. Check apache logs to look for web
crawlers. A bad link that causes a process loop can eat your system. A power
blink that causes a small file corruption can generate similar symptoms.

php and other embedded program code will run as apache even though the
web server framework itself rarely uses much CPU. You might be able
to tell from the httpd logs what kind application requests you are
getting, and sometimes you can use 'strace -p process_id' to get some
idea of what the running code is doing - at least the system call
portion. If it is just in a compute loop you wouldn't see that.
Also note that security vulnerabilities are pretty common in php web
apps. Someone may have injected code you didn't expect.