fuser forking uncontrollably in cron job

[Will Shackleton]

Hi

On my Ubuntu Server 11.10 box, several thousand fuser processes are being created every half an hour. (This didn't happen in 11.04.) I traced the problem back to the cron job '/etc/cron.d/php5', which contains the entry:

[ -x /usr/lib/php5/maxlifetime ] && [ -d /var/lib/php5 ] && find /var/lib/php5/ -depth -mindepth 1 -maxdepth 1 -type f -cmin +$(/usr/lib/php5/maxlifetime) ! -execdir fuser -s {} 2>/dev/null \; -delete

I think this removes old PHP sessions.

This is an extract from 'ps aux':

root      1636  0.0  0.0      0     0 ?        Z    23:51   0:00 [fuser] <defunct>
root      1637  0.0  0.0      0     0 ?        Z    23:51   0:00 [fuser] <defunct>
root      1638  0.0  0.0      0     0 ?        Z    23:51   0:00 [fuser] <defunct>
root      1639  0.0  0.0      0     0 ?        Z    23:51   0:00 [fuser] <defunct>
(many, many more...)
...

real    0m10.832s
user    0m2.796s
sys     0m4.952s

There are thousands of these, but they do disappear eventually. Once the cron job exits, all the defunct processes are cleaned up, presumably by init.
This also puts my system load up to about 2 every half an hour.

I could remove the cron job, but I'd imagine PHP sessions wouldn't be cleared up then. I couldn't find any bugs to do with fuser, but it could be perhaps to do with 'find' calling fuser.

Attached is a graph showing load average of this server.

Any ideas on what to do?

Thanks,
Will

[Hizeh]

I have the exact same problem. Upgraded Ubuntu 11.04 -> 11.10 then I started getting the error "Failed to fork: Cannot allocate memory" at random times. When I looked into it I found the exact same issue Will describes.

I disabled that cron job to fix it for now..

[RastaFeri]

i have exactly the same problem too (11.04 -> 11.10)

[devilmastah]

Registerd to confirm this bug
Updated two servers both have the issue.
Also: testing with disabled cron on one server as i speak will report back!


EDIT:
So far: disabling the cron did the trick.
Now trying new version of php

[Hizeh]

Did updating php5 fix it?

[worley]

I've just upgraded 35 servers from 11.04 to 11.10 and we only have one server that has this issue.
Running PHP-fpm as with all the others, and it's doing exactly as described above.
I've seen 18,000 or so defunct fuser processes within a few hours of upgrading the server.
This is with the new PHP security release that came out today.

[worley]

# ps -ef | grep fuser|wc -l
29751

It's getting pretty high now, that is with an uptime of 1hr 19 min
I've disabled the script in /etc/cron.d/php5 temporarily.

[devilmastah]

Did updating php5 fix it?

Nope it didnt
Waiting for a patch

[matt_symes]

Hi

I couldn't find any bugs to do with fuser, but it could be perhaps to do with 'find' calling fuser.

Total shot in the dark but do you still have problems if you edit the cron job to use

lsof -t {}

instead of

fuser -s {}

They both return pids so i assume they would perform the same function and an error code of 1 on failure; 0 on sucesss.

From man lsof

-t This option specifies that lsof should produce terse output with process identifiers only and no header - e.g., so that the output may be piped to kill(1).
This option selects the -w option.

From man fuser

fuser displays the PIDs of processes using the specified files or file systems.

Might be worth a try ?

EDIT: You might have to pipe stdout and stderr to /dev/null.

Kind regards

[Hizeh]

I tried matt_symes fix and the "lsof -t" does not fork uncontrollably like fuser was doing.

But, I'm not sure how to check if it is actually clearing old PHP sessions.