Log file problem

[techsharp]

All,

Not sure if anyone has run into this or if this is a Scalix problem, however if anyone has run into this please let me know some helpful ways to solve..

On the Scalix server we have a separate /var/log partition and it is filling up:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb2 1008M 946M 12M 99% /var/log

The problem is - there is almost nothing taking up that much space! I see about 100MB of log files and nothing more. So we have around 900MB of space that is not in use, but the partition is seeing it that way.

Before It was a 0% available. I then cleaned up some log files and did free up some space, but it will probably go back to 0% available again.

Anyone else have this problem or any ideas of what could be causing this?

Thanks

[Valerion]

Are there any hidden files (starting with a .)?

Do a ls -lah to see the sizes.

A du -sh * and du -sh .* may also shed some light.

[techsharp]

Valerion -

Thanks -

I was doing a -lah before hand and still found nothing that would be taking up any space. I searched all the directories as well - and still nothing.

I also ran the other commands as well and still could not find anything big taking up space.

[btisdall]

IIRC if you delete any logfiles with the process that owned them still running they will still occupy space. Try an:

Code: Select all

lsof /var/log

(note no trailing slash).

You can then kill or restart the processes in question.

[techsharp]

Interesting - here is the output:

[root@mars log]# lsof /var/log
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
python 2306 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2306 mailman 4u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2309 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2309 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2309 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2310 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2310 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2310 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2311 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2311 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2311 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2312 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2312 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2312 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2313 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2313 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2313 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2314 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2314 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2314 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2314 mailman 6u REG 8,18 600 65576 /var/log/mailman/smtp
python 2315 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2315 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2315 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
python 2316 mailman 3u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2316 mailman 4u REG 8,18 0 65584 /var/log/mailman/error .1 (deleted)
python 2316 mailman 5u REG 8,18 729 65579 /var/log/mailman/qrunn er
stunnel 2331 nobody 3w REG 8,18 859115906 56 /var/log/stunnel.log ( deleted)
su 9821 root cwd DIR 8,18 4096 2 /var/log
bash 9863 root cwd DIR 8,18 4096 2 /var/log
lsof 9908 root cwd DIR 8,18 4096 2 /var/log
lsof 9909 root cwd DIR 8,18 4096 2 /var/log
httpd 13280 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13280 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13280 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 13280 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 13280 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 13280 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 13599 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13599 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13599 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 13599 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 13599 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 13599 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 13604 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13604 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13604 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 13604 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 13604 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 13604 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 13605 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13605 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13605 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 13605 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 13605 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 13605 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 13628 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13628 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 13628 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 13628 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 13628 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 13628 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 14967 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 14967 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 14967 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 14967 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 14967 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 14967 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 15088 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 15088 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 15088 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 15088 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 15088 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 15088 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 27205 root 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 27205 root 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 27205 root 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 27205 root 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 27205 root 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 27205 root 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
clamd 27406 clamav 3wW REG 8,18 4235 16 /var/log/clamd.log
bash 28431 tformoso cwd DIR 8,18 4096 2 /var/log
httpd 28993 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 28993 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 28993 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 28993 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 28993 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 28993 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 28994 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 28994 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 28994 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 28994 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 28994 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 28994 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
httpd 28995 apache 2w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 28995 apache 9w REG 8,18 4197 81929 /var/log/httpd/error_l og (deleted)
httpd 28995 apache 10w REG 8,18 8779 81935 /var/log/httpd/ssl_err or_log (deleted)
httpd 28995 apache 11w REG 8,18 34015645 81923 /var/log/httpd/access_ log (deleted)
httpd 28995 apache 12w REG 8,18 2567398 81933 /var/log/httpd/ssl_acc ess_log (deleted)
httpd 28995 apache 13w REG 8,18 2864661 81936 /var/log/httpd/ssl_req uest_log (deleted)
java 30842 root cwd DIR 8,18 4096 2 /var/log
syslogd 31855 root 2w REG 8,18 716 15 /var/log/messages
syslogd 31855 root 3w REG 8,18 279595 22 /var/log/secure
syslogd 31855 root 4w REG 8,18 11315642 24 /var/log/maillog
syslogd 31855 root 5w REG 8,18 52466 27 /var/log/cron
syslogd 31855 root 6w REG 8,18 0 25 /var/log/spooler
syslogd 31855 root 7w REG 8,18 0 26 /var/log/boot.log

It looks like most all were deleted..the ones that are not are small.

[btisdall]

Restarting the daemons in question should sort that out.

[techsharp]

Edit:

Alright I stopped and started all the services in question (syslogd, httpd, clamd etc.)

It still is showing 97% full..

Anything else I could try? I was thinking of unmounting the drive and running fsck to see if it found any problems and then re-configured it to show the amount of free space.

[btisdall]

Well, anything referenced as 'deleted' by lsof should be restarted.

Edit: fixed code typo!

Code: Select all

lsof /var/log | grep '(deleted)' | awk  '{ print $1 }'

However, in the case of python the actual daemon is mailman, so you need to restart that.

Restarting the services should cause at most momentary disruption for your users.

[techsharp]

Hi btisdall,

Sorry - I just got through editing my post ;p

I did restart all the daemons including mailman and still shows the same thing - 97% full.

[btisdall]

What happens if you stop one of the services listed as having a deleted file open - does it still show up in the lsof output? If so you'll need to manually kill that process by its PID.

[techsharp]

Here is the output after I restarted the services:

[root@mars mailman]# lsof /var/log
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
stunnel 2331 nobody 3w REG 8,18 859724553 56 /var/log/stunnel.log (deleted)
bash 5400 root cwd DIR 8,18 4096 2 /var/log
syslogd 5579 root 1w REG 8,18 1512 15 /var/log/messages
syslogd 5579 root 2w REG 8,18 294314 22 /var/log/secure
syslogd 5579 root 3w REG 8,18 13012944 24 /var/log/maillog
syslogd 5579 root 4w REG 8,18 57637 27 /var/log/cron
syslogd 5579 root 5w REG 8,18 0 25 /var/log/spooler
syslogd 5579 root 6w REG 8,18 0 26 /var/log/boot.log
clamd 5779 clamav 3w REG 8,18 5980 16 /var/log/clamd.log
freshclam 5785 clamav 3w REG 8,18 24245 19 /var/log/freshclam.log
python 5940 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5943 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5943 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5943 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5944 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5944 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5944 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5945 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5945 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5945 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5946 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5946 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5946 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5947 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5947 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5947 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5948 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5948 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5948 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5949 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5949 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5949 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
python 5950 mailman 3u REG 8,18 0 65566 /var/log/mailman/error
python 5950 mailman 4u REG 8,18 0 65566 /var/log/mailman/error
python 5950 mailman 5u REG 8,18 6083 65579 /var/log/mailman/qrunner
httpd 6292 root 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6292 root 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6292 root 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6292 root 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6292 root 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6292 root 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6294 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6294 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6294 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6294 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6294 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6294 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6295 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6295 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6295 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6295 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6295 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6295 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6296 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6296 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6296 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6296 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6296 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6296 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6297 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6297 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6297 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6297 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6297 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6297 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6298 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6298 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6298 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6298 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6298 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6298 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6299 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6299 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6299 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6299 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6299 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6299 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6300 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6300 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6300 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6300 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6300 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6300 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
httpd 6301 apache 2w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6301 apache 9w REG 8,18 450 81922 /var/log/httpd/error_log
httpd 6301 apache 10w REG 8,18 0 81923 /var/log/httpd/ssl_error_log
httpd 6301 apache 11w REG 8,18 310119 81924 /var/log/httpd/access_log
httpd 6301 apache 12w REG 8,18 9179 81925 /var/log/httpd/ssl_access_log
httpd 6301 apache 13w REG 8,18 9713 81926 /var/log/httpd/ssl_request_log
su 9821 root cwd DIR 8,18 4096 2 /var/log
bash 9863 root cwd DIR 8,18 4096 65537 /var/log/mailman
lsof 11404 root cwd DIR 8,18 4096 65537 /var/log/mailman
lsof 11405 root cwd DIR 8,18 4096 65537 /var/log/mailman
bash 28431 tformoso cwd DIR 8,18 4096 2 /var/log
java 30842 root cwd DIR 8,18 4096 2 /var/log

All the deleted are gone except for the one stunnel..

[techsharp]

I just killed the stunnel (deleted) PID and now..

/dev/sdb2 1008M 99M 858M 11% /var/log

It now shows the correct amount of space being used.

Thank you very much! Very helpful!

I will have to watch our stunnel log file - I think that grows fast.

[btisdall]

Check the manpage for stunnel - you may have the log level set inappropriately high. The default is 5 but 4 seems more appropriate for a working installation.

Also check that logrotate is configured to manage the stunnel logs.

Best,

Ben.

[techsharp]

OK I will do that - as of today it was fine - maybe it had not been cleared in 5-6 months or so.

I will check what you said as well - thanks again for the help.