Server Crash - Where do I look

[dougp23]

I came into work this morning, and when I checked for new mail, it said "Contacting server, sending login and password" and just sat there. Through the browser, webmail and sac were unavailable. "tail -f /var/log/maillog" came up empty and did nothing. "ps ef | grep mail| came up with two VERY long sentences.

To keep my tale short, I had to reboot the server which immediately started complaining about orphan inodes and such. Running "fsck" got me back up and going.

So I want to know, where do I start the research to see what caused the server to bomb out like this? var/log/messages doesn't seem to show anything. Any scalix logs I should look for? Maybe a malformed MIME brought it down?

Would appreciate any help!

[ScalixSupport]

Hi!

Can you run the commands below:
omstat -a
omstat -s
omshowlog -l 3
ps ax | grep tomcat

Thanks,
Subir

[dougp23]

(Note that I rebooted at 8:15)

omstat -a

PC Monitor Started NON-STOP 0
Directory Relay Server Started 08:15:08
Notification Server Started 08:15:08 0
Shared memory daemon Started NON-STOP
Notification Monitor Started NON-STOP
Session Monitor Started NON-STOP
Indexer Started NON-STOP
Stats Daemon Started NON-STOP
Container Access Monitor Started NON-STOP
Item Structure Server Stopped
Database Monitor Started 08:15:08
Licence Monitor Daemon Started NON-STOP
LDAP Daemon Started 08:15:08
Queue Manager Started NON-STOP
Item Delete Daemon Started NON-STOP
IMAP Server Daemon Started 08:15:08
SMTP Relay Started 08:15:08
Mime Browser Controller Started 08:15:09
Event Server Started 08:15:09

omstat -s

Service Router Started 08:15:11 0
Local Delivery Started 08:15:11 0
Internet Mail Gateway Started 08:15:11 0
Local Client Interface Enabled 04.02.07 0
Remote Client Interface Enabled 04.02.07 0
Test Server Started 08:15:11 0
Request Server Started 08:15:11 0
Print Server Started 08:15:11 0
Bulletin Board Server Started 08:15:11 0
Background Search Service Started 08:15:11 0
CDA Server Started 08:15:11 0
POP3 interface Started 08:15:11 0
Omscan Server Started 08:15:11 0
Archiver Started 08:15:11 0


omshowlog -l 3

ERROR Browser (Service 14 ) 04.17.07 15:26:58
[OM.MIME 4000] Browser Args :index.browse -c -o /var/opt/scalix/ql/s/temp/mime_c ache/mimeT8xy8r 00011a854eee210e
Last Msg Id: 23316890.1176826733114.JavaMail.SYSTEM(a)lhrbod-aplxw006
Last Msg DirectRef: 00011a9a1d7797e5


ERROR Browser (Service 14 ) 04.17.07 15:26:58
[OM.MIME 4000] Browser Args :index.browse -c -o /var/opt/scalix/ql/s/temp/mime_c ache/mimeR8VVNa 00011a854eee210e
Last Msg Id: 005901c78125(036)2326d8f0(036)0a01a8c0(a)ossrec170f70e2
Last Msg DirectRef: 00011a8ad9ad2c39


ERROR Browser (Service 14 ) 04.18.07 12:15:41
[OM.MIME 4000] Browser Args :index.browse -c -o /var/opt/scalix/ql/s/temp/mime_c ache/mimeLyIvLX 00011aed08ccf711
Last Msg Id: NKEGKJFFHDNMEEFPNBKJEEMJCFAA.bbenvenuti(a)new.gov
Last Msg DirectRef: 00011aeb91912d97


ERROR Browser (Service 14 ) 04.18.07 12:15:42
[OM.MIME 4000] Browser Args :index.browse -c -o /var/opt/scalix/ql/s/temp/mime_c ache/mimeDuUhAx 00011aed08ccf711
Last Msg Id: AUTOANS-00064653.1176911871.qmail.new.gov
Last Msg DirectRef: 00011af282ddd148


ERROR Browser (Service 14 ) 04.18.07 12:15:42
[OM.MIME 4000] Browser Args :index.browse -c -o /var/opt/scalix/ql/s/temp/mime_c ache/mime3h0fp7 00011aed08ccf711
Last Msg Id: PNECLADJOBGLKGIKJCKFGEPACMAA.kcastle(a)new.gov
Last Msg DirectRef: 00011aede9220c82


ERROR Browser (Service 14 ) 04.18.07 12:15:42
[OM.MIME 4000] Browser Args :index.browse -c -o /var/opt/scalix/ql/s/temp/mime_c ache/mimeZshreH 00011aed08ccf711
Last Msg Id: AUTOANS-00064659.1176911882.qmail.new.gov
Last Msg DirectRef: 00011af981e83940

ps ax | grep tomcat

3777 ? Sl 0:49 /usr/java/jre1.5.0_06/bin/java -server -Djava.net.preferIPv4Stack=true -Dscalix.instance=/var/opt/scalix/ql -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/opt/scalix/ql/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/scalix-tomcat/common/endorsed -classpath /usr/java/jre1.5.0_06/lib/tools.jar:/opt/scalix-tomcat/bin/bootstrap.jar:/opt/scalix-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/var/opt/scalix/ql/tomcat -Dcatalina.home=/opt/scalix-tomcat -Djava.io.tmpdir=/var/opt/scalix/ql/tomcat/temp org.apache.catalina.startup.Bootstrap start


That's what I have!

[ScalixSupport]

Hi!

Run the command "omtidyallu -M" to pre-generate MIME and check again if you still
get such error(s). If the error still come, try clearing the IMAP cache for these users.
As you can see each value of "Last Msg DirectRef" are unique, each user has its own
unique "DirectRef".

Last Msg DirectRef: 00011a9a1d7797e5

To know the username, from the Direct reference value as above, you can identify the
user whose mailbox is returning the problem using the command:

Code: Select all

omdref <Direct reference number>

Note: <Direct reference number> is value next to Last Msg DirectRef in logs.

Once you know the username, we can follow the steps below to clear IMAP cache for
these users:
1. make sure the user is logged out, i.e.

Code: Select all

omstat -u all

should not show any more sessions for any user.

2. determine the user folder directory, i.e.

Code: Select all

omshowu -n <lastname> -f

Note: <lastname> should be the lastname of the user in question.

3. Go there (the so-called g-directory of the user)

Code: Select all

cd /var/opt/scalix/??/s/user/????????/

4. This should have a subdirectory called imap-cache. delete that:

Code: Select all

rm -r imap-cache

5. Try accessing the user's mailbox using IMAP again.

Now, restart scalix-tomcat service and try again.

Thanks,
Subir

[dougp23]

Thanks Subir!

Do you think it was a MIME issue?
What should I do next time? Maybe omoff and then omon?

[ScalixSupport]

Hi!

Yes, you can run an omoff-omon sequence for mbc, make sure you use -w option while
executing omoff.

Thanks,
Subir

[dougp23]

Each morning when I check mail for root, I have a little "system status" message. Mostly it tells me that clamav updated and some other stuff. But there is usually this under the httpd header:

--------------------- httpd Begin ------------------------


A total of 791 unidentified 'other' records logged
GET /sis/indexer?fn=add&uid=03300000bbbe6f54-012.1.861.291&pdref=00010a0b41656
d64&dref=00011c7a014bf155&indexid=8b885e4a-45f6ebbb-462d0c84-679d&flags=unseen,u
nflagged,unanswered,undeleted,undraft,unlabel1,unlabel2,unlabel3,unlabel4,unlabe
l5,unlabel6,unlabel7,unlabel8,unjunk,unnonjunk,unforwarded HTTP/1.1 with respons
e code(s) 2 204 responses
POST /sis/indexer?fn=index&uid=03300000bbbe6f54-012.1.861.291&pdref=00010a0b41
656d64&dref=00011c42bbce2c82&indexid=8b885e4a-45f6ebbb-462d0267-676a&flags=unsee
n,unflagged,unanswered,undeleted,undraft,unlabel1,unlabel2,unlabel3,unlabel4,unl
abel5,unlabel6,unlabel7,unlabel8,unjunk,unnonjunk,unforwarded HTTP/1.1 with resp
onse code(s) 2 200 responses

So today it's 791 'other' records, other days it will be over 2,000. Is this OK??

[ScalixSupport]

Hi!

See the following post:
viewtopic.php?t=5497

This behavior is considered to be normal.

Thanks,
Subir