Scalix Forums


http://www.scalix.com/forums/

Lots of Errors and potentially server hangs

http://www.scalix.com/forums/viewtopic.php?f=2&t=8551

Page 1 of 1

Lots of Errors and potentially server hangs

Posted: Tue Aug 21, 2007 12:16 pm
by asd_itops
Scalix 11.1.0, RHEL 4, fully patched and updated
~100users (mostly Outlook 2003, other webmail, evolution, or Mac iMail)

We have begun seeing a ton of these errors:

SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com



And the ubiqitous statuses to help in triage:

[root@rhel-sv-mail ~]# omstat -a
PC Monitor Started NON-STOP 0
Directory Relay Server Started 07:17:11
Notification Server Started 07:17:11 0
Shared memory daemon Started NON-STOP
Notification Monitor Started NON-STOP
Session Monitor Started NON-STOP
Indexer Started NON-STOP
Stats Daemon Started NON-STOP
Container Access Monitor Started NON-STOP
Item Structure Server Started 07:17:11
Database Monitor Started 07:17:11
Licence Monitor Daemon Started NON-STOP
LDAP Daemon Started 07:17:11
Queue Manager Started NON-STOP
Item Delete Daemon Started NON-STOP
IMAP Server Daemon Started 07:17:11
SMTP Relay Started 07:17:11
Mime Browser Controller Started 07:17:11
Event Server Started 07:17:11
[root@rhel-sv-mail ~]# omstat -s
Service Router Started 07:17:12 0
Local Delivery Started 07:17:12 0
Internet Mail Gateway Started 07:17:12 0
Sendmail Interface Started 07:17:12 0
Local Client Interface Enabled 07:17:12 0
Remote Client Interface Enabled 07:17:12 88
Test Server Started 07:17:12 0
Request Server Started 07:17:12 0
Print Server Started 07:17:12 0
Directory Synchronization Started 07:17:12 0
Bulletin Board Server Started 07:17:12 0
Background Search Service Started 07:17:12 0
Dump Server Started 07:17:12 0
CDA Server Started 07:17:12 0
POP3 interface Started 07:17:12 0
Omscan Server Started 07:17:12 0
Archiver Started 07:17:12 0

[root@rhel-sv-mail ~]# ps aux | grep java
root 3866 2.8 21.1 1099196 878072 ? Sl 07:17 4:41 /usr/java/jre1.5.0_11/bin/java -server -Djava.net.preferIPv4Stack=true -Xms768m -Xmx768m -Dscalix.instance=/var/opt/scalix/rl -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/opt/scalix/rl/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/scalix-tomcat/common/endorsed -classpath /usr/java/jre1.5.0_11/lib/tools.jar:/opt/scalix-tomcat/bin/bootstrap.jar:/opt/scalix-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/var/opt/scalix/rl/tomcat -Dcatalina.home=/opt/scalix-tomcat -Djava.io.tmpdir=/var/opt/scalix/rl/tomcat/temp org.apache.catalina.startup.Bootstrap start

:( :!:

Posted: Tue Aug 21, 2007 1:13 pm
by mikethebike
Are they all errors for the same message? I would be tempted to check the audit log, and note the subject.
Then set up a rule to capture it at the Service Router.

Mick

Posted: Tue Aug 21, 2007 3:13 pm
by asd_itops
There are about 10-15 different messages. The two I checked out as samples were nothing fancy... simple messages to individually addressed people on the same server sent from Outlook 2003 with Scalix 11.1.046. Simple Rich text Message. I was unabel to reproduce by replying to or forwarding the message on my machine... but other people who had replied to or forwarded those messages did sometimes reproduce.

Posted: Wed Aug 22, 2007 4:30 am
by mikethebike
It seems real strange to me there are so many entries for the same message id?

Posted: Thu Aug 23, 2007 1:12 pm
by gren
Are there any messages on your poison queue? If local delivery keeps on failing to deliver the same message it typically evenutally moves it to the poison queue.

omstat -q POISON

If you have any such messages, then using omqdump to dump them to files, and tarring up the results and sending them to me would be great. From that we may be able to track down the bug that is causing the issue.

omqdump's password is 'A##E' where ## is the current day of the month + 10.
Use the "g" command to GET the message from the queue.
Use the "o" command to output the open message to files.

I'm Gren Elliot, from which you can guess my email address :-)

Thanks and regards,
Gren.

Posted: Thu Aug 23, 2007 1:44 pm
by asd_itops
poison queue is empty, but we are still seeing about 100 errors a day related to this

Posted: Thu Aug 23, 2007 5:42 pm
by mikethebike
do the recipients have any rules set?
1. Check in the audit log, searching for the message-id, and see who its being delivered to.
2. See if they have any strange rules. Look in their "g" directory, if they have a 000003d file, rename it, that will disable server side rules. tfbrowse the file (tfbrowse -i 000003d) to see what they have set.
3. Are there any routing rules? (omshowrt -q all -d). Check that they are not doing anything funky with that message.
4. Check the recipient mailboxes for any corruption, maybe do a quick active omscan on the recipient mailboxes (and copy them out to make sure they copy OK).

Mick

Posted: Fri Aug 24, 2007 3:50 am
by Richard Hall
I believe this is another case of:
viewtopic.php?t=7885

No serious error is really occurring, and this is fix in the 11.2 release:
http://bugzilla.scalix.com/show_bug.cgi?id=15541
All times are UTC-04:00
Page 1 of 1
Powered by phpBB® Forum Software © phpBB Limited
https://www.phpbb.com/