Scalix Forums

Skip to content

Lots of Errors and potentially server hangs

Discuss the Scalix Server software

Moderators: ScalixSupport, admin

Post Reply
asd_itops
Posts: 82
Joined: Tue Feb 06, 2007 12:33 pm

Lots of Errors and potentially server hangs

Postby asd_itops » Tue Aug 21, 2007 12:16 pm

Scalix 11.1.0, RHEL 4, fully patched and updated
~100users (mostly Outlook 2003, other webmail, evolution, or Mac iMail)

We have begun seeing a ton of these errors:

SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com


SERIOUS ERROR Local Delivery(Local Delivery) 08.21.07 09:58:07
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x926ee6]
/opt/scalix/lib/libom_cvc.so(cvc_enhCnvString+0x107)[0x3a9230]
/opt/scalix/lib/libom_cvc.so(cvc_ConvertString+0x3d)[0x3a9cc5]
/opt/scalix/lib/libom_rtfl.so(rtfl_BuildLine+0x3e9)[0x401897]
/opt/scalix/lib/libom_rtfl.so[0x403a50]
/opt/scalix/lib/libom_rtfl.so(rtfl_Parse+0x175)[0x40531e]
/opt/scalix/lib/libom_rtfl.so(rtfl_search+0x109)[0x402d83]
/opt/scalix/lib/libom_flt.so[0x8a6ef9]
/opt/scalix/lib/libom_flt.so(flt_ApplyTextMatch+0xe1)[0x8a7036]
/opt/scalix/lib/libom_flt.so(Test_TextBody_Att+0x1eb)[0x8a4afe]
/opt/scalix/lib/libom_flt.so(flt_ApplySingle+0x6c8)[0x8a36c0]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x228)[0x8a29da]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyOrGroup+0xb6)[0x8a2ca5]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1e8)[0x8a299a]
/opt/scalix/lib/libom_flt.so(flt_ApplyAndGroup+0xb6)[0x8a2adc]
/opt/scalix/lib/libom_flt.so(flt_ApplyNextFilter+0x1c9)[0x8a297b]
/opt/scalix/lib/libom_flt.so(flt_ApplyOuterGroup+0xcb)[0x8a2663]
/opt/scalix/lib/libom_flt.so(flt_ApplyFC+0x140)[0x8a2184]
local.delivery 21381304[0x8057d95]
local.delivery 21381304[0x8053160]
local.delivery 21381304[0x805c367]
local.delivery 21381304[0x805e05f]
Last Msg Id: L7EB8DB4772934b9a86EBD02C3E1ACCCD.1187715478.rhel-sv-mail.allstardirectories.com



And the ubiqitous statuses to help in triage:

[root@rhel-sv-mail ~]# omstat -a
PC Monitor Started NON-STOP 0
Directory Relay Server Started 07:17:11
Notification Server Started 07:17:11 0
Shared memory daemon Started NON-STOP
Notification Monitor Started NON-STOP
Session Monitor Started NON-STOP
Indexer Started NON-STOP
Stats Daemon Started NON-STOP
Container Access Monitor Started NON-STOP
Item Structure Server Started 07:17:11
Database Monitor Started 07:17:11
Licence Monitor Daemon Started NON-STOP
LDAP Daemon Started 07:17:11
Queue Manager Started NON-STOP
Item Delete Daemon Started NON-STOP
IMAP Server Daemon Started 07:17:11
SMTP Relay Started 07:17:11
Mime Browser Controller Started 07:17:11
Event Server Started 07:17:11
[root@rhel-sv-mail ~]# omstat -s
Service Router Started 07:17:12 0
Local Delivery Started 07:17:12 0
Internet Mail Gateway Started 07:17:12 0
Sendmail Interface Started 07:17:12 0
Local Client Interface Enabled 07:17:12 0
Remote Client Interface Enabled 07:17:12 88
Test Server Started 07:17:12 0
Request Server Started 07:17:12 0
Print Server Started 07:17:12 0
Directory Synchronization Started 07:17:12 0
Bulletin Board Server Started 07:17:12 0
Background Search Service Started 07:17:12 0
Dump Server Started 07:17:12 0
CDA Server Started 07:17:12 0
POP3 interface Started 07:17:12 0
Omscan Server Started 07:17:12 0
Archiver Started 07:17:12 0

[root@rhel-sv-mail ~]# ps aux | grep java
root 3866 2.8 21.1 1099196 878072 ? Sl 07:17 4:41 /usr/java/jre1.5.0_11/bin/java -server -Djava.net.preferIPv4Stack=true -Xms768m -Xmx768m -Dscalix.instance=/var/opt/scalix/rl -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/opt/scalix/rl/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/scalix-tomcat/common/endorsed -classpath /usr/java/jre1.5.0_11/lib/tools.jar:/opt/scalix-tomcat/bin/bootstrap.jar:/opt/scalix-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/var/opt/scalix/rl/tomcat -Dcatalina.home=/opt/scalix-tomcat -Djava.io.tmpdir=/var/opt/scalix/rl/tomcat/temp org.apache.catalina.startup.Bootstrap start

:( :!:
Top
mikethebike
Posts: 566
Joined: Mon Nov 28, 2005 4:16 pm
Location: England

Postby mikethebike » Tue Aug 21, 2007 1:13 pm

Are they all errors for the same message? I would be tempted to check the audit log, and note the subject.
Then set up a rule to capture it at the Service Router.

Mick
Top
asd_itops
Posts: 82
Joined: Tue Feb 06, 2007 12:33 pm

Postby asd_itops » Tue Aug 21, 2007 3:13 pm

There are about 10-15 different messages. The two I checked out as samples were nothing fancy... simple messages to individually addressed people on the same server sent from Outlook 2003 with Scalix 11.1.046. Simple Rich text Message. I was unabel to reproduce by replying to or forwarding the message on my machine... but other people who had replied to or forwarded those messages did sometimes reproduce.
Top
mikethebike
Posts: 566
Joined: Mon Nov 28, 2005 4:16 pm
Location: England

Postby mikethebike » Wed Aug 22, 2007 4:30 am

It seems real strange to me there are so many entries for the same message id?
Top
gren
Scalix
Scalix
Posts: 264
Joined: Thu Mar 25, 2004 10:27 am
Contact:
Contact gren

Postby gren » Thu Aug 23, 2007 1:12 pm

Are there any messages on your poison queue? If local delivery keeps on failing to deliver the same message it typically evenutally moves it to the poison queue.

omstat -q POISON

If you have any such messages, then using omqdump to dump them to files, and tarring up the results and sending them to me would be great. From that we may be able to track down the bug that is causing the issue.

omqdump's password is 'A##E' where ## is the current day of the month + 10.
Use the "g" command to GET the message from the queue.
Use the "o" command to output the open message to files.

I'm Gren Elliot, from which you can guess my email address :-)

Thanks and regards,
Gren.
Top
asd_itops
Posts: 82
Joined: Tue Feb 06, 2007 12:33 pm

Postby asd_itops » Thu Aug 23, 2007 1:44 pm

poison queue is empty, but we are still seeing about 100 errors a day related to this
Top
mikethebike
Posts: 566
Joined: Mon Nov 28, 2005 4:16 pm
Location: England

Postby mikethebike » Thu Aug 23, 2007 5:42 pm

do the recipients have any rules set?
1. Check in the audit log, searching for the message-id, and see who its being delivered to.
2. See if they have any strange rules. Look in their "g" directory, if they have a 000003d file, rename it, that will disable server side rules. tfbrowse the file (tfbrowse -i 000003d) to see what they have set.
3. Are there any routing rules? (omshowrt -q all -d). Check that they are not doing anything funky with that message.
4. Check the recipient mailboxes for any corruption, maybe do a quick active omscan on the recipient mailboxes (and copy them out to make sure they copy OK).

Mick
Top
Richard Hall
Scalix
Scalix
Posts: 147
Joined: Fri May 20, 2005 5:37 am

Postby Richard Hall » Fri Aug 24, 2007 3:50 am

I believe this is another case of:
viewtopic.php?t=7885

No serious error is really occurring, and this is fix in the 11.2 release:
http://bugzilla.scalix.com/show_bug.cgi?id=15541
Top
Post Reply

Return to “Scalix Server”



Who is online

Users browsing this forum: Google [Bot] and 3 guests

Powered by phpBB® Forum Software © phpBB Limited