New Up date on Local Delivery Service Mail Problem

[BigBirdy]

I have more detailed info on my issue of local mail not getting out of the "Local Delivery" que, so I thought it best to start a new thread as it does not appear to be related to the symbolic link to the /var/opt/scalix directory.

RHES4 System

* Long before running the upgrade to V10, I moved the /var/opt/scalix directory to /home/scalix and created a symbolic link from /var/opt/scalix to /home/scalix. Everything worked fine.
* Last week I upgraded to V10 but never did a reboot (not sure if this is where/how the problem started), and everything worked fine and all has been working great since.
* Yesterday I had a powerbar blox and so the server shutdown abruptly. I simply rebooted and although everything seemed fine, later in the day I noticed no new mail arriving in any of the mailboxes.
* I was in Webmin doing other things and went to the Sendmail area and into "Read User Mail" and get the error below. But I am not sure when this started to happen and can only guess this started after the reboot.
* Later I also noticed that due to misstakenly sending some large files to the wrong location, the root directory/drive filled completely. But this is now fixed and there is around 2gb of space available.

Webmin error when going to Read User Mail module:

HTTP/1.0 500 Perl execution failed Server: MiniServ/0.01
Date: Sat, 11 Mar 2006 17:41:03 GMT Content-type: text/html

* In Scalix the "Local Delivery" service wont start and so the mail is getting que'd. Below is the error log for the local delivery service.

SERIOUS ERROR Local Delivery(Local Delivery) 03.11.06 08:19:08
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 23400
Procedure trace follows:
-> nm_AppendFieldMem
-> nm_AddSeparators
<- nm_AddSeparators
-> nm_ParseORN
<- nm_ParseORN
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_PutFieldMem
-> nm_AppendFieldMem
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_ParseORN
<- nm_ParseORN
<- ul_utUnpackUserEnt
<- ul_FindPrimeUser
<- ul_IsEnu

SERIOUS ERROR Local Delivery(Local Delivery) 03.11.06 08:19:08
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x51df06]
/opt/scalix/lib/libom_er.so[0x51e1d6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x51e37f]
/lib/tls/libpthread.so.0[0xc2a7c8]
[0x696c6e69]

* I can also login as a local user on the server and send mail with "mail -s "Test" stork@localhost" and it seems to work fine?

* I have now also seen this message in the /var/log/messages

Mar 11 13:33:13 penguin ctl_mboxlist[11747]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming the worst: No such file or directory
Mar 11 13:33:13 penguin ctl_mboxlist[11747]: skiplist: recovered /var/lib/imap/mailboxes.db (0 records, 144 bytes) in 0 seconds

I am at a loss to determine the problem or how to repair and I dont have a backup of the /var/lib/imap folder if that is the problem.

[ScalixSupport]

/var/lib/imap/ is nothing to do with Scalix so we're not in a position to check up on this.

For Local Delivery, how many messages are queuing ? What can you see in the fatal log ? Is there any more information other than what you have posted.

One command to run is omshowenu. This will give you the address of the user who is configured to be the Error Notification user.

You should also check the permissions on /home/scalix to make sure that it is owned by scalix:scalix.

Cheers

Dave

[BigBirdy]

Right now I have over 600 messages in the que!!! System gets around 800-1000 messages a day...This is getting more critical by the minute!! Please, please try and offer more assistance, I have also sent a couple mails to support but no reply at all as yet!!

Results of omshwoenu indicate that I am the Error Notification User, but since I cant get my mail that doesnt do me any good.

Results of "ls -la /home/scalix

total 256
drwxrwxr-x 53 scalix scalix 4096 Feb 19 17:25 .
drwxr-xr-x 8 root root 4096 Mar 9 20:12 ..
drwxrwxr-x 5 scalix scalix 4096 May 21 2005 access
drwxrwx--x 8 scalix scalix 4096 May 21 2005 acl
drwxrwxr-x 4 scalix scalix 4096 Mar 11 20:26 admin
drwxrwx--- 3 scalix scalix 4096 May 25 2005 bb
drwxrwx--- 3 scalix scalix 4096 May 21 2005 cda
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:26 Copyright
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:26 csetconv
drwxrwx--- 194 scalix scalix 4096 May 26 2005 data
drwxrwx--- 3 scalix scalix 4096 Feb 19 17:26 dblogs
drwxrwx--x 2 scalix scalix 4096 May 21 2005 diag
drwxrwx--- 3 scalix scalix 4096 May 21 2005 dir
drwxrwx--- 2 scalix scalix 4096 Mar 10 07:12 dirchlog
drwxrwx--- 2 scalix scalix 4096 May 21 2005 dits
drwxrwx--- 2 scalix scalix 4096 Feb 19 17:22 ds
drwx------ 2 scalix scalix 4096 May 21 2005 .elinks
drwxrwxr-x 3 scalix scalix 4096 May 21 2005 is
drwxrwx--x 2 scalix scalix 4096 May 21 2005 lic
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:25 License
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:25 LicenseKeys
drwxrwxr-x 2 scalix scalix 4096 Mar 11 18:15 logs
drwxrwx--x 2 scalix scalix 4096 May 21 2005 msglists
drwxrwx--x 2 scalix scalix 4096 May 23 2005 msgpool
drwxrwxr-x 2 scalix scalix 4096 May 21 2005 nits
drwxrwxr-x 15 scalix scalix 4096 May 21 2005 nls
drwxrwxr-x 3 scalix scalix 4096 Mar 11 20:26 notifs
drwxrwxr-x 2 scalix scalix 4096 May 21 2005 notify
drwxr-xr-x 2 scalix scalix 4096 Mar 11 12:26 oldDbLogs
drwxrwxr-x 51 scalix scalix 4096 Feb 19 17:25 omhtml
drwxrwx--- 4 scalix scalix 4096 Jun 23 2005 orphans
drwxrwx--- 2 scalix scalix 4096 May 21 2005 pmove.store
drwxrwx--- 2 scalix scalix 4096 Mar 11 20:26 pop3
drwxrwx--x 2 scalix scalix 4096 May 21 2005 queues
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:25 release.cst
drwxrwxr-x 8 scalix scalix 4096 Feb 19 17:25 release.sys
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:25 release.tou
drwxrwxr-x 2 scalix scalix 4096 May 21 2005 req
drwxrwx--- 2 scalix scalix 4096 Nov 1 05:13 rules
drwxrwxrwx 2 scalix scalix 4096 Feb 19 17:25 rules.eg
drwxrwx--x 2 scalix scalix 4096 Mar 10 11:13 scan.serv
drwxrwx--x 2 scalix scalix 4096 Mar 5 00:00 scan.work
drwxrwxr-x 2 scalix scalix 4096 Mar 11 20:26 se
drwxrwxr-x 2 scalix scalix 4096 May 21 2005 structdb
drwxrwxr-x 2 scalix scalix 4096 May 21 2005 structlog
drwxrwxr-x 18 scalix scalix 4096 Mar 11 12:26 sys
drwxrwx--x 4 scalix scalix 45056 Mar 11 20:26 temp
drwxrwxrwx 2 scalix scalix 4096 Mar 11 12:26 tmp
drwxrwxr-x 2 scalix scalix 4096 Feb 19 17:26 toupper
drwxrwx--- 2 scalix scalix 4096 May 21 2005 trace
drwxrwx--x 12 scalix scalix 4096 Feb 19 19:40 user
drwxrwxr-x 2 scalix scalix 4096 May 21 2005 webusers
drwxrwx--x 2 scalix scalix 4096 May 21 2005 xport.hold

So all is owned by scalix:scalix (including the /home/scalix directory)

/var/opt/scalix/logs/fatal.log (after a "services scalix restart"):

SERIOUS ERROR Local Delivery(Local Delivery) Sat Mar 11 20:35:22 2006
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 24450
Procedure trace follows:
-> nm_AppendFieldMem
-> nm_AddSeparators
<- nm_AddSeparators
-> nm_ParseORN
<- nm_ParseORN
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_PutFieldMem
-> nm_AppendFieldMem
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_ParseORN
<- nm_ParseORN
<- ul_utUnpackUserEnt
<- ul_FindPrimeUser
<- ul_IsEnu
Pid of logging process: 24450


SERIOUS ERROR Local Delivery(Local Delivery) Sat Mar 11 20:35:22 2006
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x613f06]
/opt/scalix/lib/libom_er.so[0x6141d6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x61437f]
/lib/tls/libpthread.so.0[0xc2a7c8]
[0x696c6e69]
Pid of logging process: 24450

[ScalixSupport]

Please can you mail your SSH details to the support address.

Cheers

Dave

[BigBirdy]

Not readily available, this is the office's test environment, for pre-production rollouts of upgrades, including for Scalix. its another business location. So I would prefer to try a few more things before opening things up, but of course if its the last resort. I also tried something else just to eliminate some variables. I free'd up some space on /, removed the symblic link from /var/opt/scalix to /home/scalix, then did a "cp -r -p /home/scalix /var/opt and restarted.

I got this in the /var/log/maillog

Mar 11 20:57:07 penguin sendmail[26996]: k2C4v7jI026990: to=Johnny.Stork@johnnystork.ca, ctladdr=<nagios@penguin.johnnystork.ca> (103/105), delay=00:00:00, xdelay=00:00:00, mailer=scalix, pri=30825, relay=penguin, dsn=4.0.0, stat=Deferred: 421 Message rejected by OM.UX Gateway on 'penguin': Gateway not enabled

And now in /var/opt/scalix/logs/fatal.log after restarting I have one more entry..about the que manager..

SERIOUS ERROR Administration(omstat ) Sat Mar 11 20:57:05 2006
[OM.DMON 1013] The queue manager is not running.
Pid of logging process: 26977

SERIOUS ERROR Local Delivery(Local Delivery) Sat Mar 11 20:57:37 2006
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 27217
Procedure trace follows:
-> nm_AppendFieldMem
-> nm_AddSeparators
<- nm_AddSeparators
-> nm_ParseORN
<- nm_ParseORN
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_PutFieldMem
-> nm_AppendFieldMem
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_ParseORN
<- nm_ParseORN
<- ul_utUnpackUserEnt
<- ul_FindPrimeUser
<- ul_IsEnu
Pid of logging process: 27217

SERIOUS ERROR Local Delivery(Local Delivery) Sat Mar 11 20:57:37 2006
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x1b0f06]
/opt/scalix/lib/libom_er.so[0x1b11d6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x1b137f]
/lib/tls/libpthread.so.0[0x1a67c8]
[0x696c6e69]
Pid of logging process: 27217

[BigBirdy]

Do you think running the installer again might help? Just re-install all components?

[ScalixSupport]

No, I don't think that will help.

Providing us remote access to your server is going to resolve this a lot quicker than playing e-mail tennis back and forth.

I'm not sure I understand your configuration, are you saying that this is a production server in a test environment ?

Cheers

Dave

[ScalixSupport]

It's gone quiet. Are you still there ?

Cheers

Dave

[BigBirdy]

Sorry, took a break. No this is a test environment for the office, but its run in another location. It was created to be identical, right down to the symbolically linked /var/opt/scalix location, since there was no space, nor spare servers in the office for testing patches and updates prior to deployment. I will follow up in the morning with the SSH details. Should have them around 7-8 am my time (Pacific).

Thanks for your help (will send it to the support email)

[jorgen3048]

I've got the exact same error message, any solution to this problem??

I have more detailed info on my issue of local mail not getting out of the "Local Delivery" que, so I thought it best to start a new thread as it does not appear to be related to the symbolic link to the /var/opt/scalix directory.

RHES4 System

* Long before running the upgrade to V10, I moved the /var/opt/scalix directory to /home/scalix and created a symbolic link from /var/opt/scalix to /home/scalix. Everything worked fine.
* Last week I upgraded to V10 but never did a reboot (not sure if this is where/how the problem started), and everything worked fine and all has been working great since.
* Yesterday I had a powerbar blox and so the server shutdown abruptly. I simply rebooted and although everything seemed fine, later in the day I noticed no new mail arriving in any of the mailboxes.
* I was in Webmin doing other things and went to the Sendmail area and into "Read User Mail" and get the error below. But I am not sure when this started to happen and can only guess this started after the reboot.
* Later I also noticed that due to misstakenly sending some large files to the wrong location, the root directory/drive filled completely. But this is now fixed and there is around 2gb of space available.

Webmin error when going to Read User Mail module:

HTTP/1.0 500 Perl execution failed Server: MiniServ/0.01
Date: Sat, 11 Mar 2006 17:41:03 GMT Content-type: text/html

* In Scalix the "Local Delivery" service wont start and so the mail is getting que'd. Below is the error log for the local delivery service.

SERIOUS ERROR Local Delivery(Local Delivery) 03.11.06 08:19:08
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 23400
Procedure trace follows:
-> nm_AppendFieldMem
-> nm_AddSeparators
<- nm_AddSeparators
-> nm_ParseORN
<- nm_ParseORN
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_PutFieldMem
-> nm_AppendFieldMem
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_ParseORN
<- nm_ParseORN
<- ul_utUnpackUserEnt
<- ul_FindPrimeUser
<- ul_IsEnu

SERIOUS ERROR Local Delivery(Local Delivery) 03.11.06 08:19:08
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0x51df06]
/opt/scalix/lib/libom_er.so[0x51e1d6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x51e37f]
/lib/tls/libpthread.so.0[0xc2a7c8]
[0x696c6e69]

* I can also login as a local user on the server and send mail with "mail -s "Test" stork@localhost" and it seems to work fine?

* I have now also seen this message in the /var/log/messages

Mar 11 13:33:13 penguin ctl_mboxlist[11747]: DBERROR: reading /var/lib/imap/db/skipstamp, assuming the worst: No such file or directory
Mar 11 13:33:13 penguin ctl_mboxlist[11747]: skiplist: recovered /var/lib/imap/mailboxes.db (0 records, 144 bytes) in 0 seconds

I am at a loss to determine the problem or how to repair and I dont have a backup of the /var/lib/imap folder if that is the problem.

[ScalixSupport]

What "exact same" error message are you getting? The webmin error, the Scalix error or the Cyrus IMAP error?

Please post relevant logfiles, thanks.

Cheersm

Sascha.

[jorgen3048]

This is from the fatal log:
SERIOUS ERROR Local Delivery(Local Delivery) Mon May 15 13:46:24 2006
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 2601
Procedure trace follows:
-> nm_AppendFieldMem
-> nm_AddSeparators
<- nm_AddSeparators
-> nm_ParseORN
<- nm_ParseORN
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_PutFieldMem
-> nm_AppendFieldMem
<- nm_AppendFieldMem
<- nm_PutFieldMem
-> nm_ParseORN
<- nm_ParseORN
<- ul_utUnpackUserEnt
<- ul_FindPrimeUser
<- ul_IsEnu
Pid of logging process: 2601


SERIOUS ERROR Local Delivery(Local Delivery) Mon May 15 13:46:24 2006
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0xb1cf06]
/opt/scalix/lib/libom_er.so[0xb1d1d6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0xb1d37f]
/lib/tls/libpthread.so.0[0xcf3888]
Pid of logging process: 2601


omstat -s:
Service Router Started 13:27:23 0
Local Delivery Partially Abor 13:30:10 268
Internet Mail Gateway Started 13:27:23 0
Local Client Interface Enabled 13:27:23 0
Remote Client Interface Enabled 13:27:23 3
Test Server Started 13:27:23 0
Request Server Started 13:27:23 0
Print Server Started 13:27:23 0
Bulletin Board Server Started 13:27:23 0
Background Search Service Started 13:27:23 0
CDA Server Started 13:27:24 0
POP3 interface Started 13:27:24 0
Omscan Server Started 13:27:24 0
Archiver Started 13:27:24 0

[ScalixSupport]

ok, please show us the output of

omshowenu

and

omshowu -n "name of above user"

This looks like there are too many Internet addresses set for this user. But let's see what we get here, first.

Cheers,

Sascha.

[jorgen3048]

[root@webftp bin]# omshowenu
Jorgen Jonsson /webftp,saitek/CN=Jorgen Jonsson
[root@webftp bin]# omshowu -n "Jorgen Jonsson"
Authentication ID: jorgenj
User Name : Jorgen Jonsson /CN=Jorgen Jonsson
MailNode : webftp,saitek
Internet Address : "Jorgen Jonsson" <Jorgen.Jonsson@saitek.se>="Jorgen Jonsson" <abuse@saitek.se>="Jorgen Jonsson" <root@saitek.se>="Jorgen Jonsson" <bigb@saitek.se>="Jorgen Jonsson" <jorgen@saitek.se>="Jorgen Jonsson" <jorgen_jonsson@saitek.se>="Jorgen Jonsson" <lionking@saitek.se>="Jorgen Jonsson" <jorgen.jonsson@webftp.saitek.se>="Jorgen Jonsson" <apache@webftp.saitek.se>
System Login : 60542
Password : set
Admin Capabilities : YES
Mailbox Admin Capabilities : NO
Language : AMERICAN
Virtual Vault : Enabled (default)
Mail Account: Unlocked
Last Signon : 05.15.06 13:33:14
Receipt of mail : ENABLED
Service level : 0
Excluded from Tidying : NO
User Class : Full

[ScalixSupport]

OK, please try: using SAC trim the email addresses down to, say three.

Then, run

omoff -d0 ld; omon ld

Cheers,

Sascha.