Messages Don't Send Out

[mjkruzeliii]

I am running Scalix 11.3 on OpenSus 10.2.

I have ran into a problem with messages not sending out. On a clean boot everything works fine. The next morning I come in and Scalix won't send out any emails to an external domain ... (all messages sent from our domain to our domain work). It looks like Scalix has these messages queued under 'Internet Mail Gateway'. When I reboot the machine the messages start sending out and will work through the work day. I have not been able to determine what causes the crash and how long it takes before it stops sending messages out.

This is the case for both SWA and Outlook (not using the Connector).

Any help is greatly appreciated.

[gren]

If you do

Code: Select all

omstat -s

when this occurs, does the internet gateway show up as aborted?

Did you try restarting the Internet gateway?

Code: Select all

omoff -d0 -w unix
omon unix

Then use

Code: Select all

omstat -q UNIX

to see whether the internet gateway is successfully processing messages.

Hope this helps,
Gren.

[les]

If you do

Code: Select all

omstat -s

when this occurs, does the internet gateway show up as aborted?

Did you try restarting the Internet gateway?

Code: Select all

omoff -d0 -w unix
omon unix

Then use

Code: Select all

omstat -q UNIX

to see whether the internet gateway is successfully processing messages.

Hope this helps,
Gren.

and....have a look in /var/opt/scalix/XX/s/logs/fatal

If any errors or crashes occurred with the internet mail gateway then they should be logged there.

[mjkruzeliii]

Gren, Les,
Thanks for the responses!

Gren:
localhost:~ # omstat -s
Service Router Started 07:54:54 0
Local Delivery Started 07:54:55 0
Internet Mail Gateway Started 07:54:55 468
Local Client Interface Enabled 07:54:55 0
Remote Client Interface Disabling 02.15.08 0
Test Server Started 07:54:55 0
Request Server Started 07:54:55 0
Print Server Started 07:54:55 0
Bulletin Board Server Started 07:54:55 0
Background Search Service Started 07:54:55 0
CDA Server Started 07:54:56 0
POP3 interface Started 07:54:56 0
Omscan Server Started 07:54:56 0
Archiver Started 07:54:56 0

This is how it looks when no messages are being relayed. I know on a restart that I won't lose any messages ... is the case with restarting the 'Internet Mail Gateway' with
Code:
omoff -d0 -w unix
omon unix


Les:
I have some smtp errors in the fatal file you referenced. These are the only types of errors listed in that file. Any thoughts as to what might be causing these errors? See Below:

ERROR SMTP Relay (SMTPD Relay Pr) Thu Feb 7 08:57:19 2008
[OM.DMON 2171] The SMTP Relay could not listen on port 25.
Pid of logging process: 5853


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4830] Error in starting up a subsystem.
Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4826] Can't abort process 5846 of
part aborted subsystem SMTP Relay.

Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4826] Can't abort process 5848 of
part aborted subsystem SMTP Relay.

Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4826] Can't abort process 5851 of
part aborted subsystem SMTP Relay.

Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Omscan Server (Omscan Server ) Fri Feb 8 20:11:28 2008
[OM 11170] Could not obtain exclusive access to a queue.
Pid of logging process: 7606
Current errno value: 42

[mikethebike]

do you have any other apps on that server that uses port 25?

[mjkruzeliii]

When I type in

code:
lsof -i:25

I get sendmail and omsmtpd as running processes. I don't think that I have any other processes competing for port 25. If I disable sendmail Scalix doesn't relay any longer.

[les]

ERROR SMTP Relay (SMTPD Relay Pr) Thu Feb 7 08:57:19 2008
[OM.DMON 2171] The SMTP Relay could not listen on port 25.
Pid of logging process: 5853


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4830] Error in starting up a subsystem.
Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4826] Can't abort process 5846 of
part aborted subsystem SMTP Relay.

Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4826] Can't abort process 5848 of
part aborted subsystem SMTP Relay.

Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Administration(omon ) Thu Feb 7 09:01:15 2008
[OM 4826] Can't abort process 5851 of
part aborted subsystem SMTP Relay.

Pid of logging process: 6042
Current errno value: 3


SERIOUS ERROR Omscan Server (Omscan Server ) Fri Feb 8 20:11:28 2008
[OM 11170] Could not obtain exclusive access to a queue.
Pid of logging process: 7606
Current errno value: 42

The above errors indicate that when scalix started up that the smtp port 25 was in use by another process.

Were these errors logged before or after you restarted the server?

it's difficult to tell what that might have been, could have been sendmail or another app starting on port 25 or even scalix with the previous smtp server in a bad state.

Sometimes after a hard reboot you can get issues where the state of a service doesn't close properly and you need to use omreset to force it to off before you can start it.

What you need to do is check the logs when its broken and look for those errors.

p.s.

do you have spam and/or antivirus integration via sendmail or the service router?

are you using apparmor or anything else like selinux?


If they indicate a problem with a process id, as above, use ps -ef to find out what process that is. That will give some more clues as to what is going on.

[mjkruzeliii]

I think sendmail is crashing. I am not really sure how to test this but this morning I came into work and there were 200 messages queued again that had not been sent out.

> omstat -s
shows no errors. Internet Mail Gateway had a 'Started' status.

then
> lsof -i:25
took a really long time to run >5 minutes and listed off a lot of sendmail connections, many of which were under a CLOSE_WAIT status. See Below for a part of the command output.

I did a restart on sendmail and that did nothing. All the same connections still existed. I stopped sendmail and then killed all remaining sendmail processes. I started sendmail and then >lsof -i:25 came up normally.

I checked >omstat -s to see if my queued messages were being sent ... and observed that after the sendmail 'restart' Internet Mail Gateway Subsystem was in an ABORTED state. So I ran >omon "Internet Mail Gateway" and the status returned to STARTED and my queued messages started sending out again.


something I didn't think to do yesterday is to clear out the logs before I left. I will do that for tonight and when it crashes tomorrow it might provide a more clear idea of what is happening. Additionally, I don't have spam or antivirus integration (unless there is some defualt installation of spam/antivirus that I didn't catch with Scalix or OpenSuse 10.2) on my Scalix Server. The server sits on its own DMZ that is firewalled and removes spam and viruses through a proxy before any traffic gets to Scalix.


>lsof -i:25
sendmail 12216 root 11u IPv6 949367 TCP localhost.localdomain:43949->68.166.2 51.66:smtp (SYN_SENT)
sendmail 13815 root 11u IPv4 946405 TCP localhost.localdomain:37324->68.166.2 51.66:smtp (SYN_SENT)
sendmail 15325 root 11u IPv6 950639 TCP localhost.localdomain:35502->8.15.231 .104:smtp (SYN_SENT)
sendmail 15603 root 11u IPv6 952042 TCP localhost.localdomain:crestron-cip->1 32.155.58.8:smtp (SYN_SENT)
sendmail 17294 root 11u IPv6 949534 TCP localhost.localdomain:43973->68.166.2 51.66:smtp (SYN_SENT)
sendmail 18326 root 11u IPv6 951394 TCP localhost.localdomain:34842->66.227.0 .10:smtp (ESTABLISHED)
sendmail 18326 root 12u IPv6 951394 TCP localhost.localdomain:34842->66.227.0 .10:smtp (ESTABLISHED)
sendmail 18566 root 11u IPv6 942266 TCP localhost.localdomain:35396->195.55.1 27.105:smtp (CLOSE_WAIT)
sendmail 18566 root 12u IPv6 942266 TCP localhost.localdomain:35396->195.55.1 27.105:smtp (CLOSE_WAIT)
sendmail 18566 root 13u IPv6 942354 TCP localhost.localdomain:37966->212.51.6 2.10:smtp (CLOSE_WAIT)
sendmail 18566 root 16u IPv6 942354 TCP localhost.localdomain:37966->212.51.6 2.10:smtp (CLOSE_WAIT)
sendmail 18566 root 17u IPv4 948587 TCP localhost.localdomain:46449->195.55.127.103:smtp


Again, Thanks so much for taking the time to help me through this problem!

[mikethebike]

How are you starting sendmail? You must not be "listening" for connections with sendmail (so not using the "-bd" option in the sendmail startup).

Check out the sendmail startup in the init.d on you server and remove those options, leave it to the smtpd to listen for connections

[mjkruzeliii]

Thanks mikethebike!

I looked at the /etc/init.d/sendmail file and there was -bd option in one of the arguements ... see below.

I have removed the -bd from the first if statement. Does anything else need to be changed? Does this statement in the third if check cause a problem because of the LISTEN?
"$SMTPD_LISTEN_REMOTE" != "yes"


******************************************************************************
test -s /etc/sysconfig/mail && \
. /etc/sysconfig/mail

test -s /etc/sysconfig/sendmail && \
. /etc/sysconfig/sendmail

if test -z "$SENDMAIL_ARGS" ; then
SENDMAIL_ARGS="-L sendmail -Am -bd -q30m -om"
fi
if test -z "$SENDMAIL_CLIENT_ARGS" ; then
SENDMAIL_CLIENT_ARGS="-L sendmail-client -Ac -qp30m"
fi
if test "$SMTPD_LISTEN_REMOTE" != "yes" ; then
SENDMAIL_ARGS="-O DaemonPortOptions=Addr=127.0.0.1 $SENDMAIL_ARGS"
fi
msppid=/var/spool/clientmqueue/sm-client.pid
srvpid=/var/run/sendmail.pid

[mikethebike]

It shouldbe fine, it will be using the new arguments you have provided.

Mick

[mjkruzeliii]

Scalix is still having some problems. It stopped sending emails out again. There are nothing in the logs and omstat -s showsthat all services are running ok.

I haven't rebooted since I changed the sendmail config file suggested by Mick.

After I rebooted all messaged queued sent out again ... but I am not entirely sure that the problem has been resolved. Any other thoughts?


Thanks!!

[kmcelwain]

I didn't see the answer to the spam question, we were using mailwasher and it was simply getting really mad and fighting with the SMTP Relay.

We had to get rid of Mailwasher and use an untangle box, transparent, and things have never been better.

[mjkruzeliii]

I have not loaded any antivirus onto the scalix server. All spamand viruses get filtered out by a firewall prior to getting to the scalix server.

That being said I am not sure if there is a default installation of antivirus software with OpenSuse 10.2 or with Scalix that I might have missed. I checked all servicesand didn't see anything that would suggest that antivirus software was installed or running. Anything/Anywhere I should check?

The problem is somewhere with sendmail, or a conflict causing sendmail to crash. If I reinstall sendmail or try to find an updated version would it cause scalix a problem to reinstall sendmail?

[kmcelwain]

hmmm...opensuse. I wonder if it installed spamassassin by default. You might want to make sure that it didn't do that. If it did install AV it would most likely be clamav

If you look in the /etc folder you would most likely see a config file for one of those if they exist.

Another possibility...DNS. How is the server setup to query DNS. Are you using in house or your ISP?

I actually started using OpenDNS and I'll never go back to anything else. I'm not saying it's perfect, but it's been pretty solid.