URGENT: smtpd - 421 No Resources

[fgarat]

Hi,
Scalix SMTPD answer with "421 No resources" to new connections.

Any ideas?.


thanks.
Facundo.-

[kanderson]

I'd start with:

lsof -i tcp:25 >> /tmp/connections.txt

then

omoff -d0 smtpd && omon smtpd.

Then I'd see if the problem is resolved. If it is, I'd check /var/opt/scalix/??/s/logs/fatal to see what you can turn up. Also check "omshowlog -p 90 -s smtpd -l 13"

I'd also follow my gut and check the results of "df -h"

Then I'd post the /tmp/connections.txt here if problems still persist.

Kev.

[fgarat]

All have been done...no luck...

Problem persist.

I have a very biiig connection load.... I've been doing some tuning in other parts of the system (sendmail, amavis, scalix unix service), everything work great to improve usability.

But, on external SMTP (scalix smtpd) i get 421 "No resources". I'm trying to improve the amoung of connections but i can't find any parameter for this...

is there any way to configure Scalix SMTP to support high load?.

This is breaking my system down.

Thanks to all

[kanderson]

How many users do you have?

Can you please post that connections.txt file I had mentioned? Or PM it to me?

Thanks
Kev.

[fgarat]

send output of lsof -i :25 to PM


12.000+ users.

[kanderson]

Ok, you have a TON of open connections, but with that many users, I suppose they might be legit.

How are the users connecting? MAPI or POP/IMAP. Have you spoken to Scalix Support?

Kev.

[fgarat]

All pop3

SWA with IMAP.


I even increment the amount of smtpd and unix childs with omsetsvc -x

I wil try increment the number of childs for other scalix services.

[fgarat]

any way to configure connections rate and concurrent connections for ip??!

[kanderson]

From what I see, it appears that you're doing SPAM filtering on this server as well as email. I'd move the spam filtering to another server. That should help.

Kev.

[fgarat]

Thanks for the advice.
The server we're using right now has 2 Quad core Xeon, so we don't have cpu load problems.

And the only service that is giving real problems is Scalix SMTPD. (port 25 to the outside world)

[kanderson]

The problem won't be load, it'll be TCP I/O buffers and connections.

With a filter, you take the inbound message on one port. You process it and hand it to another. Then you hand it back up to sendmail on a third. It's quite possible you have LOTS of open ports for do this many DNS dueries and LDAP lookups too, depending on which scanner you're using and how it's configured.

Moving that to another server will drastically cut the load.

You can look in sendmail.cf for things like:

# how many jobs can you process in the queue?
#O MaxQueueRunSize=10000

# maximum number of new connections per second
#O ConnectionRateThrottle=0

# timeouts (many of these)
#O Timeout.initial=5m
#O Timeout.connect=5m
#O Timeout.aconnect=0s
O Timeout.iconnect=30s
#O Timeout.helo=5m
#O Timeout.mail=10m
#O Timeout.rcpt=1h
#O Timeout.datainit=5m
#O Timeout.datablock=1h
#O Timeout.datafinal=1h
#O Timeout.rset=5m
#O Timeout.quit=2m
#O Timeout.misc=2m
#O Timeout.command=1h
O Timeout.ident=0s
#O Timeout.fileopen=60s
#O Timeout.control=2m
O Timeout.queuereturn=5d
#O Timeout.queuereturn.normal=5d
#O Timeout.queuereturn.urgent=2d
#O Timeout.queuereturn.non-urgent=7d

O Timeout.queuewarn=4h
#O Timeout.queuewarn.normal=4h
#O Timeout.queuewarn.urgent=1h
#O Timeout.queuewarn.non-urgent=12h

#O Timeout.hoststatus=30m
#O Timeout.resolver.retrans=5s
#O Timeout.resolver.retrans.first=5s
#O Timeout.resolver.retrans.normal=5s
#O Timeout.resolver.retry=4
#O Timeout.resolver.retry.first=4
#O Timeout.resolver.retry.normal=4
#O Timeout.lhlo=2m
#O Timeout.auth=10m
#O Timeout.starttls=1h


Things like that. But make REALLY sure you have a backup of the sendmail.cf file before you start playing in there....

Kev.

[kanderson]

You could do that with iptables, I guess if you want to change settings directly at the IP level. I wouldn't, but I guess you could...

Kev.

[fgarat]

I understand the tcp buffers and connections problem.

I'm using amavisd with amavis-milter.

To mitigate this problem I configure sendmail with this parameters:
I've this parameters on /etc/mail/access
# Don't delay for internal hosts
GreetPause:localhost 0
GreetPause:127.0.0.1 0
# Do not limit number of connections for localhost
ClientRate:127.0.0.1 0
# Do no limit number of simultaneous connections from localhost
ClientConn:127.0.0.1 0

And the SO Kernel with this:
fs.file-max = 655355
net.core.somaxconn = 3000
kernel.threads-max = 131072
net.core.wmem_max = 1048576
net.core.rmem_max = 1048576
net.core.netdev_max_backlog = 4096
net.ipv4.tcp_max_tw_buckets = 131072
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_fin_timeout=15
net.ipv4.tcp_tw_reuse=1
net.ipv4.tcp_tw_recycle=1



Thanks.

[kanderson]

Like I said, I'd recommend moving Amavis to a diffferent box. It'll make a noticeable different on this one.

Alternately you could talk to Scalix support. They will obviously have more extensive knowledge than anyone else when dealing with user counts this high... I suspect they'd love to discuss it with you, Actually, if you want, I can have a rep call you directly. Just PM me the details.

Kev.

[fgarat]

Hi, finally, last nigth with the only response that I've to buy support and that is something impossible here i decided to change sendmail (supported smtp) for postfix (unsupported smtp)...

Following CompletePostfix article and changing some things everythings is working just fine.

Thanks all for the ansers.

Facundo.-