Heavy IMAP Usage is Killing the Server

[florian]

You will first need to find out what unix user id the user is using, e.g. do

Code: Select all

[root@sting ~]# omshowu -n "florian test"
Authentication ID: Florian.Test@scalix.com
Globally Unique ID: 1d310000e411a844-05.33.71.01
User Name : Florian Test /CN=Florian Test
MailNode : us,adv
Internet Address : "Florian Test" <Florian.Test@scalix.com>
System Login : 60553
Password : set
Admin Capabilities : NO
Mailbox Admin Capabilities : NO
Language : C
Mail Account: Unlocked
Last Signon : 10.10.06 21:05:03
Receipt of mail : ENABLED
Service level : 0
Excluded from Tidying : NO
Recovery Folder visible : NO
User Class : Full
SIS URL : sxidx://sting.scalix.local/0d310000e411a844-05.33.71.01


The line reading System Login: will have your Unix user ID. You can then check under which user ID the processes are running. In the ps -ef output, this is the first column.

Cheers,
Florian.

[paintbuoy]

I ran the synchronisation process on OSX Mail and as it went through I tracked the number of related processes:

Code: Select all

scalix:/ # ps -U 60535
  PID TTY          TIME CMD
15038 ?        00:00:00 in.imap41d
15039 ?        00:00:19 in.imap41d
15365 ?        00:00:00 in.imap41d
18585 ?        00:00:00 in.imap41d
19115 ?        00:00:00 in.imap41d
19280 ?        00:00:00 in.imap41d
19309 ?        00:00:00 in.imap41d
19356 ?        00:00:00 in.imap41d
19357 ?        00:00:00 in.imap41d
19833 ?        00:00:00 in.imap41d
19958 ?        00:00:01 in.imap41d
20343 ?        00:00:00 in.imap41d
20347 ?        00:00:04 in.imap41d
21256 ?        00:00:00 in.imap41d

The number of processes slowly grew to a maximum of 13 and the synchronisation task completed successfully. I had however only run the sync task a few hours earlier so not that many new emails needed to be downloaded to the local mail cache which probably explained why the processes peaked at 13.

I've found this related thread on macosxhints which illustrates the general problem faced by OSX Mail users: http://www.macosxhints.com/article.php?story=20031130150826388

I'm pleased now that I know the cause of the problem, it would be nice if OSX Mail provided an interface to limit connections and/or Scalix could handle more than 16 simultaneous IMAP threads.

If anyone knows a way of achieving either of these things please post the answer, until then I feel better knowing this issue is not the fault of Suse or LVM

[florian]

Well, even though I'm a Apple Mail user myself, I can't speak for what they can do - however, nice that you were able to dig this up.

On the Scalix side, actually that limit should be configurable for any type of session, however it is not for IMAP, which is tracked with us as a bug.

I have raised the severity of the Bug (now: major), please see http://bugzilla.scalix.com/show_bug.cgi?id=12059 for details.

I'll discuss in the appropriate group if we can fix this for Scalix 11......

Stay tuned,
Florian.

[paintbuoy]

Thanks a lot that would be great to see.
I've been struggling to identify technical reasons for updating to Scalix 11 but if this hard coding limit was removed and Mail could safely synchronize I would upgrade in an instant.

[jch]

You might find "ps -ef | grep in.imap41d | sort -k1", or "ps -ef | grep in.imap41d | grep <uid>" is more useful. You'll then find out which user (if any) is hitting the seventeen connection limit.

jch

[paintbuoy]

You might find "ps -ef | grep in.imap41d | sort -k1", or "ps -ef | grep in.imap41d | grep <uid>" is more useful. You'll then find out which user (if any) is hitting the seventeen connection limit.

Yes that's the direct way, I got there the long route by first identifying the UID of the known problem user and then just listing the processes for that partcular UID. Same result but your way is more useful in environments where you don't know who is causing the problem.

[florian]

Hi,

we're still investigating into this.

quick question....

I am running Scalix 10 and a single OSX Mail client using an IMAP connection can take down the server if they choose to synchronise their account after not being online for a long period of time.

What exactly do you mean by "taking down" the server - is this really the server not accepting any more connections under these circumstances or is this effectively a crash of the IMAP server process?

That would be a very different scenarios.....

TIA,
Florian

[paintbuoy]

The IMAP server stops accepting connections from any currently connected client or further connection attempts. The imap server processes continue to run and log error messages about:

Code: Select all

imapSatAuthenticate:Could not register with Session Monitor.

The failure of this authentication process also stops Webmail from working but does not effect SMTP or POP3 services. Unfortunately 100% of users access their mail via IMAP or Webmail.

[florian]

So, to verify again - once this happens for ONE user, other users (with other user names) cannot log in??

-- Florian.

[florian]

I just tested this.

For my testing, the one user could not logon, i.e. create "more" sessions anymore, however, the IMAP server would still respond to other users's logins properly.

Can you confirm that this is the case and only the single user is affected?

Thanks,
Florian.

[paintbuoy]

I've looked through the logs and this is what seems to happen:
1. When an OSX IMAP client performs a synchronize task it creates multiple simultaneous connections to the server.
2. If the number of connections exceeds approximately 17 the IMAP process refuses a connection with the following error:

Code: Select all

ERROR IMAP Server Da(IMAP Server Pr) 10.11.06 14:39:21
[OM 24070] Debug message for Lab use :
imapSatAuthenticate:Could not register with Session Monitor.

3. A short while afterwards further IMAP connections by different clients are refused with the following error:

Code: Select all

WARNING IMAP Server Da(IMAP Server Pr) 10.11.06 14:41:31
[OM.DMON 2205] Failed to connect to Session Monitor: mdc_connect: No such file or directory

This error has one reference in the forums:
http://www.scalix.com/community/viewtopic.php?t=743
This maybe two separate errors in close succession but the timing of the second would suggest it has some relationship to the first.

[jch]

There's something interesting and weird going on here. What you're showing of the log indicates that thes session monitor has shut down, as opposed to crashing because it looks as though it's cleaned its socket up. Can you check ti see if the session monitor has shut down ("omstat -a" should show it's "Started" and "NON-STOP".)

I'm guessing and somewhat in the dark -- is there any chance you can post a complete log from the time that you get the first problem? Starting with the time of the session monitor refusal on the 17th login --- there may be other log messages relating to that immediately before the session monitor one.

If it's too big to post here, send an e-mail through the forum and I can give you a mail address to send the log to.

jch

[paintbuoy]

Which log are you interested in exactly?
From the SAC the messages posted are the only ones shown in the IMAP server log.
The Session Monitor section has no log entries.
Is there a text file on disk that holds more useful information?
If so tell me where to look and I'll put it up here or email it through.

[jch]

Sorry. Server engineers can't spell SAC

You can show all the errors since a particular time by logging into the server and running this on the command line:

Code: Select all

omshowlog -F 10:21

If you want a particular date, then there's a "-f" option as well. So to show all errors since part way through this afternoon I'd do:

Code: Select all

omshowlog -f 13.10.2006 -F 14:30

(Oddly enough, omshowlog doesn't show errors that haven't happened yet ...)

Annoyingly -- for me anyway -- the date format is locale specific so if I was in America I'd type "-f 10.13.2006".

jch

[paintbuoy]

Okay tomorrow is the weekend so I'll purposefully cause the Session Monitor to fail thanks to OSX Mail and track what it says on its way down, stay tuned....