fetchmail big emails causes timeout

[cli]

Hi,

We have a lot of users on IMAP/POP3 boxes. We want to keep these mailboxes until we have migrated all users to Scalix. We use fetchmail to move emails from the IMAP mailboxes to Scalix.

The problem is related to big emails. For some reason big emails (size is in megabytes) are always timed out by fetchmail. It seems like the forwarding/moving of big emails is extremely slow?

I have increased the fetchmail timeouts to 1800 seconds, and this was not enough for an email that was aprox. 9 MB. The I increased it to 3600 seconds and it ended also in a timeout.

Here's the output from fetchmail:
fetchmail: reading message jdoe34567@pop.service.com:1 of 18 (2417 header octets) fetchmail: (9655035 body octets) fetchmail: timeout after 1800 seconds.
fetchmail: socket error while fetching from pop.service.com
fetchmail: Query status=2 (SOCKET)

fetchmail: reading message jdoe34567@pop.service.com:1 of 21 (2417 header octets) fetchmail: (9655035 body octets) fetchmail: timeout after 3600 seconds.
fetchmail: socket error while fetching from pop.service.com
fetchmail: Query status=2 (SOCKET)

Here's an example of the fetchmail rc file:

set daemon 500
set bouncemail
set logfile /var/log/fetchmail.log
set postmaster "jdoe@mydomain.com"
poll pop.service.com protocol imap timeout 60 username "jdoe34567" password "a374397"
is "john.doe@mydomain.com" smtphost "mail.mydomain.internal" fetchall limit 0 fetchsizelimit 0

I could increase the timeout to infinity and beyond, but then one user could block moving/forwarding of emails for other users (in case of huge emails). One idea I had was to have two separate fetchmail processes, one that does not timeout for hours and another one that has a normal timeout of say 1-5 minutes. But I havent't yet investigated how fetchmail or the IMAP mailboxes behave in a setup like this...

Any help appreciated!

Thanks.

- CLi

[dkelly]

So this is timing out on retrieving the data from your other (non-Scalix) server ?

What do the logs say on that other server.

This doesn't look like a Scalix issue so I'm not sure how we can assist in this case.

Cheers

Dave

[cli]

Unfortunately we have no access to the service providers IMAP/POP3 email server. They use postfix, that much I could interpret from the email headers.

The strange thing is that when we use an IMAP/POP in Outlook we are able to download big emails within reasonable time limits (in minutes, not hours).

I have contacted the service provider, but they are slow with smaller customers as is the trend nowadays.

Sendmail logs have entries like this (which I believe are related to the problem):

Oct 12 16:35:01 mail1 sendmail[2220]: k9CDYe2A002220: to=<jane.smith@mydomain.com>, delay=00:00:21, xdelay=00:00:00, mailer=relay, pri=3071479, relay=[12
7.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (k9CDZ1lQ002221 Message accepted for delivery)
Oct 12 16:35:02 mail1 sendmail[2225]: k9CDZ1lQ002221: to=<jane.smith@mydomain.com>, delay=00:00:01, xdelay=00:00:01, mailer=scalix, pri=3161691, relay=ma
il1, dsn=2.0.0, stat=Sent (Ok)
Oct 12 16:35:02 mail1 sendmail[2229]: k9CDZ21D002229: from=<vmail@mbox.service.provider.com>, size=2760, class=-60, nrcpts=1, msgid=<20061012103801.81B7F18B2FD@smtp1.song
.fi>, proto=ESMTP, relay=root@localhost
Oct 12 16:35:02 mail1 sendmail[2230]: k9CDZ2Ah002230: from=<vmail@mbox.service.provider.com>, size=2965, class=-60, nrcpts=1, msgid=<20061012103801.81B7F18B2FD@smtp1.song
.fi>, proto=ESMTP, daemon=MTA, relay=localhost.localdomain [127.0.0.1]
Oct 12 16:35:02 mail1 sendmail[2229]: k9CDZ21D002229: to=<jane.smith@mydomain.com>, delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=140760, relay=[127
.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (k9CDZ2Ah002230 Message accepted for delivery)
Oct 12 16:35:02 mail1 sendmail[2234]: k9CDZ2Ah002230: to=<jane.smith@mydomain.com>, delay=00:00:00, xdelay=00:00:00, mailer=scalix, pri=230965, relay=mai
l1, dsn=2.0.0, stat=Sent (Ok)
Oct 12 16:36:15 mail1 sendmail[2238]: k9CDZ2eC002238: collect: premature EOM: unexpected close
Oct 12 16:36:15 mail1 sendmail[2238]: k9CDZ2eC002238: collect: unexpected close on connection from localhost, sender=<cheryl.crow@mydomain.com>
Oct 12 16:36:15 mail1 sendmail[2238]: k9CDZ2eC002238: from=<cheryl.crow@mydomain.com>, size=1334149, class=0, nrcpts=1, proto=ESMTP, relay=root@localhost

Any ideas, how I could trace the problem? I have turned up Audit logging (level 5) on the service router and internet mail gateway.

How about fetchmail settings, any recommendations of reasonable values for moving of emails to Scalxi from around 160 IMAP boxes?

Thanks.

- CLi

[ScalixSupport]

As Dave said, this does not look like a scalix issue. This is a fetchmail issue. You may want to do some googling for fetchmail settings.

Regards,
Don

[cli]

I finally managed to get some more info from fetchmail using "fetchmail -v -v -f file.cfg". The result now looks like this:

fetchmail: IMAP< * 23 FETCH (RFC822.SIZE 468450)
fetchmail: IMAP< * 24 FETCH (RFC822.SIZE 2232)
fetchmail: IMAP< A0005 OK Fetch completed.
fetchmail: IMAP> A0006 FETCH 1 RFC822.HEADER
fetchmail: IMAP< * 1 FETCH (RFC822.HEADER {2313}
fetchmail: reading message imapbox234@pop.service.provider.com:1 of 24 (2313 header octets)
fetchmail: About to rewrite Return-Path: <john.smith@mydomain.nu>
Rewritten version is Return-Path: <john.smith@mydomain.nu>

fetchmail: About to rewrite From: John Smith <john.smith@mydomain.nu>
Rewritten version is From: John Smith <john.smith@mydomain.nu>

fetchmail: About to rewrite To: hubert.longbow@mydomain.com
Rewritten version is To: hubert.longbow@mydomain.com

fetchmail: SMTP< 220 mail.mydomain.internal SMTPD
fetchmail: SMTP> EHLO localhost
fetchmail: SMTP< 250-mail.mydomain.internal Hello mail.mydomain.internal [10.11.83.252], pleased to meet you
fetchmail: SMTP< 250-AUTH LOGIN CRAM-MD5 DIGEST-MD5 GSSAPI PLAIN
fetchmail: SMTP< 250-AUTH=LOGIN
fetchmail: SMTP< 250-DSN
fetchmail: SMTP< 250 8BITMIME
fetchmail: forwarding to mail.mydomain.internal
fetchmail: SMTP> MAIL FROM:<john.smith@mydomain.nu>
fetchmail: SMTP< 250 john.smith@mydomain.nu... Sender ok
fetchmail: SMTP> RCPT TO:<hubert.longbow@mydomain.com>
fetchmail: SMTP< 250 2.1.5 <hubert.longbow@mydomain.com>... Recipient ok
fetchmail: SMTP> DATA
fetchmail: SMTP< 354 Enter mail, end with "." on a line by itself (relay)
fetchmail: IMAP< )
fetchmail: IMAP< A0006 OK Fetch completed.
fetchmail: IMAP> A0007 FETCH 1 BODY.PEEK[TEXT]
fetchmail: IMAP< * 1 FETCH (BODY[TEXT] {9661884}
fetchmail: (9661884 body octets) fetchmail: timeout after 60 seconds.
fetchmail: socket error while fetching from pop.service.provider.com
fetchmail: 6.2.5 querying pop.service.provider.com (protocol IMAP) at Thu 12 Oct 2006 11:13:39 PM EEST: poll completed
fetchmail: Query status=2 (SOCKET)
fetchmail: Deleting fetchids file.

This log snippet looks to me like the email is fetched fine from the service provider but then Scalix SMTPD stalls for some reason? Probably because I have not configured something properly...

I changed the SMTP Relay greeting, that's why it says just "220 mail.mydomain.internal SMTPD".

I assume xport.in is invoked by the Scalix SMTP or Service Router process and it should not be running as a daemon. This is a bit unclear in the documentation, though I found one sentence on page 27 in the Administration guide: "To monitor who is sending messages into Scalix, the xport.in process checks the Linux user ID
of the process invoking it (the xport.in process is invoked for each message)."

Investigation continues...

- CLi

[cli]

Ups, my mistake. Actually the log snippet indicates exactly what Dave and Don indicated, timeout when fetching the email from the service provider IMAP box.

- CLi

[cli]

The closure of this problem, in our case, was that we activated forwarding of emails for each IMAP box to a address that our Scalix server handles. Luckily the ISP provided this service for each IMAP/POP box.

The ISP message was clear: they do not support fetchmail. We had to throw that approach away...

- CLi