TImeouts on Imap Move and Delete (SLOW)

[eightysix]

Hello,

Having issues, with Imap timeouts, when deleting mails using Thunderbird 1.5 and connecting with Imap.

Having a IMAP subfolder with 400 E-mails, select ALL and the E-mails in Thunderbird, and hit the delete key, after a period of time a see a message (Connection to server timed out).

Tried again with selecting just 10 E-mails and the system can move just the 10 e-mails at a time with no problems, I have also comfirmed this is a issue with the copy command when coping large amounts of e-mails.

Is there any settings that I can set to correct this issue? This appears to be very poor imap performance issues.

I have been reading post http://www.scalix.com/community/viewtop ... highlight=
Would some of the fixes in that post towards the temp dirs help with this situation?

[eightysix]

Hi again,

Does Scalix 11 address this issue? With improved IMAP code of some sort?

[dkelly]

The one thing that your thread and also the other one has done is to focus a number of people on the investigation.

There are a good deal of us that use IMAP in our daily work and don't experience the problem as you describe. However, that doesn't mean there isn't a problem which is why we're working with the other poster as part of his support agreement to determine what is going on.

When we have seen performance issues in the past, the first thing we would do is to get details on hardware configuration. Please can you provide us with that ? In particular, your disk configuration. We do know that we read and write a lot of things to/from disk and so a performant disk subsystem is a must. A common theme we see is misconfigured disks. Typically this is disk arrays that are not striped and/or mirrored. This turns an array with multiple I/O channels into a single disk. RAID 5 is definitely not recommended as the write performance in that configuration is too slow.

Similarly, although it's probably not common now, make sure that, if you are using IDE drives, the BIOS has the highest transfer rate selected. We did find that Dell had a habit of shipping their servers with the transfer rate set at a low rate.

In the first page of the post you quoted were some sample scripts. Have you run these to get any comparative figures ?

Cheers

Dave

[dkelly]

Do you have any answers to the questions I asked ?

Cheers

Dave

[neronsp]

Make sure your drives are fast and got enough memory and processing power. In my experience I prefer at least SATA II, 1GB ram & Pentium IV XEON. Works perfectly fast on IMAP stuff.. but it gets grumpy when you're trying to move 50000+ emails from an old IMAP to a new IMAP.


I wonder if there's a tool for migrating old IMAP to scalix

[jch]

Make sure your drives are fast and got enough memory and processing power. In my experience I prefer at least SATA II, 1GB ram & Pentium IV XEON. Works perfectly fast on IMAP stuff.

Actually, that's not entirely relevant. Yes, you need enough memory and enough processing power for the number of logged in users you're trying to support and yes, you need fast enough disks. The number and speed of processors and the amount of memory is relatively easy to estimate (but don't ask me for anything other than a wild guess for a real installation). But to say "SATA II" is misleading.

You have to take a more holistic view of I/O. SATA II will gie you 300Mb/s (it says on the tin, IFRC) but what does that mean? Well, it means that if you have a disk fast enough you can stream data off it at that speed. That's not useful -- with a mail server you're not streaming data off the disk almost all the time: you're actually doing lots of relatively small transactions. So what counts is not the overall bandwidth (300Mb/s) but the number of transactions that the disk subsystem can support. If we assume that you have yourself a 10K rpm disk with an average seek time of 5ms (typical for SCSI, optimistic for SATA) then you're talking about something like 150-200 I/O transactions per second depending where you are on the disk: there are more sectors per track at the outside edge. Some disks quote access time instead of seek time and latency which is less attractive from a marketing perspective because people, myself included, have a tendency to regard "access time" and "seek time" as the same. But I digress.

So you can get, say, 200 I/O transactions per second out of this disk. That's going to seriously limit the number of users you can support. You're nowhere near the total bandwidth of the disk because you're only going to be doing 800K/s but your disk is going as fast as it can. If you look at the output of iostat you'll see the number of I/O transactions as well as the data rate and you'll discover two interesting things.

The first thing you'll discover is that I'm probably being overly pessimistic and you're getting somewhere in the range of 350-500 transactions. (I'm not using transactions in the database sense, you understand: some people write "IOs" but I prefer English!) The second thing you'll discover is that, mostly, the number of reads and writes are very similar even though it's pretty well obvious that a mail server does a lot more reading than writing.

The better I/O rate is down to the fact that data tends to be clustered so that the disk heads are not whizzing backwards and forwards completely randomly -- that's general goodness from the file system. The relatively high write proportion (or low read proportion) is because the kernel caches file contents in memory unless you're really short of memory.

Anyway, it boils down to the fact that what limits the performance of the mail server is the speed at which you can read or write a single 4k block from the disk and that is a function of the mechanical nature of the disk and, short of spinning your disk ever faster and putting in ever bigger magnets to move the disk heads (and using lots more power) you'll not change that.

What you can do, on the other hand, is increase the number of disks. If you stripe your file system across several disks then you'll increase the I/O rate more-or-less linearly. If you have, say, 10 disks then you'll get 3500-5000 transactions per second (which is looking pretty good). If you invest in a fibrechannel controller and a dirty great EMC array then you'll get a lot more out of the system,

Before you give EMC a quarter of a million (if you're lucky) you need to be more holistic: I/O is a whole system operation, not just the disk and the controller. You need to worry about the PCI controllers, how disk I/O interferes with network I/O; whether support for a graphics card has compromised I/O. PCI Express is good but cheap motherboards don't actually give you the performance you'd hope for. When you shell out $1000 on a motherboard for a server you're paying for the ability to do I/O: your $100 motherboard for your home system (even if it supported the number of CPUs the amount of memory) simply won't be able to compete no matter what fancy fibrechannel controller you put in there.

The moral of the story is that if you're shopping for a server, go somewhere that offers servers and knows what they're talking about. Ask around if you're not sure (the amd64-list hosted at redhat.com has some good information if you're new to this). Make sure you have enough disks for the I/O rate you need and make sure you configure them properly -- it's always sickening to see someone configure an expensive array to behave like a single disk,

jch