Testing issue with imap

[dvadell]

Hi everybody,
I'm testing scalix using imapsync. I have about 20k mails in a cyrus imap server that I get thru the internet to a scalix server inside a VMWare virtual machine. I saw that listing the folders and the messages in everyfolder (the first step that imapsync does) takes much more time in my local scalix server than in the cyrus imap server.
So got into the scalix server, and ran strace against the imap server, and I saw that it was opening and reading lots of files, where scalix saves the mail messages. So I think scalix has to read every message in order to get the metadata (Subject, From, size, To, etc) that imapsync asks for.
As far as I know, cyrus stores all the metadata in a berkeley database, so when a mail client asks for it, it doesn't have to open and read every single mail in the mail spool. On the downside, if the berkeley database gets corrupted, it may return stale information or no information at all.

My question is: how does scalix handle this metadata?

Thanks in advance,
-- Diego.

[Valerion]

Scalix doesn't consider that metadata, but part of the mail proper, since it is the headers.

It splits a mail into bodyparts (Headers, mail body, each attachment). Each bodypart is stored as a seperate file on disk. To collate all this into a single mail, there is a container file written that has a pointer to the file each bodypart is stored in. Each mail folder is also a container file. To get to a mail you just walk the pointers downwards, which the server can do very fast.

If a mail is received by more than one recipient, a single entry is stored, and pointers are written to the relevant user folders. IIRC there's one stored per 50 recipients, one per mailnode and one per BCC recipient.

The advantage of a file-based approach for me is resilience. I once had a disk develop bad sectors (quite a few, actually), and the client's backups was corrupt as well. I managed to get the mail back, losing only the ones that was actually on the bad sectors.

[dvadell]

Hi,
So it receives the mail, parses it (dividing it into headers, body and attachments and storing pointers to each of them to get fast access later), but it has to make at least one I/O access for each file. That explains why it was getting slow in making the totals, and then fast for any other operation after.

Thanks!

-- Diego.

[a.schild]

Hi,
So it receives the mail, parses it (dividing it into headers, body and attachments and storing pointers to each of them to get fast access later), but it has to make at least one I/O access for each file. That explains why it was getting slow in making the totals, and then fast for any other operation after.

Thanks!

-- Diego.

Hello,

we also see some performance problems in our setup, replated to disk IO.
Is there a recommendation on what FS is best used for the /var/opt/scalix stuff ?

Actually we use reiserfs, but often the calendar can not be loaded in the webfrontend.
After a logout/login it then works.
Monitoring the disk during this initial login, shows that a lot of disk IO is done....


André

[tenaciousC]

if a folder is accessed via IMAP, the first time can be very slow, but the subsequent times, the headers load much faster. I think there is a server side header cache in the the user's folder /var/opt/scalix/??/s/user/g??????/imap-cache.

Access using the Outlook connector seems much faster than imap, so this much use a different technique. This sort of connection can work quite effectively on lower spec servers where lots of IMAP access creates disk meltdown and high iowait times.

Perhaps someone from Scalix can explain the differences?

Cheers

C