Very slow SXMBOXIMP !!!!

[ivo_toshev]

Dear All,

In Scalix 11.0.1:
I exported a user mailbox with sxmboxexp. Export file is 1,2GB.
The export procedure took around 15 minutes. BUT :
Then i deleted the user, then recreated the account and started sxmboximp. So far ~ 2,5 HOURS !!! And it is imported just around 40 % till now .... Will see if it took 6 HOURS at all.. But this is very,very,very slow. Why ?
I did not see anything wrong with omshowlog...

[ivo_toshev]

It took 3 HOURS !

[lleahu]

I have experienced this too!

Quad CPU - Intel Xeon 3.2GHz
2GB memory

HP ML350
SLES10 + Scalix 11.0.1 + XFS filesystem w/ LVM

We did a same-server migration from SLOX4 to Scalix.

Export of 20GB of data (using IMAP protocol) - 4 hours.
Import of 16GB of data (using IMAP protocol) - 16 hours.

Why is importing so slow?
This is even with the SIS indexing feature turned off.

please advise!

[Richard Hall]

There is a lot of disk write activity with the import that makes it slower than the export.
Are you setting the --nosync option? This will speed it up a bit.
- Richard

[ivo_toshev]

No i didnt.
I read about it, but did not know how usable the mailbox will be before background sync to take place.

[lleahu]

There is a lot of disk write activity with the import that makes it slower than the export.
Are you setting the --nosync option? This will speed it up a bit.
- Richard

I monitored the whole system, and the only processes which were active, after disabling SIS Indexing, was in.imapd.

There were two processes of in.imapd.
The first process was the one writing to the disk and stayed mostly in a disk-wait state, never going above 20% cpu utilization.
The second process just slept most of the time, never really going above 5% cpu utilization.

There were no other processes active on the server.

Where does the '--nosync' option get set at? If you are speaking of the 'sync' option that is passed when mounting the file systems, then no, we do have have that option set.

[ivo_toshev]

No, he talks about --nosync option passed to the sxmboximp command.

See man sxmboximp

[florian]

You can try the -nosync without any problem; the only thing is that IF the system crashes while the import is running, the mailbox will most like be corrupt and will need to be recreated. This is not really a problem in the situation given.

Also, imap is not used during the import, so load on IMAP processes has nothing to do with a command-line import either.

Cheers,
Florian.

[lleahu]

No, he talks about --nosync option passed to the sxmboximp command.

See man sxmboximp

You can try the -nosync without any problem; the only thing is that IF the system crashes while the import is running, the mailbox will most like be corrupt and will need to be recreated. This is not really a problem in the situation given.

Also, imap is not used during the import, so load on IMAP processes has nothing to do with a command-line import either.

This still does not explain the problem sufficiently to me.

I imported email using IMAP4 protocol, across the network. I did not use the sxmboximp tool.

When I extracted my email from the old system (same exact haredware as the new system), it tool 4 hours to do.

When I imported this into Scalix, it tool 16 hours to do!!

I need you to explain to me why this tool so long.
As far as I'm concerned this is a bug in Scalix 11 and needs to be fixed!

I can imagine (and I anticipated, because Scalix 11 uses databases to store mail, and they are proprietary) that the import process would take no more than twice as long as the export process (x * 2).

However this was like four times as along (x * 4) and that is completely unacceptable performance.

Does the IMAP4 daemon have any options to speed this up ?

[florian]

Hi lleahu,

first of all, please do try to keep forum threads separate for different things if possible. This thread initially talked about sxmboxexp/sxmboximp performance and now we changed to IMAP performance. These are two completely unrelated subjects and should not be intermixed.

second, I don't really take your point about factors between export/import operations. I just tested some raw disk performance on one of our servers, using the internal disks, which were not really fast. In my testing, raw disk performance was about 2x faster for read access than for write access when using asynchronous writes.

For Scalix message store access, we do use sync writes to ensure maximum data integrity in case of a system crash. This is a design decision which I believe is hghly valid for an enterprise system.

We do provide the option to turn this off for sxmboximp as it can be controlled on a per-process basis. We do not provide the option to control this per IMAP session as IMAP access is more commonly connected to user access where this is much more critical. The user should not be able to control this. However, we do provide a general.cfg setting that can turn off sync writes for the whole server - this could be used temporarily in migration situations, our support can tell you how.

In addition, write operations for IMAP are a lot more complex than read operations. We need to breakdown the message into our internal message store format that allows to store advanced message properties needed for our clients-of-choice-architecture, including MAPI clients like Outlook. This requires some additional processing time during the load. Finally, with Scalix 11 some further activities are triggered by loading messages into the system, e.g. indexing of the message. If this is run within that same pipeline, it will create further load on the system.

All in all, the numbers you're reporting here seem perfectly reasonable to me. Of course, it might be necessary to analyze your hardware sizing and the setup of your disk subsystem in particular as Scalix is seen as a very IO-intensive application, similar to a busy relational database system. Our support can help you with such analysis.

Note, btw., that Scalix 11 does NOT use databases to store message data; as before, the actual data is kept in the filesystem for best performance, scaleability, ease of backup and manageability. The Postgres database was introduced in Scalix 11 as an additional header cache to support the Messaging Services web services layer and as a specific header cache to speed up certain client access operations, such as loading the headers in a large mailbox environment with SWA, in particular with distributed servers.

Hope this helps,
Florian.

[lleahu]

For Scalix message store access, we do use sync writes to ensure maximum data integrity in case of a system crash. This is a design decision which I believe is hghly valid for an enterprise system.

I agree with you, but I am trying to perform a migration.

We do provide the option to turn this off for sxmboximp as it can be controlled on a per-process basis. We do not provide the option to control this per IMAP session as IMAP access is more commonly connected to user access where this is much more critical. The user should not be able to control this. However, we do provide a general.cfg setting that can turn off sync writes for the whole server - this could be used temporarily in migration situations, our support can tell you how.

Please show me how to turn of sync writes for all services.

In addition, write operations for IMAP are a lot more complex than read operations. We need to breakdown the message into our internal message store format that allows to store advanced message properties needed for our clients-of-choice-architecture, including MAPI clients like Outlook. This requires some additional processing time during the load. Finally, with Scalix 11 some further activities are triggered by loading messages into the system, e.g. indexing of the message. If this is run within that same pipeline, it will create further load on the system.

Yes, I did notice that SIS Indexing was slowing things down.
Yes, I did turn all off all SIS Indexing after I determined this.
Yes, I did turn of the indexing about 1/4 of the way through the migration after noticing this, and yes, it was still slow.

[lleahu]

We do provide the option to turn this off for sxmboximp as it can be controlled on a per-process basis. We do not provide the option to control this per IMAP session as IMAP access is more commonly connected to user access where this is much more critical. The user should not be able to control this. However, we do provide a general.cfg setting that can turn off sync writes for the whole server - this could be used temporarily in migration situations, our support can tell you how.

Please show me how to turn of sync writes for all services.

Is there any answer to my question?
I have looked in the Administrator and Configuration manuals, and I have emailed support@scalix.com, but I have receive no answer!

Please reply, my client is getting very angry!