SWA Performance Question

[zach]

Why does Scalix Web Access load the entire inbox folder upon logging in? Wouldn't the SWA interface be much more efficient if the email was divided up into pages and messages were viewed in increments of 25-50?

[kurtk]

At first glance that would seem true. For some reason though, my SWA account is twice as fast as my G-Mail account and G-Mail splits it to 50/page. Plus, it's incredibly convienent to have everything at your fingertips. Granted, Google's servers have a much high load than mine but on the flip side, I'm sure they are much, much better servers than the single "budget" server I have.

[Eugene]

Good day Kurt

You say your SWA is fast - how did you manage that?

We have around 160-170 users (most of them using the SWA) on our Scalix 11.4 Server and the SWA is often very slow.

Any performance tips that you would like to share?

Thanks,
Eugene

[kurtk]

Certainly. I don't have nearly as many users as you do, but I did have some severe speed issues until I did what was detailed in the following post. (viewtopic.php?t=11436) Jimtombari is the one who posted the fix to my problem. I don't know what your server setup is or what your definition of slow is, but from what I've read, even a modest server should be pretty fast with how many users you have. I have a modest server that is also used for services other than Scalix for only a handful of users. (I spent $1800 almost 2 years ago and most of that was spent on hard drives that are not used for Scalix) But it is blazing fast when running SWA.

Here's the post again on it's own line so it's easier to copy.

viewtopic.php?t=11436

To add to this post, I'll post a few tips I've found as I was reading through this forum. Some of these may seem like common sense but now's the time to check them if you are having some speed issues.

-Make sure you have enough memory in your server. If you are swapping a lot, you probably need more ram.
-Make sure the hard drive that is mounted to /var is fast. This is where all the Scalix Mailboxes and Temp files are. For a lot of users a high speed SCSI drive or RAID setup is best but I'm just running software mirror on 7200rpm SATA drives.
-Make sure all DNS settings and IP security settings in ALL the setup files are correct (this was my problem)
-Clear out IMAP temp folders if you have some issues after changing any settings or if you have an issue with a single user. This seems to be the Scalix equivalent of the "Bill Gates IT Secret." There are posts that tell you how to do this (and scripts that do it for you) if you search through the forums.
-If you have a lot of SWA users, you may want to off load that service onto a separate server from your main Scalix server.
-Make sure your install is up to date. I noticed you are on 11.4. 11.4.1 U1 is out now and has several speed fixes over 11.4.

I hope I was able to help some. I am fairly new to Scalix. As you can see I only have 9 or 10 posts and almost every one of those I'm asking for help. I am very interested to know if any of this helped. Please post a reply with your results.

Thanks,

Kurt

[kurtk]

a couple more things.

-Check your log files for any errors that can point you in the right direction.
-Make sure your LDAP settings are all correct and your firewall is set up to allow LDAP. I've seen several posts that this was the problem.
-Make sure you have 1gb of ram just for Java and your Java settings are set to use it.
-Adjust your Apache, IMAP and/or Tomcat settings to allow more connections if your server can handle it. Again there are posts here that tell you how.
-Search through these forums to find many more tips.

I don't know much about your setup other than you are running 11.4 and have 170 users. If these tips didn't help, post your setup and some log files. I'd be happy to help more if I can and I'm sure some of the Scalix employees or Scalix Star members that have a lot more experience than me will find this post and help if they have some more information to work with.

[kanderson]

SWA is seen as slow in two different ways.

1) Login takes a long time. I suspect this is what you're seeing, and it points to a problem with postgres. You can verify that by signing into the mobile client. If it works, then postgres is fine, if it doesn't work, then check /var/opt/scalix/??/postgres/pgstartup.log. Generally, this is a permission problem due to a changed IP address. The easiest fix is to use the installer to uninstall, and then reinstall postgres. There's nothing in it that matters.

2) SWA is slow for everything. The number one reason for this is disk speed. You want 15K SCSI disks in a RAID 10. RAID 5 will kill you performance wise, but it seems to be what most people want to choose. It costs less, but what you save in cost, you lose in performance many times over. Mirrored drives will also be faster than RAID 5, with the same advantages, but they just aren't as fast as RAID 10. The more drives you have in a RAID 10 array, the faster it is, so 4 - 72G drives are almost twice as fast as 2 - 146G drives.

2b) The second most common reason for slow SWA is RAM. Plan 512M + 12M per concurrent SWA user at peak load. That's just for Tomcat. Remember that to use more than 2G of RAM, you need a 64 bit processor and OS. I don't care if Intel says otherwise, it's marketing nonsense. A 32 bit processor can only address 2G fo RAM, the rest is used as a SWAP file, fast as it may be...

Kev.

[Eugene]

Hi Kev, Kurt

We originally installed Scalix 11.3 in February and upgraded to 11.4 in July.

Our server has a 3.0GHz Intel Xeon Processor, 1M Cache, EM64T with 800MHz FSB.
Currently it has got access to 3GB RAM.

2x36GB 10K RPM drives in a RAID 1 setup is used for the OS, CentOS 5.
4x146GB 15K RPM drives in a RAID 10 setup is used for the Scalix Message Store.
It's raid controller has got 256MB RAM onboard.

We're using one Gigabit NIC to connect it to the Gigabit backbone switch.

Not a bad machine but with the bundle of SWA users on it it sounds as if we need more...

Going by the 512MB base +12MB per additional user I need around 2.5GB RAM just for Tomcat...

I guess that is the hardware discussed.

Our SWA is currently very slow as in Kev's slowness point number 1.
Its performance used to be ok-ish as in Kev's slowness point number 2.

We've also got the issue where users will have unread items in their Inbox (as indicated by the little number next to Inbox in the folder list) but then these items are not displayed in the message list.
On one of the forum posts a suggested work-around was to restart Postgres frequently (we do it every 20 minutes) via a cronned job.

Do I have any hope in getting decent performance for my SWA users on this hardware?

Thanks to you both for trying to help me out over here....

Eugene

[kanderson]

Can you add RAM?

I'd say you're fine everywhere else. That's a pretty wicked little server, and it should meet your needs for 200ish users quite easily.

You've done everything right, that's for sure. Maybe a bit shy on RAM, but my specs were definately at the higher end.

Can you post /etc/opt/scalix-tomcat/scalix-tomcat.conf?

Also, can you make sure you can sign into the mobile client successfully? It'll be at http://server.domain.com/m on your server.

If you ultimately decide that you should use a different box for Scalix, I'd advise a smaller front end box (maybe 2) for SWA. That would offload SWA from your main server, and should improve performance. With 2 boxes, they can be smaller, and you'd gain redundancy in the solution. It's actually quite easy to set that up.

Kev

[kanderson]

Ok, so I'd like to have some feel for why you have a problem.

Can you run top, and watch the %io wait time near the top of the output? If you hit s and then .5 (decimal 5) it'll update more frequently for you (every half second).

Just get a feel for what the IO load is. That's usually indicitive of an overly busy disk subsystem.

One last question. You said the machine "has access to" 3G of RAM. Is this running on VMware? If so, you'll likely need to give the disk that Scalix is on a higher priority.

Lastly, is this Community, Small Business, or Enterprise Edition?

Kev.

[Eugene]

Hi Kev

The disk wait time is through the roof really... %wa spikes over 90% quite often and that is when no-one can get into their mailboxes. When I ran top just now it was jumping all over the show spending a lot of time upwards of 60%...

It is not a virtual server, but the reason why I said it has access to 3GB ram is because the server has got 4GB installed but it is only "seeing" 3GB.

I've also got more ram available (due to another box being upgraded) but first I will have to find out why its not picking up the 4GB correctly...

I was also able to get into the mobile client without any trouble.

The contents of scalix-tomcat.conf follows below:
=========================================

#
# Configuration file for Scalix Tomcat
#

# The directory where Java is installed:
JAVA_HOME="/usr/java/jre1.5.0_13"

# Additional arguments to be passed to Java:
JAVA_OPTS="-server -Djava.net.preferIPv4Stack=true -Xms1024m -Xmx1024m"

# Maximum number of file descriptors
ULIMIT_NUM_FILES="10000"

========================================

Lastly, we're running the Business edition.

Thanks again!
Eugene

[kanderson]

Ok, that IO Wait means disk speed is where the problem lies.

There are 2 reasons that might be happening.

1) The first possibility is that the disks are simply busy. That's harder to address. Adding disks to a RAID 10 makes it much faster, but it's still a pain to expand (or rebuild) the array, and I don't know if you have space in your server to house additional disks.

2) The second is that you're out of RAM and the server is swapping, that's easier to address if it just means adding ram. Unfortunately, the 3G limit would make me think that perhaps you have a 32bit OS on 64 bit Hardware. I'd tell you here that your best bet to resolve that would be to rebuild the OS with a 64 bit version. And that answer sucks too, I know. Can you post the results of uname -a, and then we'll know for sure. You can change the hostname if you want, that part isn't important...

3) As an option, you could grab yourself a couple of servers. They don't need to be overly high end. Lets say decent P4 Processors, 2G of RAM, Mirrored Disks for redundancy. Build them up running just SWA, and point them back at your primary server when asked. This might be the best option... Certainly the least downtime.

===============================

Having said that...

Before you take any of these labor intensive options, try this:

in /etc/opt/scalix-tomcat/scalix-tomcat.conf change the java opts line to:

JAVA_OPTS="-server -Djava.net.preferIPv4Stack=true -Xss1024K
-XX:-UseConcMarkSweepGC -XX:NewRatio=2 -XX:SurvivorRatio=4 -Xms2048m
-Xmx2048m"

This will better optimize Java, and it'll allocate 2G for Java rather than just 1 as you have now. I don't suspect this will help with Disk IO, but it might see Java perform better. With that many users, Java needs more than 1G, that's for sure.

Then, in /var/opt/scalix/??/s/sys/general.cfg, add the following lines...

IDX_MAXLOAD=2
IDX_MINLOAD=1

This will stop the indexer when the load on your server is high. That means that searching messages might be somewhat behind delivery, but that's OK, compared to having everyone impacted by slow server performance.

You'll need to restart Scalix and Scalix-tomcat to make these changes effective.

Java changes in this post comes with thanks to Spencer, though I never see him in here.

Kev.

[Eugene]

Hi Kev

Thanks again for getting in touch.

I won't be able to add any more disks to that server as its disk bays are fully populated...

I've also just been looking at sysstats and running "sar -W" on the server reports that there is virtually no swapping going on.
Also, we do indeed have a 32bit OS...note the uname output below:

Linux the.servers.fully.qualified.name.com 2.6.18-53.1.13.el5 #1 SMP Tue Feb 12 13:01:45 EST 2008 i686 i686 i386 GNU/Linux

I've also found an utility called iotop which in theory (I haven't been able to try it yet) will allow me to determine which processes are responsible for the IO usage. Only catch is it needs kernel version 2.6.20 or higher and python 2.5 (which in turn needs newer versions of stuff that Scalix depends on). I am also a bit hessitant to upgrade the kernel on a production server half-way through the week.

Option 3 may have to sit on the backburner for just a little while, as we have just ordered some rather impressive servers (2 x Xeon E5450 CPUs, 8GB RAM, 2x36GB 15K HDDs for OS, 4x73GB 15K HDDs for Data).
Then again, given the fact that that the SWA replaces Outlook (and its associated licensing) I may just be able to justify swapping one of those servers with the mail server...do you think it would it be worth it to run Scalix and the SWA on one such a box or should I still try to split them?

I'm going to make the changes outlined in your post after hours (just after 17:00 GMT+12) today and will report back on the outcome.

Just a couple of questions though...

* What do you mean by "searching messages might be somewhat behind delivery"?
* The server currently reports that it is using 2.5GB of the RAM. Will allocating 2GB to Java (which is more than currently available) cause it to have other issues?

Again...Thank you very much for your help!

Eugene

[kurtk]

That sounds like a very nice server. I agree with Kev that the server should be plenty for you especially if you can put some more RAM in it. The RAM problem you are having could be directly related to your speed problems. Did you try running Memtest?

Is SWA slow in off peak hours when barely anyone is accessing it?

Are you using this server for anything other than Scalix?

Is the disk IO going up that high because it is using the Swap more heavily?

Which OS are you running and is it 32bit or 64bit? If it is 32, Tomcat can use no more than 2GB of RAM. If it is 64, it can use as much as you can put in the system. With the current number of users Tomcat needs 2.5 GB as you mentioned. ( see Florian's post below )

viewtopic.php?t=11470&highlight=64bit

I agree that putting SWA on another server, even if it's smaller, would help. Especially if you are currently running a 32 bit OS on your current server, put a 64 bit OS on the SWA server and give it plenty of RAM. This would keep you from having to re-install everything for your current server. That is if it's within your budget.

Kurt

[Eugene]

Hi Kurt

The SWA performs pretty well when there are just a couple of users logged on.

The box isn't used for anything but Scalix.

There appears to be hardly any swapping going on at this stage.

At this stage we're running the 32bit version of CentOS 5.

I haven't yet tried memtest - I may just give that a go.

Thanks for your continued interest & help...

Eugene

[kanderson]

Those servers won't help your issue here. Deploy them where they were intended to go initially. If they're replacing older servers, consider using the old server for SWA.

Your issue here is Disk IO, adding processors won't improve that. RAM might, but you aren't using all the RAM you have now, so lets start by fixing that.

If you could temporarily borrow one of those servers, THAT might be beneficial, build it to run Scalix, move your data to it, rebuild your server with a 64 bit OS, migrate back. That would help you out, for sure... But that's just a day or two of use. Maybe onto the temp server one weekend, back the next, that's the worst case I can think of...

If I was to order SWA servers, personally, I'd save money on the Processors, anything you can find now will be plenty, then I'd give it 4G of RAM, plus I'd mirror some little SCSI 15K disks, 36G or 73 is plenty. But I'd buy 2 of them. Mostly because an outage on 2 servers means half the consequence, and 2 will likely address the load better than a single large server. You can upgrade 1 without impacting people, etc. If it gets slow later down the line, add a third. This can be done without even rebooting your current production server, which is a big benefit. These can be 1U machines, you could even buy them used if you REALLY wanted to save on the budget. (I bought 2 for $215 each last week, and that's after splurging for a hot standby drive). Since they are all working together, and a user can connect to any server without it mattering, there's no need to pay extra for better reliability. Having 3 little ones allows a more complete failure (or scheduled downtime or whatever) than a single server. And you can include your main server in that group, if you'd like.

If it was me, I'd go in this order.

1) Make the config changes.
2) Install a 64 bit OS.
3) Add more RAM up to 6 or 8G.
4) Add a front end SWA server.

Scalix is an IO hog, there's no real way around that. IMAP connections (which SWA is) are the highest load you can put on the server. So you've loaded the server pretty heavily in your case. Adding processors will not help, that's for sure. Adding RAM is a good way to start. In order to add RAM, you need a new OS. (Changing just the Kernel doesn't completely help, though it makes it better).

Leaving more data in RAM will help, so will faster disks. Faster disks (adding more to your RAID 10) isn't possible in this box, so target RAM first.

Q&A looks like this.
1) Scalix uses Postgres to indes your messages for faster searching. By allowing that process to run only during slower periods, it's very likely that you'll have times where a search will not show results from messages that are newish. Newish might mean 30 mins, or it might mean a several hours. Either way, I'd do it. People generally use search to find a really old message. They can remember what messages they received that day.
2) I'm not sure what you mean when you say that it's using 2.5G of RAM. You said you can "see" 3G. I'd be comfortable allocating 2 of that to Tomcat. Though really, this would work MUCH better with a 64 bit OS, because as I mentioned earlier, anything above 2G is just Hocus Pocus on a 32 bit system. It doesn't REALLY work. You need 64 bit.

This thread is becoming a novel... :)

Kev.