SWA slow since update 11.4.1

[pdtec]

Till I updated my Scalix Server to 11.4.1 the SWA is loading very slow. The first time SWA loads a folder it takes about 1 Minute to open it. It doesn't matter if there is none, one ore 1000 Mails in it. After I opened a folder the first time, it is fast again and I can also switch fast between opened folders. The server load is low - so I have no idea what is broken since update to 11.4.1.
Before I had 11.4

[Donsell]

Till I updated my Scalix Server to 11.4.1 the SWA is loading very slow. The first time SWA loads a folder it takes about 1 Minute to open it. It doesn't matter if there is none, one ore 1000 Mails in it. After I opened a folder the first time, it is fast again and I can also switch fast between opened folders. The server load is low - so I have no idea what is broken since update to 11.4.1.
Before I had 11.4

Sorry, I'm not sure I follow. Are you saying that things were slow on 11.4 and then are fine since the upgrade to 11.4.1. Or are you saying they were faster before the upgrade to 11.4.1.?

[kurtk]

SWA is currently running very slow for me too. This morning I tried logging in and I gave up after waiting 15 minutes for my mailbox to open. I only have 4 users. I'm running AMD Dual Core Opteron w/ 4gb of memory. CPU was not going over 40% and it was only using 1gb of 4 gb of memory. I have /var mounted to a linux (software) raid 1 on 2 SATA drives. Everything else is mounted to a 4 hard disk linux software raid 5. I restarted the server and now it takes a little over a minute to open a folder up or log in. Is over one minute normal? Can you lead me to some logs or anything to try and pinpoint this problem? I recently updated to 11.4.1 but it seemed slow before the update too. I currently have everyone using Outlook. We are having lots of problems with Outlook that are not related to Scalix and I'd like to move everyone to SWA but it's just too slow to be practical right now. Please help.

Thanks,

Kurt

[Donsell]

I've been battleing the same SWA slowness on 11.4.1 today. Watching server performance load seems fine, but there are a lot of processes in an i/o wait state. If you're running Linux, could you run top and see what you're %wa is? Mine is swinging wildly beteen 0 and 90%. This usually indicates disk intensive I/O. Scalix is all that's running on the server. I don't see an obviously process running besides java that would be doing all the disk I/O

[florian]

after some of our upgrades, a number of cache data structures are cleared because they have changed internally. The first login after an upgrade then rebuilds those caches, so it can take a little bit longer, which should be considered normal. typically things stabilize after the first one or two logins per user.

Scalix 11.4.0 and 11.4.1 should not have much difference in terms of SWA performance. Salix 11.4 should in general be faster than 11.3 and before for SWA access. Scalix 11.5 [no, no dates yet...] will be faster again as we're working on performance improvements in various areas.

Florian.

[jimtombari]

I also upgraded from 11.3 - 11.4.1 and I am experiencing the same problems. Once you wait 30 sec. to a min. you can view a folder and everything seems fine. It has been 5 days with many logins and the problem has not improved. Tech support has attempted java and tomcat setting adjustments and the SWA is still pretty much useless. SAC and Outlook work fine.

[florian]

if the delays continue that's usually a sign that the so-called 'platform' header cache is not being used and SWA is falling back to use IMAP. This should be visible in the tomcat logfiles, maybe some config file is messed up.

Tech Support should be able to investigate and understand this. I can say with certainty that my own SWA in 11.4.x is faster than the previous 11.3 setup.

Florian.

[kurtk]

Thank you florian for pointing me to which logs to look at. I really love the functionality of Scalix and I know with a little guidance I can get it to run as fast as many of the other users on this forum are running. I'm still pretty new with both Linux and Scalix so I hope you can help me interpret this log. I don't really feel the update had much to do with the speed issue. SWA seems like it's been gradually slowing down for months. I have a feeling I have a setting somewhere that's causing it. I've combed through this forum for hours looking for things to help. I've cleared the IMAP cache and made sure java has plenty of memory as mentioned in other posts but nothing seems to help. I've checked the firewall to make sure all the ports that need to be open are (I've even tried disabling the firewall to make sure there was nothing that I was missing but it didn't help)

These are the logs starting from the last time I restarted the server. I notice it says several times java.net SocketException: broken pipe. I don't know what that means.

Thanks again and I hope you can help.

Could not write SOAP response: org.apache.catalina.connector.ClientAbortException: null
at com.oddpost.soap.SoapResponseImpl.flushAndHandleException(SoapResponseImpl.java:371)
at com.oddpost.soap.SoapResponseImpl.send(SoapResponseImpl.java:350)
at com.oddpost.server.HttpRequestHandler.sendResponse(HttpRequestHandler.java:292)
at com.oddpost.server.HttpRequestHandler.handleRequest(HttpRequestHandler.java:263)
at com.oddpost.server.SoapServlet.doPost(SoapServlet.java:44)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
at com.oddpost.server.filter.HttpConfFilter.doFilter(HttpConfFilter.java:181)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
... skipping the last 18 lines as irrelevant...
Caused by: ClientAbortException: java.net.SocketException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:366)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:432)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:314)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:293)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:98)
at com.oddpost.soap.SoapResponseImpl.flush(SoapResponseImpl.java:312)
at com.oddpost.soap.SoapResponseImpl.flushAndHandleException(SoapResponseImpl.java:359)
... 31 more
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(Unknown Source)
at java.net.SocketOutputStream.write(Unknown Source)
at org.apache.jk.common.ChannelSocket.send(ChannelSocket.java:537)
at org.apache.jk.common.JkInputStream.doWrite(JkInputStream.java:162)
at org.apache.coyote.Response.doWrite(Response.java:560)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:361)
... 37 more

[florian]

if these errors occur over and over again, i would surely say that it's related. it's hard to tell, it may be postgres not up and running, etc., etc. - the whole debugging excercise for this is probably a little bit to big to run over this forum. what i would do is to shutdown tomcat, clear out the logs, then restart, go on with a single session and really see the sequence of events. obviously this may be pointing at some apache config going sideways or any of the tomcat properties file....

hm.

Florian.

[kurtk]

It seems like my swa installation has deteriorated even more. After a fresh restart it takes about a minute to log in. After 2 days or so the server needs to be restarted to even be able to log in. I get an error "unable to fetch calander events." when trying to log in and the spash screen keeps scrolling across until I close the window. I'm pretty sure I got this error before the last restart too. postgres is running. I cleared out my logs and the logs that are in there now look very similar to the ones I posted earlier although they were not in there after the very first login to swa. I looked through the apache settings and nothing looks out of order, but like I said I'm relatively new to both Linux and Scalix. Any other ideas?

Thanks,

Kurt

[Donsell]

I think our 11.4.1 installation is running better tonight. We spent most of yesterday going over everything and make sure our spam defense is stong. Hopefully when the server gets under load tomorrow it'll continue to perform well.

Do you know what resource you're running out of when you're forced to restart your server? Check file handles and inodes, make sure your swap space is 2x memory. Make sure your DNS is functioning like it should.

[jimtombari]

I had been having the same speed problems with swa loading folders after upgrading to 11.4.1 from 11.3. It turned out to be the following 2 configs had an old Ip address in them. I changed the ip to 172.17.7.253 about a year ago but it did not pose a problem until the upgrade.

Updated the Postgres Client Authentication
modified the file /var/opt/scalix/NN/postgres/data/pg_hba.conf.

host scalix scalix 172.17.7.253/32 md5

Updated the Search and Index Service
modified the file /var/opt/scalix/NN/sis/sis.properties.

index.client.whitelist=172.17.7.253,127.0.0.1

search.client.whitelist=172.17.7.253,127.0.0.1

As soon as I made these changes and rebooted SWA speed was great.

I hope that if anyone is as dumb as me and did not update these files that this helps them.

[pdtecag]

Hi folks,
jimtombari description did the trick. During the update I set the IP addresses for indexing and postgres to 127.0.0.1 which apparently caused the problem.

Is there any documentation about how to secure a stand alone server where it shouldn't be necessary to have all the services listen to the public IP?



Have fun

Marc

[kurtk]

Thanks jimtombari. I'm sure I'm "dumber" than a lot of people on this forum. That's why I like this forum so much. It's filled with smart people that help me . I will definately try your tip this evening. I've not changed my ip but it may have been somthing I missed during setup.

Thanks,

kurt

[kurtk]

Donsell, I am using very little of the available swap and only 1/4 of my physical memory. CPU is running 50% and below. I will check the file handles and inodes and post my findings. I think jimtombari is onto something. A lot of the posts on this forum about poor SWA speed has been due to Postgres not running. I know the service is running, but I'm not certain it is running correctly . Thanks to everyone who is helping.

Kurt