TB/TB-2007-05-WEBCONF

TB -> TB-2007-05-WEBCONF

Scalix 11 Web Access Configuration and Tuning for large deployments

When Scalix 11 is installed the web applications are automatically configured to make optimal use of small and medium sized machines. Accordingly the default configuration is suitable for most small and medium sized installations (up to 500 users). For larger deployments Scalix recommends that the configuration and tuning of the tomcat server hosting the web applications is reviewed and adjusted as necessary. Guidance for configuring the limits on open file handles, java virtual machine memory and tomcat thread pool size is given below:

Open file handles limit

Linux limits the number of file handles a process can keep open. Scalix Web Access uses between two and four file handles for each logged in user. The startup script (/etc/init.d/scalix-tomcat) overrides the default operating system file handle limit (which is typically 1000). The limit applied to tomcat is fetched from the file /etc/opt/scalix-tomcat/scalix-tomcat.conf. Its default value is 10000 which is sufficient for 2000 concurrently logged in users.

Note that Scalix versions prior to 11.03 did not override the default operating system limit and therefore will have a 1000 file descriptor limit enforced unless manually configured. Note also that Scalix 11 versions before 11.04 had a file descriptor leak in the search server which can cause the file descriptor limit to be reached even with relatively few users, after the server had been running for a long time. Exceeding the file descriptor limit leads to a range of symptoms including new users unable to log in. Errors in the web application log files relating to failed attempts to open files are typically seen.

Java virtual machine memory limit

The Java virtual machine in which Tomcat runs is configured with a memory usage limit. The default limit is one quarter of the system's physical memory with a lower bound of 64 Megabytes and an upper bound of 1 Gigabyte. The limit is set at install time so installing additional physical memory later will not change the configured size. The limit is configurable in the Scalix installer (under Scalix Tomcat\Tomcat Configuration\Reserve memory for Tomcat:) The Scalix web applications use some 'base' memory regardless of how many users are logged in : roughly 50 Megabytes. In addition each logged in user uses some additional memory: approximately 5 Megabytes per user. For systems that need to support more than 200 concurrently logged in users the limit should be increased. The maximum size permitted on a 32-bit system is 2 Gigabytes. 64-bit systems allow more memory to be allocated to tomcat, provided sufficient physical memory is present to avoid paging.

Tomcat thread pool size

Like many modern application servers, Apache Tomcat uses a thread pool to avoid thread creation overhead. Each client request is serviced by a thread from the pool. Once a request has been completed the thread returns to the pool to be re-used on a subsequent request. Scalix Web Access uses up to one thread per logged in user. Tomcat's thread pool has a configurable maximum size. The default size is 200. In Scalix 11.04 and later the default size is overridden in the file /var/opt/scalix/<inst>/tomcat/conf/server.xml The configured maximum size is 1000 threads. This is the largest number of threads that may be safely allowed on a 32-bit system. A larger number risks exhausting virtual memory address space because each thread is allocated 2Mbytes of stack space. (1000 x 2 = slightly less than 2G, which is the virtual address space available to user processes on 32-bit Linux systems). On a 64-bit machine Scalix uses the 64-bit Java virtual machine. 64-bit addressing allows many more threads because the available virtual address space is huge. However Tomcat is configured identically to the 32-bit case. Therefore servers that need to support more than 500 concurrently logged in users per server should use 64-bit hardware and manual configuration to allow a larger thread pool.

Symptoms of a 'starved' thread pool include inability for users to log in and users observing operations 'timing out'.

Manual configuration requires the line containing the 'maxThreads' directive in server.xml to be changed:

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009"
               address="server.doma.ain"
               maxThreads="500" minSpareThreads="50" maxSpareThreads="125"
               enableLookups="false" redirectPort="8443" protocol="AJP/1.3"
               URIEncoding="UTF-8" />