HowTos/Performance Tuning of Scalix Web Application
Scalix Web application performance tuning
Settings from one of our production server and they can be applied to Debian or Ubuntu
* JRE
[root@web indexes]# java -version java version "1.8.0_65" Java(TM) SE Runtime Environment (build 1.8.0_65-b17) Java HotSpot(TM) 64-Bit Server VM (build 25.65-b01, mixed mode)
* Operation system
[root@web indexes]# cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)
* System packages status
[root@web indexes]# yum update -y Loaded plugins: fastestmirror Repository pgdg94 is listed more than once in the configuration Repository pgdg94-source is listed more than once in the configuration bareos_bareos-14.2 base epel/x86_64/metalink epel extras pgdg94 updates (1/2): epel/x86_64/updateinfo (2/2): epel/x86_64/primary_db Loading mirror speeds from cached hostfile
* base: mirror.dattobackup.com * epel: mirror.steadfast.net * extras: mirrors.usinternet.com * updates: mirror.rackspace.com
No packages marked for update
* Installed Scalix packages
[root@web httpd]# rpm -qa | grep scalix scalix-platform-12.6.0.15502-1.noarch scalix-swa-12.6.0.17713-1.noarch scalix-tomcat-connector-12.6.0-14721.rhel7.noarch scalix-postgres-12.6.0-15963.noarch scalix-tomcat-7.0.70-45.noarch scalix-wireless-12.6.0.15589-1.noarch scalix-sis-12.6.0.16062-1.noarch [root@web httpd]#
* Memory .
[root@web data]# free -h
total used free shared buff/cache available
Mem: 7.4G 1.5G 2.4G 354M 3.5G 5.3G Swap: 3.5G 0B 3.5G [root@web data]#
* httpd.conf changes
KeepAlive off
<IfModule mpm_worker_module>
ServerLimit 16 StartServers 3 ThreadLimit 20000 MaxRequestsPerChild 20096 MinSpareThreads 1175 MaxSpareThreads 1350 ThreadsPerChild 1000 MaxRequestWorkers 1000 MaxConnectionsPerChild 0
</IfModule>
or
<IfModule mpm_worker_module>
ServerLimit 26 StartServers 10 ThreadLimit 200000 MaxRequestsPerChild 200960 MinSpareThreads 11750 MaxSpareThreads 13500 ThreadsPerChild 10000 MaxRequestWorkers 10000 MaxConnectionsPerChild 0
</IfModule>
* HTTPD Worker is enabled Understanding Apache 2 MPM (worker vs prefork)
[root@web conf.modules.d]# pwd /etc/httpd/conf.modules.d [root@web conf.modules.d]# cat ./00-mpm.conf
- Select the MPM module which should be used by uncommenting exactly
- one of the following LoadModule lines:
- prefork MPM: Implements a non-threaded, pre-forking web server
- See: http://httpd.apache.org/docs/2.4/mod/prefork.html
- LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
- worker MPM: Multi-Processing Module implementing a hybrid
- multi-threaded multi-process web server
- See: http://httpd.apache.org/docs/2.4/mod/worker.html
LoadModule mpm_worker_module modules/mod_mpm_worker.so
- event MPM: A variant of the worker MPM with the goal of consuming
- threads only for connections with active processing
- See: http://httpd.apache.org/docs/2.4/mod/event.html
- LoadModule mpm_event_module modules/mod_mpm_event.so
[root@web conf.modules.d]#
* System tuning(max opend files ...)
[root@web limits.d]# pwd /etc/security/limits.d [root@web limits.d]# cat ./20-nproc.conf
- Default limit for number of user's processes to prevent
- accidental fork bombs.
- See rhbz #432903 for reasoning.
- soft nproc 4096
- soft nofile 500000
- hard nofile 500000
root soft nproc unlimited [root@web limits.d]# [root@web sysconfig]# cat /etc/sysctl.conf .... fs.file-max = 5000000000
Since systemd can ignore settings in system (http://serverfault.com/questions/628610/increasing-nproc-for-processes-launched-by-systemd-on-centos-7)
to httpd systemd service need add 'LimitNOFILE=desired value'
[root@web system]# cat /usr/lib/systemd/system/httpd.service [Unit] Description=The Apache HTTP Server .....
[Service] Type=notify EnvironmentFile=/etc/sysconfig/httpd ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND ..... LimitNOFILE=500000
[Install] WantedBy=multi-user.target
* Postgresql tuning (in general memory ~1Gb for postgresql instance). Some settings set to debug and not needed for real production
--- /opt/hg/scalix_server/postgresql.conf_saved.2015-07-13_07:07:44 2015-12-24 09:26:21.406683172 +0200 +++ /opt/hg/scalix_server/postgresql.conf 2015-12-24 09:26:19.446629806 +0200 @@ -61,7 +61,7 @@
# defaults to 'localhost'; use '*' for all
# (change requires restart)
#port = 5432 # (change requires restart)
-max_connections = 100 # (change requires restart) +max_connections = 300 # (change requires restart)
# Note: Increasing max_connections costs ~400 bytes of shared memory per # connection slot, plus lock space (see max_locks_per_transaction). #superuser_reserved_connections = 3 # (change requires restart)
@@ -112,7 +112,7 @@
# - Memory -
-shared_buffers = 128MB # min 128kB +shared_buffers = 500MB # min 128kB
# (change requires restart)
#huge_pages = try # on, off, or try
# (change requires restart)
@@ -123,8 +123,8 @@
# per transaction slot, plus lock space (see max_locks_per_transaction). # It is not advisable to set max_prepared_transactions nonzero unless you # actively intend to use prepared transactions.
-#work_mem = 4MB # min 64kB -#maintenance_work_mem = 64MB # min 1MB +work_mem = 14MB # min 64kB +maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem #max_stack_depth = 2MB # min 100kB dynamic_shared_memory_type = posix # the default is the first option
@@ -187,7 +187,7 @@
#full_page_writes = on # recover from partial page writes
#wal_log_hints = off # also do full page writes of non-critical updates
# (change requires restart)
-#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers +wal_buffers = 7864kB # min 32kB, -1 sets based on shared_buffers
# (change requires restart) #wal_writer_delay = 200ms # 1-10000 milliseconds
@@ -196,9 +196,9 @@
# - Checkpoints -
-#checkpoint_segments = 3 # in logfile segments, min 1, 16MB each +checkpoint_segments = 32 # in logfile segments, min 1, 16MB each
#checkpoint_timeout = 5min # range 30s-1h
-#checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0 - 1.0 +checkpoint_completion_target = 0.7 # checkpoint target duration, 0.0 - 1.0
#checkpoint_warning = 30s # 0 disables
# - Archiving -
@@ -284,7 +284,7 @@
#cpu_tuple_cost = 0.01 # same scale as above #cpu_index_tuple_cost = 0.005 # same scale as above #cpu_operator_cost = 0.0025 # same scale as above
-#effective_cache_size = 4GB +effective_cache_size = 768MB
# - Genetic Query Optimizer -
@@ -298,7 +298,7 @@
# - Other Planner Options -
-#default_statistics_target = 100 # range 1-10000 +default_statistics_target = 100 # range 1-10000
#constraint_exclusion = partition # on, off, or partition #cursor_tuple_fraction = 0.1 # range 0.0-1.0 #from_collapse_limit = 8
@@ -378,7 +378,7 @@
# fatal
# panic
-#log_min_error_statement = error # values in order of decreasing detail: +log_min_error_statement = error # values in order of decreasing detail:
# debug5
# debug4
# debug3
@@ -404,13 +404,13 @@
#debug_print_rewritten = off #debug_print_plan = off #debug_pretty_print = on
-#log_checkpoints = off -#log_connections = off -#log_disconnections = off -#log_duration = off +log_checkpoints = off +log_connections = off +log_disconnections = off +log_duration = off
#log_error_verbosity = default # terse, default, or verbose messages #log_hostname = off
-log_line_prefix = '< %m >' # special values: +log_line_prefix = '%d %m' # special values:
# %a = application name
# %u = user name
# %d = database name
@@ -431,7 +431,7 @@
# %% = '%'
# e.g. '<%u%%%d> '
#log_lock_waits = off # log lock waits >= deadlock_timeout
-#log_statement = 'none' # none, ddl, mod, all +log_statement = 'all' # none, ddl, mod, all
#log_temp_files = -1 # log temporary files equal or larger
# than the specified size in kilobytes;
# -1 disables, 0 logs all temp files
- Configuration calculator for PostgreSQL
With all this settings maximum connections to the server that I have seen during testing 12.6 is ~27083(number got using live_connection_monitor). Up to 80% of them has TIME_WAIT status and system will close them after some time out - for linux kernel default value is 60 sec's but this value can be different in each OS distributions.
-->If your servers under firewall
Please read section "Configuring Direct Push to work through your firewall" https://technet.microsoft.com/en-us/library/aa997252.aspx
in httpd.conf please set
KeepAlive On
and please specify
MaxKeepAliveRequests desired_concurrent_keep_alive_conections
MaxKeepAliveRequests Directive
otherwise you will have only 150 connections and limits that you have specified in mpm_worker_module(for e.g) will be ignored.
