TB/TB-2008-04-CTASAV
TB -> TB/TB-2008-04-CTASAV
Contents
- 1 Overview
- 2 How to setup Scalix AntiSpam and Scalix ZeroHour AntiVirus
- 3 Operations
- 4 Troubleshooting
- 4.1 How do I determine if I have a valid license
- 4.2 Checking that the submission port is open
- 4.3 Test that the submission port is used for SWA/Platform/SSMTP
- 4.4 How do I check if mails are (not) scanned
- 4.5 I can not see the sxmilter service
- 4.6 Checking that sxmilter is running
- 4.7 New audit events
- 4.8 Where to find log files
- 4.9 omshowlog shows "MILTER: milter_eomActions: mkstemp failed"
- 4.10 Can not send mail from mobile web access: An error occurred: could not send message (D00001)
- 5 Implementation Details
Overview
Scalix AntiSpam and Scalix ZeroHour AntiVirus are two integrated product options which provide the respective protection for your systems. Both are based on OEM technology powered by Commtouch's reliable anti-spam, Zero-Hour virus outbreak protection . They are separately licensed products and available for commercial customers running either Scalix Small Business, Enterprise or Hosted edition. Please contact your sales representative if you are interested in one or both of them.
Known Issues
The sxmilter binary allows you to report back False Positives/Negatives when used in tool mode (refer to man sxmilter for details). As of now this will in case of False Positives only send back the message if the level is found to be 4. Messages categorized 3 or lower will not be sent back. Use the process described under How to manualy report a false negative/positive instead.
How to setup Scalix AntiSpam and Scalix ZeroHour AntiVirus
To be able to use Scalix AntiSpam and Scalix ZeroHour AntiVirus you will need to perform the following steps:
- Use a Scalix AntiSpam and/or ZeroHour AntiVirus enabled license
- Split inbound and outbound mail traffic
- Enable content scanning
- Restart services to activate configuration changes
- Customize AntiSpam and AntiVirus behaviour
Use a Scalix AntiSpam and/or ZeroHour AntiVirus enabled license
Please refer to How do I determine if I have a valid license on how to check that the license you are using includes support for Scalix AntiSpam and/or Scalix ZeroHour AntiVirus.
Split inbound and outbound mail traffic
In the default setup traffic passing the relay is a mix between inbound, outbound and internal mails. To lower the risks of false positives it is recommended to setup Scalix to not scan internal and outbound mail. For that you will need to:
- Enable Submission Port
- Change Scalix Web Access to use the new submission port
- Change Platform to use the new submission port
In case you are using IMAP clients you will also need to implement one of the following possibilities:
- Alter SSMTP stunnel setup
- Alter IMAP client settings
Scalix recommends that you setup stunnel to provide SSMTP and configure your IMAP clients to use this encrypted communications channel.
Enable Submission Port
To enable the Relay's submission port apply the following changes to ~/sys/smtpd.cfg.
Enable submission master switch:
# Uncomment the following lines to enable the Submission and LMTP listeners SUBMIT=ON
Define port to listen to:
# The following group sets the configuration for the submission listener # This listener is only active if SUBMIT=ON is above # By default it binds to port 587 [SUBMIT] LISTEN=0.0.0.0:587 # Reject all anonymous connections ANONYMOUS Log_Reject ALL
Multi server setup considerations
Please be advised that when running a multi server setup all participating servers must have their submission ports enabled and must use the same port. Running a mixed setup where some servers have the submission port enabled and some not or when servers use different ports may lead to problems when sending mail.
Change Scalix Web Access to use the new submission port
To configure SWA to use the new port you need to alter ~/webmail/swa.properties:
#swa.email.smtpServer=demo.scalix.com swa.email.smtpServer=demo.scalix.com:587
Change Platform to use the new submission port
To configure the platform to use the new port please alter ~/platform/platform.properties:
# # SMTP submission port # #smtp.port=25 smtp.port=587
Please make sure that the submission port is bound to the ip defined in smtp.host:
# # SMTP submission host # smtp.host=demo.scalix.com
Alter SSMTP stunnel setup
By changing the server's stunnel setup to use the new submission port all clients using SSMTP will automatically bypass scanning of sent messages.
For that you will need to change your stunnel configuration, typically found under /etc/stunnel/stunnel.conf, to forward connections ending at port 465 to the new submission port:
[ssmtp] accept = 465 connect = 587
Alter IMAP client settings
Please refer to your client's documentation on how to alter the port used for sending mail.
For Thunderbird this option can be found under Account Settings/Outgoing Server:
How to enable content scanning
To enable MILTER support you need to change the MILTER master switch in ~/sys/smptd.cfg to TRUE:
# master switch to enable milter support (default off) SMTPMILTER=TRUE
and uncomment the Commtouch milter definition:
# list of milters to call sequentially (default none)
INPUT_MAIL_FILTER=('CTmilter', 'S=local:~/temp/CTmilter_socket, F=T, T=C:300s;S:10s;R:10s;E:300s')
Restart Services
To enable the submission port and milter support you will need to restart the relay. To immediately restart it use:
$ omoff -d 0 "SMTP Relay" Disabling 1 subsystem(s). $ omon "SMTP Relay" Enabling 1 subsystem(s).
To activate the changes made to SWA and the platform you will need to restart scalix-tomcat. To immediately restart it use:
$ /etc/init.d/scalix-tomcat restart
In case you changed your stunnel setup you will also need to restart stunnel.
Customize Anti-Spam/Anti-Virus
Please refer to the file ~/sys/milter.cfg for detailed information on how to configure the behavior of Scalix AntiSpam and Scalix ZeroHour AntiVirus. This file provides a couple of examples, detailed description of understood options and their meaning.
Operations
How Scalix Scalix AntiSpam and Scalix ZeroHour AntiVirus alter your mails
Depending on your configuration Scalix AntiSpam and Scalix ZeroHour AntiVirus will edit, delete or add headers. The RefID header will always be added, per default it is called: X-SMP-CT-RefID:
... User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) X-SMP-CT-RefID: str=0001.0A0B0209.481B49A5.00AD,ss=1,fgs=0 X-Spam-Flag: NO X-Spam-Level: * X-Spam-Status: No, score=1 required=3 tests=Commtouch ...
How to report a false negative/positive
IMPORTANT NOTICE: There are known issues when reporting False Positives using sxmilter in tool mode (See Known Issues). Use the process described under How to manualy report a false negative/positive instead.
Through the sxmilter tool you are able to report false negatives and false positives. For this to work properly the message you want to report will need to contain a valid Commtouch RefID header.
To report a false negative spam message use
$ sxmilter -c 1 -N /tmp/this.was.really.spam.txt
Before sending the report back to Commtouch sxmilter will rescan the message to check if it is still wrongly categorized.
Please note that passing the -c option to select the category is mandatory when reporting back false negatives/positives. Refer to sxmilter's man page for more details.
How to manualy report a false negative/positive
When sending filtered messages to Commtouch for analysis, it is important to note that the original messages must be attached to your email report rather than forwarded to Commtouch. This is to ensure that all the original message-headers are also sent to Commtouch for analysis. Reports that do not contain the original message-headers cannot be analyzed.
Spam classification errors should be reported to the following addresses:
- False Negative to reportfn@blockspam.biz
- False Positive to reportfp@blockspam.biz
Virus classification errors should be reported to the following addresses:
- False Negative to reportfn@vodlab.biz
- False Positive to reportfp@vodlab.biz
Commtouch is unable to analyze old messages because spam characteristics are dynamically changing over short periods of time due to the nature of spam distribution methods. It is therefore, very important that you send reports about classification mistakes as soon as possible. In general, you should avoid sending reports that are older than one week.
How to test scan a message
You can use the sxmilter tool to perform test scans on messages. To check a message for SPAM and VIRUS classification use:
$ sxmilter -c 3 -T /tmp/example.message.txt spamLevel=1 CSDKMain_ClassifyMessage antispam completed. virusLevel=1 CSDKMain_ClassifyMessage antivirus completed.
The scanned example message contained in /tmp/example.message.txt scored 1 for both SPAM and VIRUS test.
The following test shows a message being classified as spam
$ sxmilter -c 1 -T /tmp/spam.txt spamLevel=4 CSDKMain_ClassifyMessage antispam completed.
Please refer to the to man page of sxmilter for more information.
Please make sure that the file you are trying to scan is readable by user scalix, else the test run will fail with a non zero exit code.
Troubleshooting
It is recommended that before you try out any hints given in this section you check that you have a valid license providing AntiSpam and/or AntiVirus abilities.
How do I determine if I have a valid license
You need a license with one or both of the following features listed:
| SMP_ANTISPAM_CT | for Anti-Spam support |
| SMP_ZEROHOURAV_CT | for Zero-Hour Virus protection |
To check the available features for your license run sxlicense and refer to the Features: part of the output.
$ sxlicence Scalix License Status License Type: Permanent Key Type: Scalix Standard System Class: Multi-Server Domain: demo.scalix.com ID: 42 LVID: 2008-05-31 Premium Users: 50 Standard Users: unlimited Features: TNEF WIRELESS MIGRATION HIGH_AVAILABILITY MULTI_INSTANCE MULTI_SERVER RECOVERY_FOLDER SMP_ANTISPAM_CT SMP_ZEROHOURAV_CT ARCHIVER License File: /var/opt/scalix/do/s/LicenseKeys/permanent.lic License Fingerprint (MD5): E5:4A:E5:25:27:D8:EC:CE:2F:48:9E:AC:09:0A:AF:0C
Checking that the submission port is open
You can check that the submission port is open with the help of netstat:
# netstat -anpt | grep LISTEN | grep omsmtpd tcp 0 0 0.0.0.0:587 0.0.0.0:* LISTEN 15256/omsmtpd SUBMI tcp 0 0 192.168.220.3:25 0.0.0.0:* LISTEN 15250/omsmtpd SMTP
If the port is not open please check your smtpd.cfg and be sure to restart the relay after changeing smtpd.cfg.
You can also use telnet to check that the relay is listening on the submission port:
$ telnet demo.scalix.com 587 Trying 192.168.220.3... Connected to demo.scalix.com (192.168.220.3). Escape character is '^]'. 220 demo.scalix.com ESMTP 11.4.0.11656; Sun, 20 Apr 2008 21:54:34 +0200 (CEST)
Test that the submission port is used for SWA/Platform/SSMTP
Refer to the table below and use the respective client to send an internal test mail.
| Component | Client |
|---|---|
| SWA | Scalix Web Access |
| Platform | Mobile Web Access |
| SSMTP | Thunderird configured to use SSMTP |
Then analyze the mail as described under How do I check if mails are (not) scanned.
How do I check if mails are (not) scanned
By looking at a message's headers you can find out if it has been processed by Commtouch or not. A scanned mail will always contain a RefID Header, an unscaned message not. The following example has been scanned as there is a RefID header present:
To: Feed Demo <Feed.Demo@demo.scalix.com>
Message-ID: <13928.10001208793135.demo.scalix.com@MHS>
Subject: test Mon, 21 Apr 2008 17:52:15 +0200
X-Mailer: swaks v20061116.0 jetmore.org/john/code/#swaks
X-SMP-CT-RefID: str=0001.0A0B0202.4832C20D.0082,ss=1,fgs=0
MIME-Version: 1.0
Content-Type: text/plain;
charset="US-ASCII"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline
I can not see the sxmilter service
You need to have a valid license to use Scalix AntiSpam and Scalix ZeroHour AntiVirus. If you do not have a valid license you will not be able to see the Milter Server.
If you have a valid license you are able to see the MILTER Service when running omstat:
$ omstat -a | grep Milter Milter Server Started 06:29:43
Checking that sxmilter is running
You can check if sxmilter is running by executing
$ omstat -a | grep Milter Milter Server Started 10:21:56
or through ps
$ ps fax | grep sxmilter 21032 ? SNsl 0:00 sxmilter COMMTOUCH
or through omrc
omrc | grep milter scalix 21032 1 0 06:29 ? SNsl 0:00 sxmilter COMMTOUCH
The first parameter displayed for sxmilter in the process list will be the group name you chose in ~/sys/milter.cfg:
# group config for COMMTOUCH milter [COMMTOUCH]
New audit events
The following new audit events have been introduced:
| Name | Level | Description |
|---|---|---|
| milter-start | 9 | Relay is about to pass details of the message on to the milter |
| milter-end | 9 | Milter has passed the final results back to the relay |
| milter-add-rcpt | 11 | Relay has added a recipient as requested by milter |
| milter-del-rcpt | 11 | Relay has deleted a recipient as requested by milter |
To retrieve the current configured audit level for the relay use:
$ omshowaud -a | grep SMTP SMTP Relay 0
To enable logging of milter-start and milter-end events raise the level to 9:
$ omconfaud -a smtpd 9 omconfaud : Logging level updated OK. $ omshowaud -a | grep SMTP SMTP Relay 9
You need to restart the relay to activate the changed audit level. To immediately restart it use:
$ omoff -d 0 smtpd Disabling 1 subsystem(s). $ omon smtpd Enabling 1 subsystem(s).
The audit log information will be written to ~/log/audit. Below is an example with the above settings:
SMTP-Relay time 1208792837 Mon Apr 21 17:47:17 2008 +120 originator stefan.voelkel@demo.scalix.com milter-start name=CTmilter;time=1208792837;state=accept milter-end name=CTmilter;time=1208792837;state=accept recipient Feed.Demo@demo.scalix.com mta-message-id 12526.10001208792837.demo.scalix.com message-size 498 hop-count 0 summary-target U
To enable the milter-add-rcpt and milter-del-rcpt events you will need to increase the level to 11:
$ omconfaud -a smtpd 11 omconfaud : Logging level updated OK.
After restarting the relay you can find the modification events in the log file:
... milter-start name=CTmilter;time=1208793135;state=accept milter-add-rcpt name=CTmilter;time=1208793135;rcpt=stefan.voelkel@demo.scalix.com milter-end name=CTmilter;time=1208793135;state=accept ...
As with all other audit events, please refer to man omconfaud for further information.
Where to find log files
You can use omshowlog to access current log data.
To retrieve the current log level run
$ omshowlvl -a | grep Milter Milter Server 7
To alter the milter server loglevel to 9 use
$ omconflvl -a sxmilter 7 omconflvl : Logging level updated OK.
Please refer to man omconflvl for details on logging levels.
To enable MILTER API logging, set DEBUG_LEVEL in ~/sys/milter.cfg to 6 inside the group you want to enable logging for. Then stop sxmilter and run it in foreground.
To enable SMTPD logging use omshowlog, omshowlvl and omconflvl with smtpd as shown above.
To enable relay logs set DEBUG_LOG to TRUE in ~/sys/smtpd.cfg and check the file ~/tmp/smtpd-SMTP.log for Client Milter and Backend logging information. A successful startup with logging enabled may look like this:
[2008-04-20 17:13:48] Logging Started pid 28496 [2008-04-20 17:13:48] Cli 0/? milter->milter: OpenSession name 'CTmilter' path ' local:~/temp/CTmilter_socket' flag 'T' [2008-04-20 17:13:48] Cli 0/? milter->milter: Command2 cmd 'O' len 12 pnr 0 reply 'O' [2008-04-20 17:13:48] Cli 0/? milter->milter: Negotiate version 2 actions 31 protocols 0 [2008-04-20 17:13:48] Cli 0/? milter->milter: Negotiate reply 'O' result 0 [2008-04-20 17:13:48] Cli 0/? milter->milter: OpenSession result 1 [2008-04-20 17:13:48] Cli 0/? milter->milter: Command2 cmd 'Q' len 0 pnr -1 reply 'c' [2008-04-20 17:13:48] Cli 0/? milter->milter: Close reply 'c' result 0 [2008-04-20 17:13:48] Cli 0/? milter->milter: CloseSession result 1 [2008-04-20 17:13:48] Cli 0/? cfg->milter: AddToMilterList result 1 [2008-04-20 17:13:48] SMTP Relay 11.4.0.11641 started (28496): Sun, 20 Apr 2008 17:13:48 +0200 (CEST) for SMTP
Besides the possibilities listed above you can also run sxmilter in the foreground by first stopping the service and then running it with the -i parameter:
$ omoff -d 0 "Milter Server" Disabling 1 subsystem(s). $ sxmilter -i
To run smtpd in foreground use
$ omoff -d 0 -a smtpd Disabling 1 subsystem(s). $ omsmtpd -i
omshowlog shows "MILTER: milter_eomActions: mkstemp failed"
Make sure the directory specified in the SAV_DAT_IF_AS_LEVEL_N options located in ~/sys/milter.cfg exists. With the following setting in milter.cfg:
SAV_DAT_IF_AS_LEVEL_4=~/spam/CTmilter.XXXXXX
messages categorized SPAM level 4 would be created in the directory spam located in the server's home directory. Depending on your installation the path to the server's home may vary, but will look something like this:
/var/opt/scalix/do/s/spam
With the above settings a possible filename for a spam message would be:
/var/opt/scalix/do/s/spam/CTmilter.52xVgi
Make sure that the user scalix has write permissions in this folder. To setup the above example you could use the following commands:
$ mkdir /var/opt/scalix/do/s/spam $ chown scalix:scalix /var/opt/scalix/do/s/spam
Please keep in mind that you need to take care of cleaning out these directories, as Scalix housekeeping scripts will not touch them.
Can not send mail from mobile web access: An error occurred: could not send message (D00001)
This is most likely caused by a wrong submission port configuration in platform.properties. If you are running a multi server setup make sure that all servers are configured to use the same submission port. You can check ~/tomcat/logs/scalix-api.log for an exception like this:
Caused by: javax.mail.MessagingException: Could not connect to SMTP host: demo.scalix.com, port: 587;
nested exception is:
java.net.ConnectException: Connection refused
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:1378)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:399)
at javax.mail.Service.connect(Service.java:275)
at com.scalix.api.delivery.Delivery.sendMessage(Delivery.java:114)
... 30 more
You can also check with the help of strace by attaching to the running java process, in this example its pid is 32767:
$ strace -econnect -f -p 32767
...
[pid 614] connect(54, {sa_family=AF_INET, sin_port=htons(587), sin_addr=inet_addr("192.168.220.3")}, 16) = -1 ECONNREFUSED (Connection refused)
...
A successful strace log would look something like this:
$ strace -econnect -f -p 4298
...
[pid 4426] connect(54, {sa_family=AF_INET, sin_port=htons(587), sin_addr=inet_addr("127.0.0.1")}, 16) = 0
...
Implementation Details
To improve the relay's throughput and to adapt to the blocking nature of milter calls the default configuration of the number of relay processes has been changed. Up to Scalix 11.3 there were 6 relay processes, each handling a maximum of 16 concurrent connections. Starting with Scalix 11.4 there will be 16 processes, each handling a maximum of 6 concurrent connections.
A new MILTER interface has been implemented in the Scalix SMTP Relay. The Commtouch service is integrated through this new MILTER interface by wrapping Commtouch's scanning engine ctengine. For more information on MILTERs please refer to http://milter.org
Please be advised that the option SMTPFILTER has been deprecated in favor of SMTPMILTER, and should not be used any longer.
The Commtouch MILTER will be run by the new sxmilter service.
All existing Relay rules in smtpd.cfg will be applied exactly once before the messages gets processed by any configured milters. Unlike sendmail Scalix will call milters sequentially and not in parallel to avoid conflicting changes to headers by multiple milters.
A message will only be accepted if all configured milters return the ACCEPT action. As soon as one returns with an action different than ACCEPT (e.g. DISCARD, TEMPFAIL, ...) processing stops, the later milters will not be called and the last action returned will be applied to the message.
When a message has been processed by Scalix AntiSpam or Scalix ZeroHour AntiVirus a special header, by default called X-SMP-CT-RefID, will have been added to it. This header is used to identify the message when reporting false positives/negatives.
The newly implemented MILTER interface can also be used to integrate other applications like Spamassassin, AMaViS or ClamAV through their respective milters. Be advised that this has not been tested by Scalix.
The provided MILTER for Commtouch ctengine integration is intended to be used with Scalix only, no tests have been made to ensure sendmail compatibility.
Scalix is compatible to version 2 milters.
It is possible to prefix the subject of a spam message with something like "***SPAM***", but this will ignore the subjects encoding and might break the display of some character sets.
Supported MILTER features
The following features are supported:
| Header | add, insert or change headers |
| Rcpt | add or delete recipients |
| Action | accept, reject, discard or tempfail message |
The meaning of the available actions are:
| Accept | Deliver the mail to it's recipients |
| Reject | Reject message (generates error on sending side) |
| Discard | Accept message but silently drop it |
| Tempfail | Temporary fail the message and let the sending side try again later |
Additionally the following feature is supported by the relay, but not by the Commtouch milter:
| Body | replace original body with something else |
It can be used in combination with other milters though.
Unsupported MILTER features
| Change 'from' | Not supported by Scalix 11.4 |
| Quarantine action | At this point we do not support the quarantine message action, as a workaround you may want to consider replacing the recipient instead. |
| Extended recipients | When adding or deleting recipients you can not use "First Last <first.last@scalix.com>" only plain email addresses are supported e.g. "first.last@scalix.com". |
| Symbol list | You can not pass sendmail macros to the milter. |
| Reply with code and text | You can not customize the reply code and text, e.g. "441 tempfail by milter X for reason Y" is not possible. |
Milter configuration options
You can alter several parameters influencing milter behavior in timeout situations[1]:
INPUT_MAIL_FILTER=('CTmilter', 'S=local:~/temp/CTmilter_socket, F=T, T=C:300s;S:10s;R:10s;E:300s')
The three parameters passed to the milter are
| Name | Meaning |
|---|---|
| S | Location of the socket to talk to |
| F | What to do if the milter is unavailable (must be F=T) |
| T | Timeout parameters |
Please note that unlike sendmail Scalix only supports tempfailing a message when the milter is unavailable (F=T) because at the time the milters are processed the connection has already been accepted.
The timeout parameter is composed by the following flags separated by semicolon and postfixed by either s for seconds or m for minutes:
| Flag | Meaning |
|---|---|
| C | Timeout for connecting to a filter. If set to 0, the system's connect(2) timeout will be used. Default: 5m |
| S | Timeout for sending information from the MTA to a filter. Default: 10s |
| R | Timeout for reading reply from the filter. Default: 10s |
| E | Overall timeout between sending end-of-message to filter and waiting for the final acknowledgment. Default: 5m |
In this example
INPUT_MAIL_FILTER=('CTmilter', 'S=local:~/temp/CTmilter_socket, F=T, T=C:5s;S:10s;R:15s;E:20s')
the message will be tempfailed if any of the following is true:
- connecting to the milter takes longer than 5 seconds
- writing the message to the milter takes longer than 10 seconds
- reading the reply from the milter takes longer that 15 seconds
- the whole operation takes longer than 20 seconds
Please note that changing the timeout values can have severe impacts on your system. Do not change these values without fully understanding the consequences.
Message Path if SMTPMILTER is set to TRUE
- Client->Milter: cache msg up to the last dot before reply
- Milter->Milter: call 1st milter to process the cached msg
- Handle msg modification from 1st milter (or msg action)
- Repeat step 2 and 3 for Nth milter using modified msg
- Milter->Backend: handle msg action and reply to Client
- ACCEPT: start backend (unix.in) and send the modified msg, then reply back to Client with 2xx if OK (or unix.in error if not OK)
- REJECT: drop the msg, reply back to Client with 5xx
- DISCARD: drop the msg, reply back to Client with 2xx
- TEMPFAIL: drop the msg, reply back to Client with 4xx
