Large queue - mail not sending or receiving

[ltward]

That means you have over a million messages on the unix queue.
I have never seen anywhere NEAR that amount of messages on a queue!
10,000 would be crippling; usually the queues process quite quickly.

I would try:
omshut (shut down scalix)
omrc -n (start scalix with background daemons but no services)

omqdump
The password is A, then todays date + 10, then E
So on the 25th of the month the password is A35E
On the 31st it's A41E
On the 2nd it's A12E

Option? D - delete messages from queues
Queue(): unix
Repeats (0): <- here I'd try maybe 5000 or 10000 and see what happens

Give it some time. It will take ages to process a queue of that size.
If you can delete 5K - 10K messages with omqdump, then I would try 50K or 100K.
With a queue that size I would just delete like an insane person; I would not try to save the "good" messages and delete the "bad" messages. There is simply too much to process already.

Once you get things cleaned up you can turn on the services with
omon -s all

You might want to put in some anti-spam solution. Scalix can interface with any anti-spam product that hooks into sendmail (SpamAssassin, for example). Look in the Scalix Setup and Configuration Guide (http://www.scalix.com/documentation for all manuals) in the chapter called "Spam Protection".

You can also create rules that will silently delete all messages with a certain subject, or from a certain domain. Check the Administration Guide in the chapter about "Setting Message Delivery Rules on the Router".

[SidebandSamurai]

You know,

I am having the same problem. My server has been down for over 24 hours as I deal with the internet mail queue all stuffed up. I have the server disconnected from the network and I still get incrementing internet mail queue messages. I have a feeling that my server has a rootkit on it.

I wonder if yours does too.

do from the command line:

omshut ld sr smtpd

then service sendmail stop

Then disconnect from the network to see if the internet mail queue still increments.

Sideband Samurai

[SidebandSamurai]

Here are my experiences.
In answer to your first questions, Yes and Yes unless the server is just catching up or is really busy. If its really busy, seeing 100 messages in the message queue at anyone time might be normal for your server.

In answer to your second question -> the "unix" queue is the also known as the "internet mail queue" on SAC. This is where mail goes for delivery out to the internet.

In answer to your third question, See above What are they doing, waiting to be delivered to destinations on the internet.

These are basic simplistic answers, I know but as to the flow of how the email flows through the server I don't quite understand that yet. BUT!!! have the answer to your forth question.

Well first I have Good news and bad news. First the bad news. NO

for the good news here is how you delete your messages from the internet mail queue.

This is going to take a LONG time, with that many messages. so be patient.

first execute the following commands:

Code: Select all

omoff ld sr smtpd

this shuts down some of the routing mail services

next enter this command
(this depends on what OS you are running, I am assuming CentOS or some derivative to that )

Code: Select all

service sendmail stop

this basically stops the spam mail (or any mail for that matter) from moving through the system. This allows you to then look at the email and delete it by subject.

Since you are having problems with the unix queue, these examples are only for the unix queue. replace the queue unix with what ever queue you are working with.

In the scalix wiki is a nice little script that helps you delete messages from these queues in mass.

Here is the link:

Code: Select all

http://www.scalix.com/wiki/index.php?title=Delete_messages_from_a_queue_using_omstat

now from your mail server command prompt (or putty what ever you choose) enter the following command

Code: Select all

[root@yourserver ~]# omstat -q unix

and you will see:

Code: Select all

55768183   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768187   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768191   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768195   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768199   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768207   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768215   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768220   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768228   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768236   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768244   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768252   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768260   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768268   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768272   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11
55768280   samwhite / internet    MSG   U CONTACT MR.RAYMOND WILSON A 03.05.11


Those are the messages you will want to delete.

Copy the code on the wiki page I just gave you and save it to your roots home directory.

call it what ever you like and add a #!bash to the very first line, So it will look like this

Code: Select all

#!/bin/bash
omstat -q router | grep "Check this out" | cut -d" " -f 1 | while read msgid
do
 omstat -q router -j -R -m $msgid
done


Now examine the code in your new script and you will see the lines that start like this:

Code: Select all

omstat -q router

Change router to unix in BOTH SPOTS

Now the fun begins

copy the subject from your omstat -q unix command. In the above example the subject is "CONTACT MR.RAYMOND WILSON A" your subject is one space after the priority of the message (in this case U) and one space before the date. In this case its 03.05.11

modify your script again so that you change the string grep is looking for to the subject you just copied. Your script will look like this when finished:

Code: Select all

#!/bin/bash
omstat -q unix | grep "CONTACT MR.RAYMOND WILSON A" | cut -d" " -f 1 | while read msgid
do
 omstat -q unix -j -R -m $msgid
done

Save your script and run this command:

Code: Select all

chmod 700 ./deletespam.sh

this allows your new script to be executed.

now run your script by executing the following command:

Code: Select all

./deletespam.sh

Now go have a cup of coffee or two or three or even four, Did I say this was going to take a long time? for my measly 150,000 message it took an hour to delete. I would imagine for you it will take at least 8 times as long.

If you find that its taking 5 or more minutes to remove a single message, reboot the server and immediately shut down those services I told you about at the beginning of the message, then re execute your script, and it will pickup right where you left off.

You can execute this command in another putty window as messages are being removed:

Code: Select all

watch omstat -s

which will update the omstat -s every two seconds so that you can see the messages decrement as your script runs.

Even though you have cleaned up your mail server you need to discover the source of these spam messages, it is possible that a spambot on your network is using scalix as its way of sending spam. You will need to find that machine (or machines) and remove them from the network.

you will have messages in other queues that are backed up also, you will need to change your script from unix to what ever queue you need to clean out and perform the same procedure on each and every queue that has backed up messages.

Good Luck

Sideband Samurai

[SidebandSamurai]

You can leave it alone, but if it starts deleting a message every 5 minutes then reboot the mail server and make sure you re-shutdown all the services mentioned above. Also make sure you execute

Code: Select all

omoff unix

also so that the internet mail service queue does not pass messages to sendmail (for sending out to the internet)

Then you also will need to clear out the sendmail message queue.

go to /var/spool/mqueue

execute this command

Code: Select all

rm -f *

then after that is done do an mqueue and you should see no messages in that queue.

good luck to you.

Also check your audit logs for compromised accounts. Your audit log are located in /var/opt/scalix/<first and last of host name>/s/logs/audit Look for a string called Authenticated by. You also may have to ENABLE auditing. I might suggest using these commands as follows:

Code: Select all

omconfaud unix router 9

- but check the man page just to make sure that is correct.

--Sideband Samurai