Scalix Forums

Hi folks!

I'm new to this forum, so please tell me if I do something wrong in this post.
I acquired a new customer with an existing Scalix Server and have never administered such a beast and am facing now problems with services getting stuck from time to time.

Previously, Scalix 11.0.4.25 (Community edition) was installed on this machine and when I was confronted with the problems mentioned above, I upgraded to 11.3.0.77. Instead of
seeing the problems go away, they've become much more nasty. Previously (with the old version) from time to time (mostly during the weekend [0]) the "Internet Mail Gateway" service stopped working, which means it was marked "red" by the SAC. After restarting the service everything was back fine and working but it took the server quite some time to process mails in the queue that couldn't be processed while the service was down.

After upgrading to the latest available version, the problem with the "Internet Mail Gateway" service seems to be gone, but now the server sometimes stops responding on the SMTP port, e.g. I can connect to it using `telnet`, but I don't get the server greeting and no further messages are being accepted.

I found out, that the "SMTPD Relay" service seems to be the name of this service in Scalix terms and restarted it (although it was flagged "green" in SAC) and afterwards,
everything worked fine again.

As I mentioned above, I'm relatively new to Scalix and have read the Administrator Guide as well as quite some topics in here but couldn't find information that's useful to me to solve my problem.

Maybe someone could help me out on this one by advising me to the next steps to be done to get this thing solved.
Is there any kind of debugging for the SMPTD Relay I can turn on?
What files/information do you need to help me get this solved?

Many thanks in advance,
Alexander

[0] No, there are no special backup scripts running at the weekend. I'm running full backups every day so if the problem is related to backups, it should happen every day.
What I _AM_ experiencing is a high amount of spam messages coming in at the weekends, maybe this is related to my problem...

SMTP Relay shouldn't behave like that. I also didn't have my Internet Mail Gateway crash since before Scalix 10, so that's weird as well.

Do a search on the forum for DEBUG_LOG=TRUE (you will have to add this to your smtpd.cfg). That way you will get logs of exactly what your SMTP Relay is doing at any moment.

Also, do a omshowlog -e and see if there's anything related logged.

Thanks, will give it a whirl in a few hours.

i would also check in the fatal log, as most major issues are reported there.

If you turn off the smptd can you telent onto the server using port 25 and get an smtp greeting?

Sorry for not answering earlier.

Here's the relevant information (this'll be a very long post):

`omshowlog -e` tells me about some SERIOUS ERRORs, most of them seem to be related to iconv issues, e.g.:



Don't know if that's relevant to the problem.

Sometimes I also see this error in the logs:



And I guess that after I get this error message things get weird.

I found [this very useful topic] in here and tried all the suggested actions as well as some things from other postings.
Let me summarize what I've done so far.

I experienced tons (more than 100,000) e-mails to my sxadmin account containing mail delivery notifcation reports, etc. and found out that the server I'm working on was missing appropriate entries in /etc/mail/local-host-names, so I added "domain.com" and "mail.domain.com" to this file and restarted sendmail.

That change didn't seem to help that much because I'm still getting error messages to the sxadmin mailbox and all these messages contain text like:



Please note that "info@mydomain.com" and all other addresses mentioned in these e-mails are not valid e-mail addresses on this server so I don't want them to get logged that way. If there's any chance to get rid of them, please let me know.

If the above error message gets logged, usually the following messages follow:



Some other errors I can see in the log files are lots of entries like this ("Datei oder Verzeichnis nicht gefunden" means "No such file or directory" for all non-German readers ):



What file is it looking for?

In the fatal log (/var/opt/scalix/ml/s/logs/fatal) I found out that it _MIGHT_ be looking for a file called "/var/opt/scalix/ml/s/nls/de_AT" which isn't available on my system.
In this directory I only have directories like "AMERICAN", "GERMAN", "C", ... so therefore I created a symlink for de_AT now to make these messages go away but that didn't help at all so I removed the symlink again. Am I missing some files or does this look like some kind of misconfiguration?

What about this one?



One additional question regarding the fatal log:
For the year 2008 now I have about 15526 messages marked with "ERROR" in this logfile and 1564 "SERIOUS ERROR" entries.
Most of them look quite the same (iconv issues, some of them mentioned above) but how many "ERROR" or "SERIOUS ERROR" messages usually get logged to this file during normal operation? I suspect that no error messages should get logged at all during normal operation, so please help me in getting rid of them all.

According to the recommendation to enable DEBUG_LOG in smtpd.cfg, I did this and now it started writing the smtpd-SMTP.log file. Maybe I'll be able to see more detailed information in this file after the next unexpected shutdown (until now, about 24 hours since I made some changes) the SMTPd service didn't stop anymore, but I'm sure it will
stop again.
Yesterday evening, the "Internet Mail Gateway" service seems to have stopped (according to `service scalix status`) but I don't know when this happened and so I cannot provide any logs for this case. What UNIX service or process is meant by Scalix when talking about the "Internet Mail Gateway"?

Is there any way to automatically recover in case of troubles? I found [this topic] talking about a script/program called "ommaint" that should do the trick, but on my installation, only "ommaintbb" is available. Is this the same?

I also had troubles deleting all the thousands of messages in my sxadmin mailbox with the SWA or Thunderbird, so I switched to a somewhat scripted method using the `mailtool` application from the console news- and mailreader cone.
I just want to share this information in case someone else needs it:



Another weird thing is that I usually get the famous GUI022 error codes on the SWA when doing heavy operations, e.g. deleting more than 100 e-mails at once. I searched this forum for possible solutions and tried to run `sxmkindex -r 0` which failed with a return value of 99. Searching again revealed that I needed to disable and re-enable the SIS Indexing for this user and afterwards the command completed successfully.
I then tried to `omtidyallu -M` which did complete without errors too, but the GUI022 errors haven't gone by doing this.
Any other ways to fix this?

Another thread mentioned the use of `omscan` to scan for errors and (if any) automatically repair them. It seemed to have fixed some issues but there are still some issues I'll elaborate on later.

First I tried to "fix" the sxadmin user with `omscan -AvfxSU sxadmin` where I got the following output:



It pretends to fix something but a subsequent call to scan this users mail trays reveals the problem again, so it doesn't seem to have been fixed. What to do now?

I uploaded an anonymized `omscan -a` output to [http://tuxx-home.at/temp/omscan.log] just in case someone is interested in having a look at this output and is willing to help me in fixing this issues.

The next question I want to bother you with:
What's the correct backup strategy for Scalix Servers? The guy who administered the server before I was called to help out has created a script which basically stops scalix and scalix-tomcat, copies over all the files in /opt/scalix and other paths to the backup directory and afterwards starts it again.
Is there any way to backup the Scalix database online?

Another short question:
The guy before me also made a "typo" (at least I think it is a typo) in the Scalix server configuration which annoys me everytime I'm looking at it. The mailserver is called "mail.domain" but is referred to as "mail,domain" (a comma instead of a dot) in the SAC and when running `sxmkindex` f.ex.. Any way to fix this or is a dot not allowed in this place?



After rebuilding the index, trying to clean up the sxadmin mailbox, etc. I still get strange IMAP errors like this one when trying to list the contents of my INBOX:



OK, sorry for this very long posting, I hope someone made it the way down here and is able to help me out of my many problems with this Scalix installation.
If you're missing any information, just tell me and I'll provide that information.

Many, many thanks in advance!

Hi,

Quick answer to the question. We had issues with omsmtpd being very slow to answer incoming connections. in the end, i tracked it down to a broken DNS server in /etc/resolv.conf which fixed the issue after it was gone.

Stopping and restarting services like this can cause other errors in the system, but I think there may be other issues - we used to have several items in the fatal log with older clients, but with 11.3 we only get a around 10 a day in fatal log file for Outlook clients.

Thanks.
Andrew.

We had issues with omsmtpd being very slow to answer incoming connections.

My omsmtpd is _NOT_ slow in answering connections. Telnetting to port 25 happens in realtime without any delays. The DNS service works fine, no outages, no delays.

About the 10 fatal log entries per day: Are you seeing the same entries in your logfile as I mentioned above? e.g. the iconv issues, etc.?

Anyway, any other ideas on how to solve my problems, respectively any other hints or answers to my questions?

No more ideas?

You don't have to answer all my questions at once, if that's what you're afraid of
Small bread crumbs will lead me to my goal of having a fully functional Scalix server too

These are just general comments, they may or may not address your issues.

SMTP Relay aborts when it can't listen on port 25. A default install has sendmail listening on 127.0.0.1 port 25 and SMTP Relay on the other IP addresses. SMTPFILTER is then used to pass messages from SMTP Relay to sendmail. Unless you have a need for a custom configuration I would say you should stay with the default.

It's possible your system's language settings are incorrect.

ommaint is on the Scalix Wiki (www.scalix.com/wiki)

The Internet Mail Gateway is the process that sends email out to sendmail for processing, and accepts mail from sendmail for internal delivery.

For reading a mailbox from the command line, have a look at omlogon, omlogoff, omsend, omread, omlist, etc. Quite a few such commands available.

omscan -S should only be used if you suspect the mailbox size value in the Container Access Monitor got corrupted, and then you need to do it under controlled circumstances. Not for normal use.

OMGUI errors are usually fixed by deleting the IMAP cache for the user (make sure the user is not logged on, or has an idle session open). This may also solve other IMAP issues.

The Scalix server can be backed up online if you have LVM configured correctly on the server (you need to do this at install time). Otherwise you will have to shut down or risk minor data corruption. When doing backups it is VITAL to preserve ownerships and permissions, and do the whole /var/opt/scalix, but other than that the exact method doesn't matter.

mailnodes and domains have nothing to do with one another. Mailnodes are simply containers for users and PDLs, so the system knows where each user is (useful in a multi-server environment) and to make certain admin tasks easier amongst related groups of users. Mailnodes have a hierarchy structure that can be up to 4 levels deep, and the levels are seperated by ",".

Hi

In your logs it says that omsmtpd cannot listen to port 25

can you please post the output of lsof -i:25

Mike

SMTP Relay aborts when it can't listen on port 25.

I know, but _WHY_ does this happen?
When no mails are in the queue and no mails get received, my `lsof -i :25` output looks like this:



Which looks good, but the problem seems to be that omsmtpd sometimes simply stops working.

It's possible your system's language settings are incorrect.

I think you're referring to the iconv issues with this answer, right?
How to check for improper language settings?
The guy who installed this server has set the locale and language settings to de_AT.UTF-8 which is what I never do (I always configure all my servers to use C or something similar as system language, I don't like these german error messages, etc..).

ommaint is on the Scalix Wiki (www.scalix.com/wiki)

OK, thanks, will have a look at it.
Is using `ommaint` still required for the latest Scalix release?

The Internet Mail Gateway is the process that sends email out to sendmail for processing, and accepts mail from sendmail for internal delivery.

Well, I figured that, that's what the name suggests, but _WHAT_ process is this exactly? I'm looking for the name in `ps ax` output to check for that.

For reading a mailbox from the command line, have a look at omlogon, omlogoff, omsend, omread, omlist, etc.

Thanks, that works great.

omscan -S should only be used if you suspect the mailbox size value in the Container Access Monitor got corrupted

How to check for that? As mentioned above, the sxadmin user seems to have some problems with his tray (there's this "Check Size: Should be X instead of X" message when running `omscan` for this user).

OMGUI errors are usually fixed by deleting the IMAP cache for the user

Thanks for this hint, I cleared the IMAP cache and will see if it's getting better.

When doing backups it is VITAL to preserve ownerships and permissions, and do the whole /var/opt/scalix, but other than that the exact method doesn't matter.

LVM snapshots is not what I want, so I think I'm sticking to normal backup procedures.
The last few days I modified the existing backup script a bit to _NOT_ shut down Scalix for backup and guess what? Since I did this, I had no more service losses on this system, so I definetly think that the procedure of shutting down scalix, do the backup, bring it back online is to blame for my service losses.

That would also explain why I'm getting this "can't bind to port 25" message in the logfiles. Obviously, Scalix sometimes can't be stopped completely and when starting it again after the backup has finished, some services refuse to start because they're either already started or are in a zombie-like situation.

Mailnodes have a hierarchy structure that can be up to 4 levels deep, and the levels are seperated by ",".

Thanks for explaining this to me, now it's much more clearer why there's a ',' instead of a '.'.

@Mike:
The `lsof -i :25` output is at the beginning of this message.

Thanks for your help so far!

Hi

There is one thing you may learn sooner or later. A backup or the message store (/var/opt/scalix) may ONLY be acieverd in one of two ways.
1) in real time via LVM snapshot
2) offline by shutting down Scalix

Any other configuration will render the backup unusable due to the nature of the Scalix message store!! As discussed by Valerion

Back to the issue

Can you please disble spamassassin for the time being. I know this only affects the service router and sendmail but when you errors like this we need to eliminate all possible causes.

Spam of often better dealt with at the gateway. In your case spam & malformed messages still hit the mailgateway and smtpd services.

What is in the /var/opt/scalix/xx/s/logs/fatal

Just searching

Is you Scalix server running as a VM guest?

Please post your /etc/hosts

Confirm that your internal dns server has
"A" record pointing to your mailserver
a reverse DNS record that points to your mail server
MX record = mail.yourserver.local


What is the logging level of the smtpd service?
omshowlvl

can you make it 17
omconflvl smtpd 17

Mike

There is one thing you may learn sooner or later.

I don't give up the hope that maybe Scalix may learn sooner or later that doing online backups is a vital feature for enterprise class systems.
AFAIK, exporting user mailboxes works online already, is that true?

Any other configuration will render the backup unusable due to the nature of the Scalix message store!! As discussed by Valerion

Any links for me where I can read more about this discussion by Valerion?

Can you please disble spamassassin for the time being.

How to do that in a Scalix environment? Is it enough to simply deconfigure it from sendmail.cf or are there any other places to look for? (Yes, I'm asking this out of convenience and could google for this as well)

Spam of often better dealt with at the gateway.

I don't have a separate mail-gateway, this machine is the only one connected directory to the internet and serves as mail- and fileserver for a very small company (5 PCs).

What is in the /var/opt/scalix/xx/s/logs/fatal

Well, there's lot of information in this file:



I truncated the fatal log to only contain entries for the current month and anonymized it (replacing subjects, the mail domain name, etc.).
The result (still having 115406 lines, 3.6MiB) can be found here (gzipped):

[http://tuxx-home.at/temp/fatal.gz]

Is you Scalix server running as a VM guest?

No.



Hmm... I've never looked at this file but now as I do, it looks strange, at least the last two lines are redundant (in the best case).
So I now modified the file to look like this:



PInging my-domain.at still resolves to 127.0.0.1 but I don't know if that's a common practice for Scalix so I left it in there for now. Please tell me if I should remove it but please note that the A record of the domain "my-domain.at" does _NOT_ resolve to "mail.my-domain.at" but to "www.my-domain.at" which is a completely different IP address and is not under my control.

Confirm that your internal dns server has

No internal DNS server here, I'm using the one from our provider.

"A" record pointing to your mailserver

Confirmed.

a reverse DNS record that points to your mail server

Yes, but it doesn't point to "mail.my-domain.at" but to "somethingcryptic.provider.at".
That's bad, I know, but the provider won't let us change that without having to upgrade to a much more expensive product so currently I'm using the mailserver of our provider as a smarthost when sending out e-mails to avoid getting mails rejected due to several DNS Blacklists, etc.

I will try to talk to the provider to make this change happen.

MX record = mail.yourserver.local

Confirmed.

What is the logging level of the smtpd service?

7 for all listed items, but smtpd (e.g. "SMTP Relay") is not listed in this output.
Could this be because I already have "DEBUG_LOG=true" set in the smtpd configuration?

If I try to set the loglevel, it complains:



Thanks for your help so far!

Hi

I can see this is goning to be quite a challenge

doing online backups is a vital feature for enterprise class systems

This is what LVM does when set up properly
The reference to Valerion's discussion can be found at
viewtopic.php?p=47716#47716
Spacifically

The Scalix server can be backed up online if you have LVM configured correctly on the server (you need to do this at install time). Otherwise you will have to shut down or risk minor data corruption. When doing backups it is VITAL to preserve ownerships and permissions, and do the whole /var/opt/scalix, but other than that the exact method doesn't matter.

All of our Enterprise clients use LVM. this allows them to do backups in some instances evey 20minutes. without interruption to the client workstation. True enterprise performance.

How to do that in a Scalix environment? Is it enough to simply deconfigure it from sendmail.cf or are there any other places to look for? (Yes, I'm asking this out of convenience and could google for this as well)

Do the reverse of this
http://www.scalix.com/wiki/index.php?ti ... amAssassin

I have to appoligise here as all of our installations are larger than 50 seats and have gateway device. Although Spamassassin is supported by Scalix and configurations are on the wiki I have personally not really tried this except in passing. Given the issues you have and the routing path for incomming messages this MAY be something to look at.

I don't have a separate mail-gateway, this machine is the only one connected directory to the internet and serves as mail- and fileserver for a very small company (5 PCs).

Plenty of open source sollutions exist look at ipcop. Much less greif and more bang for no bucks.

Hmm... I've never looked at this file but now as I do, it looks strange, at least the last two lines are redundant (in the best case).
So I now modified the file to look like this:

Yep that will be one of the defining factors.
Scalix relies very heavily on DNS and hosts. Looks like that was a bit of a mess. You will need to reboot the server for that to become effective. This may bring up a raft of other issues with the scalix internals as the Scalix internals rely heavily on DNS and hostname. Both of these have been woefully misconfigured.

No internal DNS server here, I'm using the one from our provider.

Sorry this does NOT work
You casn get it to sort of work which someone obviously has but as per the last paragraph, this is a network misconfiguration.

You really need an internal DNS server, (same for MS Exchange, Zimbra, and all other mail systems I know about.

In the case of Scalix your internal DNS server needs to have entries as noted above
for your convience
"A" record which has entries 192.168.0.250 mail.my-domain.at
Reverse DNS entry pointing to the same server
MX record 192.168.0.250 mail.my-domain.at

CNAME record scalix-default-mail mail.my-domain.at (Outlook uses this if you use outlook)

omconflvl smtpd 17

sorry this is omconflvl -a smtpd 17

Hope this helps

Mike

[/code]

Thanks, will have a look at the Valerion and Spamassassin stuff tomorrow.

Plenty of open source sollutions exist look at ipcop. Much less greif and more bang for no bucks.

If the meaning of this sentence is that I should look around for alternatives to Scalix: That's not up to me, because as I already said, I just inherited the server from a previous guy having it installed at my (newly acquired) customer and he was always very satisfied with Scalix in his environment until the day where everything stopped working occasionally and the guy before me wasn't able to fix it, so they fired him and now it's my turn

So, if there's at least a small chance to get this working for him, I'll do my best.

Looks like that was a bit of a mess. You will need to reboot the server for that to become effective.

Please don't get me wrong, I've no problem with rebooting this server, but why should this change need a reboot to take effect? There's no kind of nameservice caching daemon active on this system, so the changes should take effect for the next DNS query, shouldn't they?

Scalix internals rely heavily on DNS and hostname. Both of these have been woefully misconfigured.

So, is there anything I need to "reconfigure" now after this change or will it simply stop doing bad things from now on?

You really need an internal DNS server,

No problem with that, I'll give it a bind tomorrow morning and will configure appropriate zones containing all the relevant entries you mentioned above.

Thanks for clearing this up, stay tuned, will post the results after applying the suggestions tomorrow.