After system crash SWA does not work

Discuss the Scalix web client

Moderators: ScalixSupport, admin

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

After system crash SWA does not work

Postby criptus » Sat Jan 31, 2009 4:59 am

After system crash SWA does not work any more.

http://fqdn/m,webmail,sac says:

Service Temporarily Unavailable

The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

:?: :?: :?:

I performed system check and FS says it's clean.

What service do I have to check in order to get SWA working?

Please give some tips!!!

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sat Jan 31, 2009 5:34 am

/var/opt/scalix/sx/s/logs/fatal

says:

SERIOUS ERROR CDA Server (CDA Server ) Sat Jan 31 08:40:27 2009
[OM 28664] There is already a CDA server process running
Pid of logging process: 5389

Is this somehow connected with this issue?


httpd service works!
mail over outlook works fine!
/var/log/messages sasy nothing...
/var/opt/scalix/sx/tomcat/logs/scalix-swa.log says nothing special

/var/log/httpd/error_log however when try to connect says: proxy: AJP: disabled connection for (sx.fqdn)

What proxy?

:?: :?: :?:
Last edited by criptus on Sat Jan 31, 2009 11:07 am, edited 1 time in total.

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sat Jan 31, 2009 9:00 am

Some more outputs
(I run SX 11.4.3)

omstat -a returns:

Item Structure Server Stopped

others OK

Restarting tomcat5 and scalix-tomcat also returns no errors

Nothing special under /var/opt/scalix/sx/tomcat/logs when try to access sac, webmail or m...

except this:

/var/opt/scalix/sx/tomcat/logs/catalina.out says:
LifecycleException: No UserDatabase component found under key UserDatabase
at
org.apache.catalina.realm.UserDatabaseRealm.start(UserDatabaseRealm.java:261)
at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1006)
at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:443)
at org.apache.catalina.core.StandardService.start(StandardService.java:448)
at org.apache.catalina.core.StandardServer.start(StandardServer.java:700)
at org.apache.catalina.startup.Catalina.start(Catalina.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:295)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:433)
Jan 31, 2009 3:11:48 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 179 ms

smpoole7
Posts: 94
Joined: Wed Nov 19, 2008 11:47 pm
Location: Birmingham
Contact:

Postby smpoole7 » Sat Jan 31, 2009 10:19 am

I've been in your situation, so I feel for you. :)

First of all, it's normal for the Item Structure Server to be stopped.

[Edited after I reread your post!]

Code: Select all

No UserDatabase component found under key UserDatabase


Try rebuilding the user databases. Don't know why that would only kill the Web access and not everything, but a good walkthrough is in this thread:

http://www.scalix.com/forums/viewtopic.php?t=12182&highlight=

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sat Jan 31, 2009 11:38 am

No UserDatabase component found under key UserDatabase

might be a problem, but when I hit http://fqdn/m,webmail or sac I still get this:

Image

although httpd root, returns main apache intro site...

Valerion
Scalix Star
Scalix Star
Posts: 2730
Joined: Thu Feb 26, 2004 7:40 am
Location: Johannesburg, South Africa
Contact:

Postby Valerion » Sat Jan 31, 2009 12:28 pm

Check your apache logs to see why the connections are refused by tomcat.

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sat Jan 31, 2009 2:11 pm

/var/log/httpd/error_log says this:

[error] (111)Connection refused: proxy: AJP: attempt to connect to 192.168.1.8:8009 (sx.fqdn) failed
[error] ap_proxy_connect_backend disabling worker for (sx.fqdn)
[error] proxy: AJP: failed to make connection to backend: sx.fqdn
[error] proxy: AJP: disabled connection for (sx.fqdn)
[error] proxy: AJP: disabled connection for (sx.fqdni)

This is it...

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sat Jan 31, 2009 5:51 pm

I did pretty much everything smpoole7 posted here:
viewtopic.php?t=12182&highlight=

smpoole7: great job and very precise explanation!!!

I reinstalled allmost all sx services (tomcat required), except postgre DB. I recovered mail store and webaccess works fine now.

I only have some problems rebuilding user indexes.

rm -f -r /var/opt/scalix/pp/indexes/*
(rm -f -r /var/opt/scalix/pp/indexwork/*) there is no indexwork dir in sx 11.4.3

sxmkindex returns this:
sxadmin /xxx: oops: The container access monitor is not running
.
. etc the same for all usersl

What is this?

How can I get users back?

smpoole7
Posts: 94
Joined: Wed Nov 19, 2008 11:47 pm
Location: Birmingham
Contact:

Postby smpoole7 » Sat Jan 31, 2009 11:14 pm

http://www.scalix.com/wiki/index.php?title=HowTos/RebuildIndex

You disable tomcat and scalix while deleting the indexes; you restart tomcat and scalix before running sxmkindex. In other words, scalix must be running for sxmkindex to work.

If, when you enter "service scalix start," you get an error about the Container server, post that here. You've still got a problem.

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sun Feb 01, 2009 1:31 am

It looks like my happiness didn't last very long...

After reboot, the same old story happend again. If I only knew the reason..

sxmkindex however, worked as expected.

I guess I'll have to reinstall entire scalix from the scratch.

Luckily, this is still testing system, so no damage has been made. And I learned something in case of disaster, which is good afterall.

Anyway, what's the procedure to backup/restore users, resources, groups and other relations among them in case of scalix crash?

Where do users resist? Some data probably inside postgre-DB. But if you delete DB data and reindex users, they must come back from somewhere :shock:

smpoole7
Posts: 94
Joined: Wed Nov 19, 2008 11:47 pm
Location: Birmingham
Contact:

Postby smpoole7 » Sun Feb 01, 2009 9:55 am

Again, the Scalix Gurus(tm) here can speak better to this, but in general:

1. The Scalix mailstore is located in /var/opt/scalix/??, where "??" is usually the first and last letters of the hostname. Ex., mine is "pop," so my directory is "/var/opt/scalix/pp." The posts here usually refer to it as "~/" as a shorthand.

2. The "~/s" subdirectory is where the maildata resides.

3. The postgresql stuff (in the directory "~/postgres") is simply an *index* of the mail data. That's why you can delete it and rebuild it. (But that's also why the Container server has to be running when you do.)

4. All of the mail messages are actually stored in ~/s/data. You can go into those folders and see the actual messages, complete with headers and "to:/from:" stuff.

5. [edited -- added] The ~/s/user directory contains per-user data. You'll see posts here about people having imap and login problems; you can delete the per-user imap-cache in this directory and let it rebuild to solve those issues.

(BTW: on my system, I actually have "user," "user01" and "user02" directories. This is an older mailstore, and with fires and hires over the years, I guess we've grown. I haven't seen anyone else here mention that, but it intrigued me the first time I saw it.)

The directions in the link that I gave you about rebuilding contains everything you need to backup and restore. If your installation is as blown as you say, and you can reinstall Scalix, that's a good idea. But you might want to check your system (including the hardware) to find out why it should blow up like that.

If you're rebuilding from scratch, this would also be an ideal time to change OS, too. You didn't say what you're using, but I've had good results with Opensuse, and *great* results with CentOS. The post that I linked to? We had finished our CentOS rebuild about a week before that date, and it has been running without a glitch since then. Not even a burp. :)

www.centos.org -- we're using CentOS 5.2.

smpoole7
Posts: 94
Joined: Wed Nov 19, 2008 11:47 pm
Location: Birmingham
Contact:

Postby smpoole7 » Sun Feb 01, 2009 10:01 am

PS - the most important tip about backup is that Scalix MUST NOT be running when you copy that mailstore to some other safe place. You'll get a corrupted copy otherwise. The Gurus recommend "omshut" for that; I'm paranoid, so I do the entire "service ?? stop" sequence listed above. :)

I don't take any chances. :)

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Sun Feb 01, 2009 1:12 pm

I've been running fresh CentOS 5.2 too, and also HW is new so this will stay as it is. The reason why machine crashed was simply my mistake. I upgraded kernel soon after entire server was configured.

I did something wrong with grub.conf. During boot process, 'strange' :oops: things happend which caused FS unconsistency. I had to fsck /var directory even by the hand, file by file...

So apparently, FS hasn't recovered as it should.

I didn't know, postgree holds 'only' user indexes. Now I understand sx a few molecules better. Thanks smpoole7 for that, I own you a couple of beers :wink:

About backup, there is very suitable admin guide outhere: http://www.packtpub.com/scalix/book .

Now, I'll try to reconfigure user DB, before I reinstall scalix...

smpoole7
Posts: 94
Joined: Wed Nov 19, 2008 11:47 pm
Location: Birmingham
Contact:

Postby smpoole7 » Sun Feb 01, 2009 11:23 pm

criptus wrote:So apparently, FS hasn't recovered as it should.


Hate to say it, but you might want to reinstall Cent. That's usually the quickest way in a case like this. But this time, if you're going to upgrade the kernel, do it *before *installing Scalix.

About backup, there is very suitable admin guide outhere: http://www.packtpub.com/scalix/book .


I have that book myself. It is good, but I was disappointed that it didn't go very far beyond basic configuration and installation.

criptus
Posts: 86
Joined: Tue Dec 23, 2008 2:23 pm
Location: http://www.slovenia.si/

Postby criptus » Mon Feb 02, 2009 3:51 am

I have that book myself. It is good, but I was disappointed that it didn't go very far beyond basic configuration and installation.


I'd agree with that!

User DB reinstall didn't work :(


Return to “Scalix Web Access”



Who is online

Users browsing this forum: No registered users and 3 guests