Webmail and SAC not working after 11.2->11.3 *Fixed*

[Ruthiness]

Need some help here. I see some errors like this in the logs:

mod_jk.log

[Sun Feb 10 19:22:49 2008] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mailpa
[Sun Feb 10 19:23:28 2008] [info] jk_open_socket::jk_connect.c (444): connect to 10.49.1.40:8009 failed with errno=111
[Sun Feb 10 19:23:28 2008] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (10.49.1.40:8009) with (errno=111)
[Sun Feb 10 19:23:28 2008] [info] ajp_send_request::jk_ajp_common.c (1248): Error connecting to the Tomcat process.
[Sun Feb 10 19:23:28 2008] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Sun Feb 10 19:23:28 2008] [info] jk_open_socket::jk_connect.c (444): connect to 10.49.1.40:8009 failed with errno=111
[Sun Feb 10 19:23:28 2008] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (10.49.1.40:8009) with (errno=111)
[Sun Feb 10 19:23:28 2008] [info] ajp_send_request::jk_ajp_common.c (1248): Error connecting to the Tomcat process.
[Sun Feb 10 19:23:28 2008] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=2
[Sun Feb 10 19:23:28 2008] [info] jk_open_socket::jk_connect.c (444): connect to 10.49.1.40:8009 failed with errno=111
[Sun Feb 10 19:23:28 2008] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (10.49.1.40:8009) with (errno=111)
[Sun Feb 10 19:23:28 2008] [info] ajp_send_request::jk_ajp_common.c (1248): Error connecting to the Tomcat process.
[Sun Feb 10 19:23:28 2008] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=3
[Sun Feb 10 19:23:28 2008] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on th
e wrong port. worker=mailpa failed


and

This is in access_log reflecting attempts to access webmail:

10.49.1.40 - - [10/Feb/2008:20:44:28 -0500] "GET /sis/indexer?fn=add&uid=0c400000ee98b274-04.1.94.01&pdref=00011c2cbec0dffc&dref=00014aec8e30ae5f&indexid=2801310a-472b89ee-478c1ee8-2b920&flags=unseen,unflagged,unanswered,undeleted,undraft,unlabel1,unlabel2,unlabel3,unlabel4,unlabel5,unlabel6,unlabel7,unlabel8,unjunk,unnonjunk,unforwarded HTTP/1.1" 404 988 "-" "-"
10.49.1.40 - - [10/Feb/2008:20:44:38 -0500] "GET /sis/indexer?fn=add&uid=0c400000ee98b274-04.1.94.01&pdref=00011c2cbec0dffc&dref=00014aec8e30ae5f&indexid=2801310a-472b89ee-478c1ee8-2b920&flags=unseen,unflagged,unanswered,undeleted,undraft,unlabel1,unlabel2,unlabel3,unlabel4,unlabel5,unlabel6,unlabel7,unlabel8,unjunk,unnonjunk,unforwarded HTTP/1.1" 404 988 "-" "-"

I can access the basic Apache page via https, but the tomcat to apache part is not working, thus SAC and Webmail are not.

I found some info on ssl and am wondering if I need to do something with the key somewhere after the upgrade. We access SAC and Webmail ONLY over SSL. I saw another thread where it was mentioned that tomcat can't communicate over ssl.. and I see several new include files in the connector conf files dated today.

An example:

/etc/opt/scalix-tomcat/connector/jk

[root@mailpa jk]# ls -alt
total 48
-rw-r--r-- 1 root root 21 Feb 10 19:27 app-mailpa.api.conf
-rw-r--r-- 1 root root 21 Feb 10 19:27 app-mailpa.caa.conf
-rw-r--r-- 1 root root 21 Feb 10 19:27 app-mailpa.res.conf
-rw-r--r-- 1 root root 21 Feb 10 19:27 app-mailpa.sac.conf
-rw-r--r-- 1 root root 21 Feb 10 19:27 app-mailpa.sis.conf
-rw-r--r-- 1 root root 25 Feb 10 19:27 app-mailpa.webmail.conf
-rw-r--r-- 1 root root 36 Feb 10 19:22 workers.conf
drwxr-xr-x 2 root root 4096 Dec 29 20:44 .
drwxr-xr-x 4 root root 4096 Dec 29 20:44 ..
-rw-r--r-- 1 root root 1099 Dec 19 01:56 instance-mailpa.conf
-rw-r--r-- 1 root root 507 Dec 19 00:31 instance-mailpa.conf.bak
-rw-r--r-- 1 root root 19 Nov 2 16:38 app-mailpa.m.conf

more app-mailpa.webmail.conf
JkMount /webmail* mailpa

I stopped all services and did an lsof -i: 8009, found nothing there. This is what is there after restart:

httpd 1511 apache 13u IPv4 50158766 TCP mailpa.ourdomain.com:49972->mailpa.profitpointinc.com:8009 (ESTABLISHED)
(several more like this)

java 3370 root 14u IPv4 50158092 TCP mailpa.ourdomain.com:8009 (LISTEN)

(several more to other ports like the above)

It's just the apache-tomcat-scalix communication that is not working now.

smtpd and imap seem to be working fine after upgrade.

This version *did* change the JRE from 1.5_0_11 to 1.5_0_13 or something like that...

Any ideas? The corporate webmail is down till this is fixed. Please help!

[Ruthiness]

Anyone out there? This is sort of urgent...

[Ruthiness]

Update

I am able to telnet hostname 8009 from the server.

I am NOT able to telnet localhost 8009 from the server.

Does that provide any clues?

I do not have selinux but I am using iptables.

[Ruthiness]

During the upgrade, I had to change the jre - anyone know how to check if something is wrong with the "default" java it is using?

Do I need to set a link to java to the new jre in the filesystem?

[Ruthiness]

Related to my issue?

catalina.2008-02-10.log:INFO: The Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/jre1.5.0_13/lib/amd64/server:/usr/java/jre1.5.0_13/lib/amd64:/usr/java/jre1.5.0_13/../lib/amd64:/usr/lib

[Ruthiness]

Here is something in the current catalina.out:

Feb 10, 2008 7:48:49 PM org.apache.jk.common.ChannelSocket init
INFO: Port busy 8009 java.net.BindException: Address already in use
Feb 10, 2008 7:48:49 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on mailpa.ourdomain.com/10.49.1.40:8010
Feb 10, 2008 7:48:49 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=1 time=0/40 config=null
Feb 10, 2008 7:48:49 PM org.apache.catalina.storeconfig.StoreLoader load
INFO: Find registry server-registry.xml at classpath resource
Feb 10, 2008 7:48:49 PM org.apache.catalina.startup.Catalina start

then later it says this after apparently connecting to 8009:

Feb 10, 2008 9:28:12 PM org.apache.catalina.core.StandardHost start
INFO: XML validation disabled
Feb 10, 2008 9:28:12 PM org.apache.jk.common.ChannelSocket init
INFO: JK: ajp13 listening on mailpa.ourdomain.com/10.49.1.40:8009
Feb 10, 2008 9:28:12 PM org.apache.jk.server.JkMain start
INFO: Jk running ID=0 time=0/51 config=null
Feb 10, 2008 9:28:12 PM org.apache.catalina.storeconfig.StoreLoader load
INFO: Find registry server-registry.xml at classpath resource
Feb 10, 2008 9:28:12 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 823 ms
Feb 10, 2008 9:28:22 PM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 65524
Feb 10, 2008 9:28:22 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:626)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:583)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:691)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
at java.lang.Thread.run(Unknown Source)
Feb 10, 2008 10:05:31 PM org.apache.jk.common.MsgAjp processHeader
SEVERE: BAD packet signature 65524
Feb 10, 2008 10:05:31 PM org.apache.jk.common.ChannelSocket processConnection
SEVERE: Error, processing connection
java.lang.IndexOutOfBoundsException
at java.io.BufferedInputStream.read(Unknown Source)
at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:626)
at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:583)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:691)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:895)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689

Any other ideas about which log files are relevant to this error?
(404 error accessing these urls:
https://mailpa.ourdomain.com/webmail
https://mailpa.ourdomain.com/sac
)

[Ruthiness]

I'll just keep talking to myself here I guess till someone has an idea

This is mod_jk.log

[Sun Feb 10 18:14:25 2008] [info] ajp_process_callback::jk_ajp_common.c (1384): Connection aborted or network problems
[Sun Feb 10 18:14:25 2008] [info] ajp_service::jk_ajp_common.c (1731): Receiving from tomcat failed, because of client error without r
ecovery in send loop 0
[Sun Feb 10 18:14:25 2008] [info] jk_handler::mod_jk.c (1964): Aborting connection for worker=mailpa
[Sun Feb 10 19:22:49 2008] [info] jk_open_socket::jk_connect.c (444): connect to 10.49.1.40:8009 failed with errno=111
[Sun Feb 10 19:22:49 2008] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (10.49.1.40:8009) with (err
no=111)
[Sun Feb 10 19:22:49 2008] [info] ajp_send_request::jk_ajp_common.c (1248): Error connecting to the Tomcat process.
[Sun Feb 10 19:22:49 2008] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attemp
t=1

[Ruthiness]

Is this a problem perhaps? catalina_home is different from the catalina_base

4 S root 9760 1 0 77 0 - 259894 - 22:53 pts/1 00:00:05 /usr/java/jre1.5.0_13/bin/java -server -Djava.net.preferIPv4Stack=true -Xms640m -Xmx640m -Dscalix.instance=/var/opt/scalix/ma -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/opt/scalix/ma/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/scalix-tomcat/common/endorsed -classpath /usr/java/jre1.5.0_13/lib/tools.jar:/opt/scalix-tomcat/bin/bootstrap.jar:/opt/scalix-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/var/opt/scalix/ma/tomcat -Dcatalina.home=/opt/scalix-tomcat -Djava.io.tmpdir=/var/opt/scalix/ma/tomcat/temp org.apache.catalina.startup.Bootstrap start

[Ruthiness]

Anyone?????

[Ruthiness]

I found we are able to bring up the login page for mobile as in

https://ourserver/m

We don't have it configured so we can't really login that way.. but why does the /m work but not any other resource such as /sac or /webmail?

Any ideas? I've been up all night working on this...

[Ruthiness]

I have webmail and sac working now and I am trying to figure out exactly WHICH knob I turned that did it.

I will try to back out and undo each minor change I made to break it again by flipping old copies of configs into place and restarting and then I will post here.

One interesting note - the sac page shows Scalix 11.2 version and the webmail page shows Scalix 11.3.. is that supposed to be that way?

One thing it MAY have been is I noticed a lack of a link in the tomcat conf directory structure that pointed to the FQDN name linking to localhost. There was only a link there for the ".local" domain that had a soft link to locahost. I added another link for the FQDN. Was that it? I won't know till I back the changes out one at a time.

All I know is, when it started working I was on the path to get the url http://localhost:8080 to work so I could see the default tomcat page get served to make sure tomcat was working. Right after I got that working, strangely enough so did webmail and sac pages.

[Ruthiness]

One thing I changed is the following (using a fake domain name of course)

/var/opt/scalix/ma/tomcat/conf/Catalina

[root@mailpa Catalina]# ls -al
total 12
drwxr-xr-x 3 root root 4096 Feb 11 22:04 .
drwxr-xr-x 3 root root 4096 Feb 11 22:29 ..
drwxr-xr-x 2 root root 4096 Feb 11 22:03 localhost
lrwxrwxrwx 1 root root 9 Feb 11 22:04 mailpa.OurDomain.com -> localhost <--- added
lrwxrwxrwx 1 root root 9 Jan 15 01:45 mailpa.OurDomain.local -> localhost

Is this relevant? *shrug*

another thing I did was leave the httpd and tomcat processes off for about 20 minutes or so while I tweaked several config files to enable the Catalina service on 8080.

Relevant? who knows?

I also had done an omcheck -s -d > fix_perms.sh at one point earlier this evening..

the last thing I did was the link above and also checking all instances of server.xml to enable the 8080 default service...

I hate not knowing but I also hate to mess with something that IS working....