SAC - login not responsive - errors in logs

[William]

Hi,

tried logging into SAC today and the popup windows with the login form appears, but upon clicking the login button there is nothing happening.

loooking in the logs the following is seen:

/var/log/httpd/mod_jk.log

[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:43 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:43 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:43 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:43 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:44 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:44 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:44 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:44 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:44 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:44 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:44 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:44 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:44 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:44 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:44 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:44 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:44 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:44 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:44 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:44 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:44 2007] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (955): Tomcat has forced a connection close for socket 16
[Wed Sep 12 14:30:44 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:44 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:44 2007] [error] ajp_get_reply::jk_ajp_common.c (1503): Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Sep 12 14:30:44 2007] [error] ajp_service::jk_ajp_common.c (1715): receiving reply from tomcat failed without recovery in send loop 0
[Wed Sep 12 14:30:44 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:44 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:30:44 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Sep 12 14:31:14 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Sep 12 14:31:14 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Sep 12 14:31:14 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Sep 12 14:31:14 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Sep 12 14:31:14 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1

I have tried a
service scalix-tomcat restart

but nothing seems to be fixed.
Looking at the previous mod_jk logs many of these same error messages are seen, yet back then the SAC worked.

eg

/var/log/httpd/mod_jk.19.log

[Wed Mar 28 10:25:55 2007] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=mail failed
[Wed Mar 28 10:25:55 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Mar 28 10:26:05 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Mar 28 10:26:05 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Mar 28 10:26:05 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Mar 28 10:26:05 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Mar 28 10:26:05 2007] [info] jk_open_socket::jk_connect.c (444): connect to 192.168.102.1:8009 failed with errno=111
[Wed Mar 28 10:26:05 2007] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (192.168.102.1:8009) with (errno=111)
[Wed Mar 28 10:26:05 2007] [info] ajp_send_request::jk_ajp_common.c (1248): Error connecting to the Tomcat process.
[Wed Mar 28 10:26:05 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=2
[Wed Mar 28 10:26:05 2007] [info] jk_open_socket::jk_connect.c (444): connect to 192.168.102.1:8009 failed with errno=111
[Wed Mar 28 10:26:05 2007] [info] ajp_connect_to_endpoint::jk_ajp_common.c (889): Failed opening socket to (192.168.102.1:8009) with (errno=111)
[Wed Mar 28 10:26:05 2007] [info] ajp_send_request::jk_ajp_common.c (1248): Error connecting to the Tomcat process.
[Wed Mar 28 10:26:05 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=3
[Wed Mar 28 10:26:05 2007] [error] ajp_service::jk_ajp_common.c (1758): Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=mail failed
[Wed Mar 28 10:26:05 2007] [info] jk_handler::mod_jk.c (1971): Service error=0 for worker=mail
[Wed Mar 28 10:26:15 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Mar 28 10:26:15 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Mar 28 10:26:15 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Mar 28 10:26:15 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Mar 28 10:26:16 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Mar 28 10:26:16 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Mar 28 10:26:16 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Mar 28 10:26:16 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Mar 28 10:26:16 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Mar 28 10:26:16 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Mar 28 10:26:16 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Mar 28 10:26:16 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1
[Wed Mar 28 10:26:48 2007] [info] ajp_send_request::jk_ajp_common.c (1178): Socket 16 is not connected any more (errno=-1)
[Wed Mar 28 10:26:48 2007] [info] ajp_send_request::jk_ajp_common.c (1202): Error sending request. Will try another pooled connection
[Wed Mar 28 10:26:48 2007] [info] ajp_send_request::jk_ajp_common.c (1225): All endpoints are disconnected or dead
[Wed Mar 28 10:26:48 2007] [info] ajp_service::jk_ajp_common.c (1749): Sending request to tomcat failed, recoverable operation attempt=1

What do I need to restart to get the SAC login functioning again?

[William]

Code: Select all

 omstat -a
PC Monitor                    Started        NON-STOP       0
Directory Relay Server        Started        01.08.07
Notification Server           Started        01.08.07       0
Shared memory daemon          Started        NON-STOP
Notification Monitor          Started        NON-STOP
Session Monitor               Started        NON-STOP
Indexer                       Started        NON-STOP
Stats Daemon                  Started        NON-STOP
Container Access Monitor      Started        NON-STOP
Item Structure Server         Stopped        09.02.07
Database Monitor              Started        01.08.07
Licence Monitor Daemon        Started        NON-STOP
LDAP Daemon                   Started        01.08.07
Queue Manager                 Started        NON-STOP
Item Delete Daemon            Started        NON-STOP
IMAP Server Daemon            Started        01.08.07
SMTP Relay                    Started        01.08.07
Mime Browser Controller       Started        01.08.07
Event Server                  Started        01.08.07
omstat -s
Service Router                Started        01.08.07       0
Local Delivery                Started        01.08.07       0
Internet Mail Gateway         Started        01.08.07       1
Local Client Interface        Enabled        25.04.07       0
Remote Client Interface       Enabled        25.04.07       18
Test Server                   Started        01.08.07       0
Request Server                Started        01.08.07       0
Print Server                  Started        01.08.07       0
Bulletin Board Server         Started        01.08.07       0
Background Search Service     Started        01.08.07       0
CDA Server                    Started        01.08.07       0
POP3 interface                Started        01.08.07       0
Omscan Server                 Started        01.08.07       0
Archiver                      Started        01.08.07       0

I restart httpd and this did not solve the issue.

I ran omrc:

Code: Select all

omrc : Scalix appears to be running.
If you wish to restart Scalix cleanly, run "omshut" or
kill the following processes, then run "omrc".

UID        PID  PPID  C STIME TTY      STAT   TIME CMD
scalix    3831     1  0 Aug01 ?        Ss     0:04 omsessd
scalix    3834     1  0 Aug01 ?        Ss     1:27 omctmon
scalix    3839     1  0 Aug01 ?        Ss     0:00 omsmdm
scalix    3842     1  0 Aug01 ?        Ss     0:17 notif.mon
scalix    3845     1  0 Aug01 ?        Ss     5:45 indexer
root      3846  3845  0 Aug01 ?        S      0:19 index.catcher
scalix    3849     1  0 Aug01 ?        Ss     0:00 sxstatsd
scalix    3856     1  0 Aug01 ?        Ss     0:05 queue.manager
scalix    3864     1  0 Aug01 ?        Ss     0:19 omlicmon
scalix    3872     1  0 Aug01 ?        Ss     0:13 idel.server
root      3892     1  0 Aug01 ?        Ss     0:01 advmail.sckd
scalix    3918     1  0 Aug01 ?        SNs    0:00 omdrs
scalix    3920     1  0 Aug01 ?        Ss     0:00 omnsdm
scalix    3921  3920  0 Aug01 ?        S      0:00 /opt/scalix/bin/omnssck
root      3923     1  0 Aug01 ?        Ss     2:00 omdbmon -p 30
scalix    3927     1  0 Aug01 ?        Ss     0:26 mime.control
scalix    3930     1  0 Aug01 ?        SNs    0:00 sxeventserver
root      3931     1  0 Aug01 ?        Ss     0:00 omsmtpd
root      3933     1  0 Aug01 ?        S      0:00 omsmtpd
root      3935     1  0 Aug01 ?        S      0:00 omsmtpd
root      3937     1  0 Aug01 ?        S      0:00 omsmtpd
root      3943     1  0 Aug01 ?        Sl     3:07 omslapd
root      3951     1  0 Aug01 ?        Ss     0:01 in.imap41d -d
scalix    3978     1  0 Aug01 ?        Ss     0:33 unix.out
scalix    3981     1  0 Aug01 ?        SNs    0:00 print.server
scalix    3982     1  0 Aug01 ?        SNs    0:00 req.server
scalix    3983     1  0 Aug01 ?        Ss     0:02 bb.server
scalix    3987     1  0 Aug01 ?        SNs    0:56 service.router
scalix    3988  3987  0 Aug01 ?        SN     0:00 defer.manager -D
scalix    3989     1  0 Aug01 ?        SNs    0:00 test.server
scalix    3990     1  0 Aug01 ?        Ss     0:00 search.server
scalix    3991     1  0 Aug01 ?        SNs    0:00 cda.server
scalix    3992     1  0 Aug01 ?        SNs    0:00 error.manager
root      3993     1  0 Aug01 ?        Ssl    0:00 pop3.server
scalix    3995     1  0 Aug01 ?        SNs    8:11 local.delivery
scalix    3997     1  0 Aug01 ?        SNs    0:00 archiver
root      3998     1  0 Aug01 ?        SNs    5:37 omscan.server -F ~/sys/omscan.cfg



Should I restart 'scalix' with the following?:
omshut
omrc

[William]

It seems another program was causing problems for scalix, namely rpm.

this was messing up scalix also. I think scalix checks its own package versions now and then and as rpm was hung, scalix was hung.

got the solution from here:
http://rootservermanagement.com/blog/index.php?paged=2
# rm -f /var/lib/rpm/__db.00*
# killall -9 rpmq rpm
# nice -n 19 rpm --rebuilddb

and restarting scalix-postgres seems to have fixed the issue.