Server crashing, suspect indexing

[mglenney]

I have a Scalix 11.0.2.1 installation on Fedora 5. My server keeps locking up and it looks like it's related to Scalix. Possibly Indexing. Server will run fine with Scalix off but with it running server lockups occur. Server is completely frozen. I have checked temperature and all is well. I haven't run a memcheck86 yet so memory may be an issue but I don't think so.

Problem seems to occur during indexing. I have alot of errors in the logs which I will list below but here is a new one. It occurs when running sxmkindex -r0:

[root@xdsmail logs]# sxmkindex -r0
sxadmin /xdsmail,xdsinc: 479
<username> /xdsmail,xdsinc: 227
sxqueryadmin /xdsmail,xdsinc: 0
Michael Glenney /xdsmail,xdsinc: 38661
<username> /xdsmail,xdsinc: 4917
<username> /xdsmail,xdsinc: 522
*** glibc detected *** sxmkindex: free(): invalid next size (normal): 0x093ead68 ***
======= Backtrace: =========
/lib/libc.so.6[0x1c5a68]
/lib/libc.so.6(__libc_free+0x78)[0x1c8f6f]
/opt/scalix/lib/libom_mdc.so(mdc_free+0x21)[0x15c8b1]
/opt/scalix/lib/libom_os.so(os_free+0x1d)[0x8a6b4d]
/opt/scalix/lib/libom_sis.so(sis_freeRefList+0x25)[0xa814f5]
sxmkindex[0x804a355]
sxmkindex[0x804a4c6]
sxmkindex[0x804ac3b]
/lib/libc.so.6(__libc_start_main+0xdc)[0x1774e4]
sxmkindex[0x8049621]
======= Memory map: ========
00110000-00113000 r-xp 00000000 fd:00 21234137 /opt/scalix/lib/libom_lng.so
00113000-00114000 rw-p 00003000 fd:00 21234137 /opt/scalix/lib/libom_lng.so
00114000-00118000 r-xp 00000000 fd:00 21234145 /opt/scalix/lib/libom_mes.so
00118000-00119000 rw-p 00004000 fd:00 21234145 /opt/scalix/lib/libom_mes.so
00119000-0011c000 rw-p 00119000 00:00 0
0011c000-00129000 r-xp 00000000 fd:00 21234090 /opt/scalix/lib/libom_ccs.so
00129000-0012a000 rw-p 0000d000 fd:00 21234090 /opt/scalix/lib/libom_ccs.so
0012a000-00131000 r-xp 00000000 fd:00 21234100 /opt/scalix/lib/libom_cust.so
00131000-00132000 rw-p 00006000 fd:00 21234100 /opt/scalix/lib/libom_cust.so
00132000-00135000 r-xp 00000000 fd:00 21234111 /opt/scalix/lib/libom_enc.so
00135000-00136000 rw-p 00002000 fd:00 21234111 /opt/scalix/lib/libom_enc.so
00136000-0013d000 r-xp 00000000 fd:00 21234159 /opt/scalix/lib/libom_nm.so
0013d000-0013e000 rw-p 00006000 fd:00 21234159 /opt/scalix/lib/libom_nm.so
0013e000-00140000 r-xp 00000000 fd:00 21234110 /opt/scalix/lib/libom_dstring.so
00140000-00141000 rw-p 00001000 fd:00 21234110 /opt/scalix/lib/libom_dstring.so
00141000-0014b000 r-xp 00000000 fd:00 21234129 /opt/scalix/lib/libom_im.so
0014b000-0014c000 rw-p 00009000 fd:00 21234129 /opt/scalix/lib/libom_im.so
0014c000-0014d000 r-xp 00000000 fd:00 21234135 /opt/scalix/lib/libom_licc.so
0014d000-0014e000 rw-p 00001000 fd:00 21234135 /opt/scalix/lib/libom_licc.so
0014e000-00151000 r-xp 00000000 fd:00 21234158 /opt/scalix/lib/libom_nfda.so
00151000-00152000 rw-p 00002000 fd:00 21234158 /opt/scalix/lib/libom_nfda.so
00152000-00153000 r-xp 00000000 fd:00 21234170 /opt/scalix/lib/libom_pool.so
00153000-00154000 rw-p 00000000 fd:00 21234170 /opt/scalix/lib/libom_pool.so
00155000-00161000 r-xp 00000000 fd:00 21234144 /opt/scalix/lib/libom_mdc.so
00161000-00162000 rw-p 0000c000 fd:00 21234144 /opt/scalix/lib/libom_mdc.so
00162000-0028f000 r-xp 00000000 fd:00 94011427 /lib/libc-2.4.so
0028f000-00291000 r--p 0012d000 fd:00 94011427 /lib/libc-2.4.so
00291000-00292000 rw-p 0012f000 fd:00 94011427 /lib/libc-2.4.so
00292000-00295000 rw-p 00292000 00:00 0
00295000-0029d000 r-xp 00000000 fd:00 21234157 /opt/scalix/lib/libom_nf.so
0029d000-0029e000 rw-p 00008000 fd:00 21234157 /opt/scalix/lib/libom_nf.so
0029e000-002a3000 r-xp 00000000 fd:00 21234160 /opt/scalix/lib/libom_nsl.so
002a3000-002a4000 rw-p 00004000 fd:00 21234160 /opt/scalix/lib/libom_nsl.so
002a4000-002a6000 r-xp 00000000 fd:00 21234197 /opt/scalix/lib/libom_tfo.so
002a6000-002a7000 rw-p 00001000 fd:00 21234197 /opt/scalix/lib/libom_tfo.so
002a7000-002ac000 r-xp 00000000 fd:00 21234196 /opt/scalix/lib/libom_tfl.so
002ac000-002ad000 rw-p 00004000 fd:00 21234196 /opt/scalix/lib/libom_tfl.so
002ad000-002b1000 r-xp 00000000 fd:00 21234109 /opt/scalix/lib/libom_drsc.so
002b1000-002b2000 rw-p 00003000 fd:00 21234109 /opt/scalix/lib/libom_drsc.so
002b2000-002b6000 r-xp 00000000 fd:00 21234139 /opt/scalix/lib/libom_mad.so
002b6000-002b7000 rw-p 00004000 fd:00 21234139 /opt/scalix/lib/libom_mad.so
002b7000-002ba000 r-xp 00000000 fd:00 21234099 /opt/scalix/lib/libom_culb.so
002ba000-002bb000 rw-p 00002000 fd:00 21234099 /opt/scalix/lib/libom_culb.so
002bb000-002bd000 r-xp 00000000 fd:00 21234128 /opt/scalix/lib/libom_hash.so
002bd000-002be000 rw-p 00001000 fd:00 21234128 /opt/scalix/lib/libom_hash.so
002be000-002c2000 r-xp 00000000 fd:00 21234172 /opt/scalix/lib/libom_pwdl.so
002c2000-002c3000 rw-p 00003000 fd:00 21234172 /opt/scalix/lib/libom_pwdl.so
002c3000-002c4000 r-xp 00000000 fd:00 21234104 /opt/scalix/lib/libom_date.so
002c4000-002c5000 rw-p 00000000 fd:00 21234104 /opt/scalix/lib/libom_date.so
002c5000-002de000 r-xp 00000000 fd:00 94011425 /lib/ld-2.4.so
002de000-002df000 r--p 00018000 fd:00 94011425 /lib/ld-2.4.so
002df000-002e0000 rw-p 00019000 fd:00 94011425 /lib/ld-2.4.so
002e0000-002ea000 r-xp 00000000 fd:00 21234208 /opt/scalix/lib/libom_usr.so
002ea000-002eb000 rw-p 00009000 fd:00 21234208 /opt/scalix/lib/libom_usr.so
002eb000-002ed000 r-xp 00000000 fd:00 21234136 /opt/scalix/lib/libom_lkf.so
002ed000-002ee000 rw-p 00001000 fd:00 21234136 /opt/scalix/lib/libom_lkf.so
002ee000-002f0000 r-xp 00000000 fd:00 21234187 /opt/scalix/lib/libom_sml.so
002f0000-002f1000 rw-p 00001000 fd:00 21234187 /opt/scalix/lib/libom_sml.so
002f1000-002f8000 r-xp 00000000 fd:00 21234207 /opt/scalix/lib/libom_uscv.so
002f8000-002f9000 rw-p 00006000 fd:00 21234207 /opt/scalix/lib/libom_uscv.so
002f9000-002fb000 r-xp 00000000 fd:00 21234215 /opt/scalix/lib/libom_xdse.so
002fb000-002fc000 rw-p 00001000 fd:00 21234215 /opt/scalix/lib/libom_xdse.so
002fc000-00303000 r-xp 00000000 fd:00 21234083 /opt/scalix/lib/libom_acl.so
00303000-00304000 rw-p 00007000 fd:00 21234083 /opt/scalix/lib/libom_acl.so
00304000-00305000 rw-p 00304000 00:00 0
00305000-0030c000 r-xp 00000000 fd:00 21234091 /opt/scalix/lib/libom_cdl.so
0030c000-0030d000 rw-p 00006000 fd:00 21234091 /opt/scalix/lib/libom_cdl.so
0030d000-00312000 r-xp 00000000 fd:00 94011534 /lib/libcrypt-2.4.so
00312000-00313000 r--p 00004000 fd:00 94011534 /lib/libcrypt-2.4.so
00313000-00314000 rw-p 00005000 fd:00 94011534 /lib/libcrypt-2.4.so
00314000-0033b000 rw-p 00314000 00:00 0
0033b000-00342000 r-xp 00000000 fd:00 21234085 /opt/scalix/lib/libom_akt.so
00342000-00343000 rw-p 00007000 fd:00 21234085 /opt/scalix/lib/libom_akt.so
00343000-00346000 r-xp 00000000 fd:00 21234122 /opt/scalix/lib/libom_fmem.so
00346000-00347000 rw-p 00003000 fd:00 21234122 /opt/scalix/lib/libom_fmem.so
00347000-00348000 rw-p 00347000 00:00 0
00348000-00350000 r-xp 00000000 fd:00 21234147 /opt/scalix/lib/libom_ml.so
00350000-00351000 rw-p 00007000 fd:00 21234147 /opt/scalix/lib/libom_ml.so
00351000-00379000 r-xp 00000000 fd:00 21234213 /opt/scalix/lib/libom_vista.so
00379000-0037d000 rw-p 00027000 fd:00 21234213 /opt/scalix/lib/libom_vista.so
0037d000-0038c000 r-xp 00000000 fd:00 21234092 /opt/scalix/lib/libom_cl.so
0038c000-0038d000 rw-p 0000e000 fd:00 21234092 /opt/scalix/lib/libom_cl.so
0038d000-003a3000 r-xp 00000000 fd:00 21234199 /opt/scalix/lib/libom_tnef.so
003a3000-003a4000 rw-p 00015000 fd:00 21234199 /opt/scalix/lib/libom_tnef.so
003a4000-003a8000 r-xp 00000000 fd:00 21234173 /opt/scalix/lib/libom_q.so
003a8000-003a9000 rw-p 00004000 fd:00 21234173 /opt/scalix/lib/libom_q.so
003a9000-003ae000 r-xp 00000000 fd:00 21234086 /opt/scalix/lib/libom_aud.so
003ae000-003af000 rw-p 00004000 fd:00 21234086 /opt/scalix/lib/libom_aud.so
003af000-003c1000 r-xp 00000000 fd:00 52990089 /usr/lib/libz.so.1.2.3
003c1000-003c2000 rw-p 00011000 fd:00 52990089 /usr/lib/libz.so.1.2.3
003c2000-003e7000 r-xp 00000000 fd:00 21234203 /opt/scalix/lib/libom_ufcv.so
003e7000-003e9000 rw-p 00024000 fd:00 21234203 /opt/scalix/lib/libom_ufcv.so
003ec000-003fd000 r-xp 00000000 fd:00 21234212 /opt/scalix/lib/libom_vi.so
003fd000-003fe000 rw-p 00010000 fd:00 21234212 /opt/scalix/lib/libom_vi.so
003fe000-00404000 r-xp 00000000 fd:00 21234150 /opt/scalix/lib/libom_mp.so
00404000-00405000 rw-p 00005000 fd:00 21234150 /opt/scalix/lib/libom_mp.so
00405000-0040b000 r-xp 00000000 fd:00 21234152 /opt/scalix/lib/libom_msg.so
0040b000-0040c000 rw-p 00005000 fd:00 21234152 /opt/scalix/lib/libom_msg.so
0040c000-00415000 r-xp 00000000 fd:00 94011432 /lib/libnss_files-2.4.so
00415000-00416000 r--p 00008000 fd:00 94011432 /lib/libnss_files-2.4.so
00416000-00417000 rw-p 00009000 fd:00 94011432 /lib/libnss_files-2.4.so
00417000-00419000 r-xp 00000000 fd:00 94011441 /lib/libdl-2.4.so
00419000-0041a000 r--p 00001000 fd:00 94011441 /lib/libdl-2.4.so
0041a000-0041b000 rw-p 00002000 fd:00 94011441 /lib/libdl-2.4.so
0041b000-00459000 r-xp 00000000 fd:00 52997444 /usr/lib/libical.so.0.0.0
00459000-00463000 rw-p 0003d000 fd:00 52997444 /usr/lib/libical.so.0.0.0
00463000-00464000 rw-p 00463000 00:00 0
00464000-00487000 r-xp 00000000 fd:00 94011431 /lib/libm-2.4.so
00487000-00488000 r--p 00022000 fd:00 94011431 /lib/libm-2.4.so
00488000-00489000 rw-p 00023000 fd:00 94011431 /lib/libm-2.4.so
004a2000-004a3000 r-xp 00000000 fd:00 21234123 /opt/scalix/lib/libom_fstr.so
004a3000-004a4000 rw-p 00000000 fd:00 21234123 /opt/scalix/lib/libom_fstr.so
004a9000-004aa000 r-xp 00000000 fd:00 53051576 /usr/lib/gconv/ISO8859-1.so
004aa000-004ac000 rw-p 00000000 fd:00 53051576 /usr/lib/gconv/ISO8859-1.so
004ba000-004ca000 r-xp 00000000 fd:00 94011443 /lib/libpthread-2.4.so
004ca000-004cb000 r--p 0000f000 fd:00 94011443 /lib/libpthread-2.4.so
004cb000-004cc000 rw-p 00010000 fd:00 94011443 /lib/libpthread-2.4.so
004cc000-004ce000 rw-p 004cc000 00:00 0
004d0000-004db000 r-xp 00000000 fd:00 94011433 /lib/libgcc_s-4.1.1-20070108.so.1
004db000-004dc000 rw-p 0000a000 fd:00 94011433 /lib/libgcc_s-4.1.1-20070108.so.1
004f7000-004f9000 r-xp 00000000 fd:00 21234125 /opt/scalix/lib/libom_gen.so
004f9000-004fa000 rw-p 00002000 fd:00 21234125 /opt/scalix/lib/libom_gen.so
00528000-00552000 r-xp 00000000 fd:00 21234103 /opt/scalix/lib/libom_da.so
00552000-00557000 rw-p 0002a000 fd:00 21234103 /opt/scalix/lib/libom_da.so
00557000-00558000 rw-p 00557000 00:00 0
0056d000-00585000 r-xp 00000000 fd:00 21234205 /opt/scalix/lib/libom_uni.so
00585000-00586000 rw-p 00017000 fd:00 21234205 /opt/scalix/lib/libom_uni.so
005a1000-005a5000 r-xp 00000000 fd:00 21234192 /opt/scalix/lib/libom_t61.so
005a5000-005a6000 rw-p 00004000 fd:00 21234192 /opt/scalix/lib/libom_t61.so
005b3000-005cf000 r-xp 00000000 fd:00 21234101 /opt/scalix/lib/libom_cvc.so
005cf000-00610000 rw-p 0001c000 fd:00 21234101 /opt/scalix/lib/libom_cvc.so
0064b000-00651000 r-xp 00000000 fd:00 21234183 /opt/scalix/lib/libom_sdl.so
00651000-00652000 rw-p 00005000 fd:00 21234183 /opt/scalix/lib/libom_sdl.so
00658000-006a4000 r-xp 00000000 fd:00 21234097 /opt/scalix/lib/libom_ct.so
006a4000-006a8000 rw-p 0004b000 fd:00 21234097 /opt/scalix/lib/libom_ct.so
006a8000-006a9000 rw-p 006a8000 00:00 0
006d6000-00707000 r-xp 00000000 fd:00 21234108 /opt/scalix/lib/libom_dr.so
00707000-00708000 rw-p 00030000 fd:00 21234108 /opt/scalix/lib/libom_dr.so
00708000-0070b000 rw-p 00708000 00:00 0
0074a000-00753000 r-xp 00000000 fd:00 21234131 /opt/scalix/lib/libom_isl.so
00753000-00754000 rw-p 00009000 fd:00 21234131 /opt/scalix/lib/libom_isl.so
007a4000-007a5000 r-xp 007a4000 00:00 0 [vdso]
007a5000-00836000 r-xp 00000000 fd:00 52991041 /usr/lib/libglib-2.0.so.0.1000.3
00836000-00837000 rw-p 00091000 fd:00 52991041 /usr/lib/libglib-2.0.so.0.1000.3
00885000-0089a000 r-xp 00000000 fd:00 21234204 /opt/scalix/lib/libom_ul.so
0089a000-0089b000 rw-p 00014000 fd:00 21234204 /opt/scalix/lib/libom_ul.so
0089b000-0089c000 rw-p 0089b000 00:00 0
0089e000-008ab000 r-xp 00000000 fd:00 21234164 /opt/scalix/lib/libom_os.so
008ab000-008ac000 rw-p 0000c000 fd:00 21234164 /opt/scalix/lib/libom_os.so
008f3000-00901000 r-xp 00000000 fd:00 21234191 /opt/scalix/lib/libom_str.so
00901000-00902000 rw-p 0000d000 fd:00 21234191 /opt/scalix/lib/libom_str.so
00948000-00956000 r-xp 00000000 fd:00 21234130 /opt/scalix/lib/libom_inet.so
00956000-00957000 rw-p 0000e000 fd:00 21234130 /opt/scalix/lib/libom_inet.so
00957000-00958000 rw-p 00957000 00:00 0
00966000-00967000 r-xp 00000000 fd:00 21234117 /opt/scalix/lib/libom_exual.so
00967000-00968000 rw-p 00001000 fd:00 21234117 /opt/scalix/lib/libom_exual.so
0099d000-009ad000 r-xp 00000000 fd:00 21234194 /opt/scalix/lib/libom_tf.so
009ad000-009ae000 rw-p 0000f000 fd:00 21234194 /opt/scalix/lib/libom_tf.so
009bd000-009bf000 r-xp 00000000 fd:00 21234189 /opt/scalix/lib/libom_ssn.so
009bf000-009c0000 rw-p 00001000 fd:00 21234189 /opt/scalix/lib/libom_ssn.so
009c9000-009cc000 r-xp 00000000 fd:00 21234126 /opt/scalix/lib/libom_go.so
009cc000-009cd000 rw-p 00002000 fd:00 21234126 /opt/scalix/lib/libom_go.so
00a14000-00a17000 r-xp 00000000 fd:00 21234177 /opt/scalix/lib/libom_rda.so
00a17000-00a18000 rw-p 00003000 fd:00 21234177 /opt/scalix/lib/libom_rda.so
00a47000-00a50000 r-xp 00000000 fd:00 21234181 /opt/scalix/lib/libom_rtfl.so
00a50000-00a52000 rw-p 00009000 fd:00 21234181 /opt/scalix/lib/libom_rtfl.so
00a52000-00a5a000 rw-p 00a52000 00:00 0
00a6b000-00a6f000 r-xp 00000000 fd:00 21234082 /opt/scalix/lib/libom_acf.so
00a6f000-00a70000 rw-p 00003000 fd:00 21234082 /opt/scalix/lib/libom_acf.so
00a80000-00a84000 r-xp 00000000 fd:00 21234186 /opt/scalix/lib/libom_sis.so
00a84000-00a85000 rw-p 00003000 fd:00 21234186 /opt/scalix/lib/libom_sis.so
00a9c000-00aae000 r-xp 00000000 fd:00 21234146 /opt/scalix/lib/libom_mim.so
00aae000-00aaf000 rw-p 00012000 fd:00 21234146 /opt/scalix/lib/libom_mim.so
00aaf000-00ab0000 rw-p 00aaf000 00:00 0
00ac2000-00ac7000 r-xp 00000000 fd:00 21234178 /opt/scalix/lib/libom_rdudp.so
00ac7000-00ac8000 rw-p 00004000 fd:00 21234178 /opt/scalix/lib/libom_rdudp.so
00ade000-00b1d000 r-xp 00000000 fd:00 21234140 /opt/scalix/lib/libom_mapi.so
00b1d000-00b1f000 rw-p 0003e000 fd:00 21234140 /opt/scalix/lib/libom_mapi.so
00b9b000-00baf000 r-xp 00000000 fd:00 21234087 /opt/scalix/lib/libom_bb.so
00baf000-00bb1000 rw-p 00013000 fd:00 21234087 /opt/scalix/lib/libom_bb.so
00bb1000-00bb2000 rw-p 00bb1000 00:00 0
00bc7000-00bcf000 r-xp 00000000 fd:00 21234151 /opt/scalix/lib/libom_mpl.so
00bcf000-00bd0000 rw-p 00007000 fd:00 21234151 /opt/scalix/lib/libom_mpl.so
00bee000-00bf4000 r-xp 00000000 fd:00 21234202 /opt/scalix/lib/libom_uald.so
00bf4000-00bf5000 rw-p 00006000 fd:00 21234202 /opt/scalix/lib/libom_uald.so
00c01000-00c05000 r-xp 00000000 fd:00 21234175 /opt/scalix/lib/libom_qml.so
00c05000-00c06000 rw-p 00003000 fd:00 21234175 /opt/scalix/lib/libom_qml.so
00ca5000-00cab000 r-xp 00000000 fd:00 21234114 /opt/scalix/lib/libom_ext.so
00cab000-00cac000 rw-p 00006000 fd:00 21234114 /opt/scalix/lib/libom_ext.so
00cac000-00cad000 rw-p 00cac000 00:00 0
00cad000-00dcc000 r-xp 00000000 fd:00 94011532 /lib/libcrypto.so.0.9.8a
00dcc000-00ddf000 rw-p 0011e000 fd:00 94011532 /lib/libcrypto.so.0.9.8a
00ddf000-00de2000 rw-p 00ddf000 00:00 0
00e43000-00e5d000 r-xp 00000000 fd:00 21234184 /opt/scalix/lib/libom_sfl.so
00e5d000-00e5e000 rw-p 00019000 fd:00 21234184 /opt/scalix/lib/libom_sfl.so
00e5e000-00e62000 rw-p 00e5e000 00:00 0
00e68000-00e78000 r-xp 00000000 fd:00 21234141 /opt/scalix/lib/libom_mapixml.so
00e78000-00e79000 rw-p 00010000 fd:00 21234141 /opt/scalix/lib/libom_mapixml.so
00e83000-00e8e000 r-xp 00000000 fd:00 21234180 /opt/scalix/lib/libom_rsl.so
00e8e000-00e93000 rw-p 0000b000 fd:00 21234180 /opt/scalix/lib/libom_rsl.so
00e93000-00e95000 rw-p 00e93000 00:00 0
00eca000-00ed4000 r-xp 00000000 fd:00 21234112 /opt/scalix/lib/libom_er.so
00ed4000-00ed5000 rw-p 00009000 fd:00 21234112 /opt/scalix/lib/libom_er.so
00f0a000-00f1f000 r-xp 00000000 fd:00 21234102 /opt/scalix/lib/libom_cvr.so
00f1f000-00f20000 rw-p 00014000 fd:00 21234102 /opt/scalix/lib/libom_cvr.so
00f20000-00f21000 rw-p 00f20000 00:00 0
00fa0000-00fa2000 r-xp 00000000 fd:00 21234163 /opt/scalix/lib/libom_orname.so
00fa2000-00fa3000 rw-p 00001000 fd:00 21234163 /opt/scalix/lib/libom_orname.so
00fd6000-00fd9000 r-xp 00000000 fd:00 21234124 /opt/scalix/lib/libom_gcl.so
00fd9000-00fda000 rw-p 00002000 fd:00 21234124 /opt/scalix/lib/libom_gcl.so
00fda000-010b9000 r-xp 00000000 fd:00 52985863 /usr/lib/libstdc++.so.6.0.8
010b9000-010bd000 r--p 000de000 fd:00 52985863 /usr/lib/libstdc++.so.6.0.8
010bd000-010be000 rw-p 000e2000 fd:00 52985863 /usr/lib/libstdc++.so.6.0.8
010be000-010c4000 rw-p 010be000 00:00 0
0617d000-062a1000 r-xp 00000000 fd:00 52990348 /usr/lib/libxml2.so.2.6.23
062a1000-062a9000 rw-p 00124000 fd:00 52990348 /usr/lib/libxml2.so.2.6.23
062a9000-062aa000 rw-p 062a9000 00:00 0
08048000-0804c000 r-xp 00000000 fd:00 21233946 /opt/scalix/bin/sxmkindex
0804c000-0804d000 rw-p 00003000 fd:00 21233946 /opt/scalix/bin/sxmkindex
093ba000-093f9000 rw-p 093ba000 00:00 0
b7600000-b7621000 rw-p b7600000 00:00 0
b7621000-b7700000 ---p b7621000 00:00 0
b7793000-b7b26000 r--s 00000000 00:08 163845 /SYSV00000000 (deleted)
b7b26000-b7b27000 rw-p b7b26000 00:00 0
b7b28000-b7b2f000 r--s 00000000 fd:00 53051653 /usr/lib/gconv/gconv-modules.cache
b7b2f000-b7d70000 r--s 00000000 00:08 294921 /SYSV00000000 (deleted)
b7d70000-b7d72000 r--s 00000000 00:08 262152 /SYSV00000000 (deleted)
b7d72000-b7d78000 r--s 00000000 00:08 65538 /SYSV00000000 (deleted)
b7d78000-b7f78000 r--p 00000000 fd:00 52989090 /usr/lib/locale/locale-archive
b7f78000-b7f8b000 rw-p b7f78000 00:00 0
b7f8d000-b7f8e000 rw-p b7f8d000 00:00 0
b7f8e000-b7f91000 rw-s 00000000 00:08 32769 /SYSV00000000 (deleted)
b7f91000-b7f94000 rw-p b7f91000 00:00 0
bf8ad000-bf8c3000 rw-p bf8ad000 00:00 0 [stack]


I have also been getting alot of these in the fatal log:

ERROR Indexer (Indexer ) Tue Oct 30 06:29:36 2007
[OM 16082] Directory is unavailable at this time. Please retry later.

Pid of logging process: 3378
Current errno value: 4


and this:

SERIOUS ERROR Administration(Scalix Admin C) Tue Oct 30 06:12:28 2007
[OM 10270] Process about to terminate due to error.
Signal (Abort) trapped by process 3955
Procedure trace follows:
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
-> ct_CloseCtner 3002
-> sfl_CloseItem
<- sfl_CloseItem
<- ct_CloseCtner 3002
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
-> ct_CloseCtner 3002
-> sfl_CloseItem
<- sfl_CloseItem
<- ct_CloseCtner 3002
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
Pid of logging process: 3955


SERIOUS ERROR Administration(Scalix Admin C) Tue Oct 30 06:12:28 2007
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xb3)[0x636dc3]
/opt/scalix/lib/libom_er.so[0x6370ac]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x63723f]
[0xc94420]
/lib/libc.so.6(abort+0x101)[0x4f74f1]
/lib/libc.so.6[0x52a53b]
/lib/libc.so.6[0x531a68]
/lib/libc.so.6(__libc_free+0x78)[0x534f6f]
/opt/scalix/lib/libom_mdc.so(mdc_free+0x21)[0x11b8b1]
/opt/scalix/lib/libom_os.so(os_free+0x1d)[0xf18b4d]
/opt/scalix/lib/libom_sis.so(sis_freeRefList+0x25)[0x12a4f5]
sxmkindex[0x804a355]
sxmkindex[0x804a4c6]
sxmkindex[0x804ac3b]
/lib/libc.so.6(__libc_start_main+0xdc)[0x4e34e4]
sxmkindex[0x8049621]
Pid of logging process: 3955


Oh. just locked up again. I'm about to lose it and so is my company. Any suggestions?

MG

[mglenney]

Update - the big nasty "glibc detected" error is gone now. Each user it stops on I issue a

Code: Select all

ommodu -o <username> --index none

and then

Code: Select all

ommodu -o <username> --index auto

I then rerun 'sxmkindex -ro' and wait for it to fail. Whatever user it fails on I do it all over again. I'm waiting for a bit now because I have a ton of index.browse processes running which get launched when I run sxmkindex.

I've been tailing the fatal log during all this and have 2 new errors:

SERIOUS ERROR Administration(Scalix Admin C) Tue Oct 30 07:10:17 2007
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xb3)[0xb79dc3]
/opt/scalix/lib/libom_er.so[0xb7a0ac]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0xb7a23f]
[0x5be420]
/opt/scalix/lib/libom_ct.so(ct_NotifyIndexer+0x14e)[0x133fad]
[0x31646133]
Pid of logging process: 3685


ERROR Browser (Mime_Browser ) Tue Oct 30 07:21:14 2007
[OM.MIME 4000] Browser Args :mime.browse -l C -c -o /var/opt/scalix/xl/s/temp/mime_cache/mimehHIHUq 0018cdf9272d9444
Pid of logging process: 3472


I'll keep updating. Server uptime currently 23 mins. knock on wood.

MG

[kanderson]

Upgrade the server to 11.2. Do that tonight.

The 11.0.x line was... err, released prematurely, shall we say.

11.0.4 was OK, and 11.2 is excellent. On the client side, 11.2 still has a few little issues. It mostly works fine, but there are differences depending on the overall setup of a site. That mainly affects smartcache. But with 11.2, smartcache is a wicked tool that shows it's value. It's distantly ahead of anything announced by any competitors, but in my opinion, it should be off by default. When it works, which is 90% of the time, it's a tool that's ahead of it's time, It literally puts Scalix into a different class of service. On the other hand, when Smart Cache screws up, it's REALLY frustrating, and exceedingly difficult for the lay person to troubleshoot.

I suspect that upgrading the server to 11.2 will resolve your problems to a large degree. If nothing else, it'll move you to a platform that's known to be stable and production ready.

Kev.

[mglenney]

I appreciate your response Kev.

My plan is to upgrade to 11.0.4 or 11.1. I have found plenty of information about the stability of 11.0.4 but have yet to research 11.1. I am running on Fedora 5 which isn't supported by 11.2 so that's why I hadn't considered that.

There is history behind that. We had 11.0.2.1 running on FC5 as a virtual machine and were getting horrible performance. I set out to install FC7 and 11.2 on a new server (non vm) and upgrade and the problems we had with that had us down for almost 20 hours. 5 of those hours were with Scalix support and developers who also could not figure out why it woulnd't work (Not saying anything bad about those folks. They were awesome).

Because of that I ended up installing FC5 on the new box and sticking with 11.0.2.1. This new install is what is giving me fits with the lockups.

I've seen a number of indexer bugs fixed in 11.0.3 and 11.0.4 which is why I want to try 11.0.4. That, combined with FC5 support and good reviews from the community leads me to believe that is the safest path. Eventually I will need to move past this and upgrade but I will need at least 6-12 months of stable mail service before I can convince my boss that he should let me do that.

What are your (and anyone else reading this) feelings about 11.1 vs 11.0.4?

Thanks,

MG

[kanderson]

I'd rate them about equal. In either case, use the 11.2 connector.

Also, dump Fedora as quickly as possible. If you need a zero dollar OS, move to Cent.

As you have seen, Scalix runs on Fedora only for testing purposes, not for production.

Kev.

[techsharp]

MG,

11.1 was much better then 11.0.4 at least for our company. We had a rough time with 11.0.4 and things were better with the release of 11.1.

I am in the same boat as you - we have a FC5 xen image running our mail server and I wanted to do a direct upgrade to FC7 - turned out it was not going to work out so we are rebuilding the server with a fresh Install of FC7 and no Xen this time around. We will be up and running w/ 11.2 which I am looking forward to.

Kev has helped us a ton and am looking forward to this upgrade!

If you can - try to get 11.2 running preferably on CentOS since that has a longer support cycle then FC.

[mglenney]

Thanks again for the tips.

I agree about Fedora/CentOS. The only reason we went with Fedora is so we can get support from Scalix when we need it. We run CentOS for just about everything else.

Sounds like the 11.2 connector will work with 11.1. If that's the case I will certainly use it and if it performs as you say I'm sure I'll be very happy with it. The connector we're using now is slow and crashes everytime we close Outlook. Very frustrating.

MG

[kanderson]

You'll get as much support for Cent as you will for Fedora. Both are officially unsupported except for testing.

Plus, you're WAY more likely to avoid upgrade problems, meaning you won't need the support in the first place.

For the connector. You can upgrade now. When you upgrade, create a new cache, or for people who don't need the cache at all (desktops) I'd recommend disabling it, if for no reason other than to make sure you avoid those types of problems. Don't get me wrong, smartcache is a good thing, and it's getting better all the time, but there are still occasional hickups with it, and if you've lost credibility already, then you can eliminate the possibility of those problems by not using smartcache.

Kev.

[mglenney]

Well, I performed the upgrade from 11.0.2.1 to 11.1 and it was a pretty smooth transition except for my ual.remote file being overwritten which broke imap kerberos auth.

What the upgrade didn't do was fix my problem. Server crashed soon after. I still haven't had a chance to head to the colo and run some tests on the hardware but, without a doubt, the server will not lock up unless scalix is running so I have to believe that Scalix is either causing or triggering it.

There wasn't anything in the log at the time of the crash but soon after I power cycled and restarted scalix I had this:

SERIOUS ERROR Administration(Scalix Admin C) 10.31.07 01:46:26
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 3112
Procedure trace follows:
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- ul_getBySXUID
<- sis_create
-> sis_create
-> ul_getBySXUID
<- ul_getBySXUID
<- sis_create
-> sis_connect
<- sis_connect
-> sis_reflist
-> sis_disconnect
<- sis_disconnect
<- sis_reflist
-> sis_free
-> sis_free


SERIOUS ERROR Administration(Scalix Admin C) 10.31.07 01:46:26
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xb3)[0x6fadc3]
/opt/scalix/lib/libom_er.so[0x6fb0ac]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x6fb23f]
[0x39b420]
/lib/libc.so.6[0x8ded7b]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded0f]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6[0x8ded29]
/lib/libc.so.6(qsort+0x7f)[0x8deeda]
/opt/scalix/bin/sxmkindex[0x804975c]
/opt/scalix/bin/sxmkindex[0x804a421]
/opt/scalix/bin/sxmkindex[0x804a4c6]
/opt/scalix/bin/sxmkindex[0x804ac3b]
/lib/libc.so.6(__libc_start_main+0xdc)[0x8ca4e4]

[root@xdsmail logs]#

More sis and sxmkindex stuff. Can't prove this is the cause but maybe if I solve that problem the lockups will go away and I can get some sleep.

Aside from upgrading the OS and then to 11.2, does anyone have any other suggestions?

Thanks,

Mike

[kanderson]

Can you please post the output from:

tail -n 5 /var/opt/scalix/??/postgres/pgstart.log
and
/etc/init.d/scalix-postgres status
and
rpm -qa|grep -i java

It's just weird that this is bringing down your server. I've followed Scalix through a LOT of issues with the early 11 releases, including crashed on SLES9, but I don't remember seeing crashes due to search or indexing...

I'll be honest, I'm reaching here.

Kev.

[mglenney]

Hey, reach away. I appreciate you taking any time at all to help out. I'm completely stumped at this point so any help is good help. I will say I've got 9 hours of uptime on the server at this point. Made it through the early morning rush. I'm definitely happy about that. Here's what you asked for (I'm assuming you meant pgstartup.log):

Code: Select all

[root@xdsmail ~]# tail -n 5 /var/opt/scalix/xl/postgres/pgstartup.log
LOG:  logger shutting down
LOG:  logger shutting down
LOG:  logger shutting down
LOG:  logger shutting down
LOG:  logger shutting down

[root@xdsmail ~]# /etc/init.d/scalix-postgres status
postmaster (pid 30619 30618 30617 30616 30615 2929 2928 2927 2925 2923) is running...

[root@xdsmail ~]# rpm -qa | grep -i java
java-1.4.2-gcj-compat-1.4.2.0-40jpp_83rh.4


MG

[mglenney]

Well, after almost 24 hours of uptime it happened again. Here is what was logged right before the crash:

ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 3400] Invalid direct ref found
<- sfl_CloseItem
<- ct_CloseCtner 3002
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]


ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 3400] Invalid direct ref found
<- ct_ReadNextContentRec
-> ct_ReadNextContentRec
<- ct_ReadNextContentRec
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]


ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 3400] Invalid direct ref found
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]


ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 3400] Invalid direct ref found
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]


ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 3400] Invalid direct ref found
-> ct_NotifyIndexer
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]


ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 3400] Invalid direct ref found
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c:41[3,3400]

That last one repeats a lot of times. At the end of those I get:

SERIOUS ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 10270] Process about to terminate due to error.
Signal (Abort) trapped by process 3242
Procedure trace follows:
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c
<- ct_NotifyIndexer
-> ct_NotifyIndexer
<- /build/11.1.0/src/lib/ct/ct_indexnf.c
<- ct_NotifyIndexer


SERIOUS ERROR Administration(Scalix Admin C) 10.31.07 23:57:56
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xb3)[0x88edc3]
/opt/scalix/lib/libom_er.so[0x88f0ac]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0x88f23f]
[0x850420]
/lib/libc.so.6(abort+0x101)[0x30b4f1]
/lib/libc.so.6[0x33e53b]
/lib/libc.so.6[0x345a68]
/lib/libc.so.6(__libc_free+0x78)[0x348f6f]
/opt/scalix/lib/libom_mdc.so(mdc_free+0x21)[0x8a68b1]
/opt/scalix/lib/libom_os.so(os_free+0x1d)[0x19db4d]
/opt/scalix/lib/libom_sis.so(sis_freeRefList+0x25)[0x16f4f5]
/opt/scalix/bin/sxmkindex[0x804a355]
/opt/scalix/bin/sxmkindex[0x804a4c6]
/opt/scalix/bin/sxmkindex[0x804ac3b]
/lib/libc.so.6(__libc_start_main+0xdc)[0x2f74e4]
/opt/scalix/bin/sxmkindex[0x8049621]

[mglenney]

Wanted to provide an update for those who have helped me and for those who may be troubleshooting the same problem.

As of right now server has 4 days 9 hours uptime. Very promising. Upgrading to 11.1 might have solved the problem. I don't have anything strange in my logs and the indexer has been running and causing no problems. I'll update if I see anything else strange or have another crash but for now I consider this issue closed.

Thanks for the responses and for taking the time to help me troubleshoot this.

MG