Search stopped working in 11.0.3

eyalm · Postby **eyalm** » Sun Apr 22, 2007 8:21 pm

I upgraded to 11.0.3 this morning, and now search stopped working from SWA.
When i do a search I get 'No search results returned', but when I run sxsearch 'vpn' eyalm I get a bunch of results

When I start scalix I get:

SERIOUS ERROR Administration(Scalix Admin C) Sun Apr 22 13:47:35 2007
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 6365
Procedure trace follows:
<- dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
-> dr_ACICheckReadPerm
<- dr_ACICheckReadPerm
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- ul_getBySXUID
<- sis_create
-> sis_create
-> ul_getBySXUID
<- ul_getBySXUID
<- sis_create
-> sis_connect
<- sis_connect
-> sis_reflist
Pid of logging process: 6365

SERIOUS ERROR Administration(Scalix Admin C) Sun Apr 22 13:47:35 2007
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0xf7f84ee6]
/opt/scalix/lib/libom_er.so[0xf7f851e6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0xf7f8538f]
[0xffffe500]
/opt/scalix/bin/sxmkindex[0x8049719]
/opt/scalix/bin/sxmkindex[0x804a457]
/opt/scalix/bin/sxmkindex[0x804a4fa]
/opt/scalix/bin/sxmkindex[0x804ac65]
/lib/tls/libc.so.6(__libc_start_main+0xd3)[0xf7defde3]
/opt/scalix/bin/sxmkindex[0x8049611]
Pid of logging process: 6365

SERIOUS ERROR Administration(Scalix Admin C) Sun Apr 22 13:47:53 2007
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 2612
Procedure trace follows:
<- dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
-> dr_ACICheckReadPerm
<- dr_ACICheckReadPerm
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- ul_getBySXUID
<- sis_create
-> sis_create
-> ul_getBySXUID
<- ul_getBySXUID
<- sis_create
-> sis_connect
<- sis_connect
-> sis_reflist
Pid of logging process: 2612

SERIOUS ERROR Administration(Scalix Admin C) Sun Apr 22 13:47:53 2007
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0xf7f84ee6]
/opt/scalix/lib/libom_er.so[0xf7f851e6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0xf7f8538f]
[0xffffe500]
/opt/scalix/bin/sxmkindex[0x8049719]
/opt/scalix/bin/sxmkindex[0x804a457]
/opt/scalix/bin/sxmkindex[0x804a4fa]
/opt/scalix/bin/sxmkindex[0x804ac65]
/lib/tls/libc.so.6(__libc_start_main+0xd3)[0xf7defde3]
/opt/scalix/bin/sxmkindex[0x8049611]
Pid of logging process: 2612

Any ideas?

Postby **ScalixSupport** » Tue Apr 24, 2007 2:30 am

Hi!

Can you run the command "sxmkindex -r 00:00:00", see if search works now?

Thanks,
Subir

eyalm · Postby **eyalm** » Sat May 05, 2007 12:23 pm

I just tried it and still same error and no results when searching from swa:

[root@scalix ~]# sxmkindex -r 00:00:00
Cardon Administrator /scalix,cardon: 112
sxqueryadmin /scalix,cardon: 0
user1 /cardon: 242
user2 /cardon: 41
user3 /cardon: 1302
user4 /cardon: 2592
user5 /cardon: 20
user6/cardon: 2385
Eyal Marantenboim /cardon: [root@scalix ~]#

omshowlog -p5:

SERIOUS ERROR Administration(Scalix Admin C) 05.05.07 11:21:31
[OM 10270] Process about to terminate due to error.
Signal (Segmentation Violation) trapped by process 9406
Procedure trace follows:
<- dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
-> dr_ACICheckReadPerm
<- dr_ACICheckReadPerm
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- ul_getBySXUID
<- sis_create
-> sis_create
-> ul_getBySXUID
<- ul_getBySXUID
<- sis_create
-> sis_connect
<- sis_connect
-> sis_reflist

SERIOUS ERROR Administration(Scalix Admin C) 05.05.07 11:21:31
[OM 10272] BACKTRACE:
/opt/scalix/lib/libom_er.so(er_add_backtrace+0xc6)[0xf7f99ee6]
/opt/scalix/lib/libom_er.so[0xf7f9a1e6]
/opt/scalix/lib/libom_er.so(er_DumpProcAndExit+0x1f)[0xf7f9a38f]
[0xffffe500]
sxmkindex[0x8049719]
sxmkindex[0x804a457]
sxmkindex[0x804a4fa]
sxmkindex[0x804ac65]
/lib/tls/libc.so.6(__libc_start_main+0xd3)[0xb80de3]
sxmkindex[0x8049611]

Postby **ScalixSupport** » Mon May 07, 2007 7:29 am

Hi!

What result does omshowlog -l 3 give? Check if you find any error in SIS related log files
scalix-sis-indexer.log and
scalix-sis-search.log
located in /var/opt/scalix/??/tomcat/logs?

Thanks,
Subir

eyalm · Postby **eyalm** » Mon May 07, 2007 10:25 am

scalix-sis-indexer.log has this from last time I restarted scalix:

2007-04-29 10:36:15,494 INFO [main] [SISConfig.init:46] Starting Scalix Search and Index Service (SIS) version 11.0.3.31
2007-04-29 10:36:15,548 INFO [main] [SISConfig.loadConfigFromFile:131] SIS Indexer configuration loaded from file /var/opt/scalix/s
x/sis/sis.properties
2007-04-29 10:36:15,559 INFO [main] [SISConfig.init:89] Index root is /var/opt/scalix/sx/indexes
2007-04-29 10:36:15,671 INFO [main] [SISConfig.load:32] SIS initialized and ready for indexing

And then a bunch of:

2007-04-29 10:36:30,690 WARN [TP-Processor96] [IndexUtil.isIndexIDInContentIndex:211] There are 2 documents matching indexID 210101
0a-4486d87f-45f1cab6-1d2921 in content index
2007-04-29 10:36:30,955 WARN [TP-Processor94] [IndexUtil.isIndexIDInContentIndex:211] There are 2 documents matching indexID 210101
0a-4486d87f-45f565d6-1f4a4c in content index
2007-04-29 10:36:33,788 WARN [QueueManager] [BatchUpdater.performContentDeletes:409] No documents deleted for DeleteContentInfo{Ind
exInfo{userID='5fa72000f78d6844-33.1.1.01'}, indexID='2101010a-4486d87f-45f566a6-1f4c45'}
2007-04-29 10:36:33,829 INFO [QueueManager] [BatchUpdater.processMods:219] User 5fa72000f78d6844-33.1.1.01: added 0 content, added
0 refs, deleted 1 content, deleted 1 refs in 102 ms
2007-04-29 10:36:33,993 WARN [TP-Processor97] [IndexUtil.isIndexIDInContentIndex:211] There are 2 documents matching indexID 210101
0a-4486d87f-45f1cab6-1d2921 in content index
2007-04-29 10:36:34,060 INFO [QueueManager] [BatchUpdater.processMods:219] User 64c72000f78d6844-33.1.1.01: added 0 content, added
1 refs, deleted 0 content, deleted 3 refs in 229 ms

scalix-sis-search.log:

2007-05-05 11:21:31,674 ERROR [TP-Processor78] [WebUtil.errorResponse:61] Failed to get searcher
java.lang.IllegalArgumentException: attempt to access a deleted document
at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:282)
at org.apache.lucene.index.MultiReader.document(MultiReader.java:104)
at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:83)
at com.scalix.sis.web.InternalAdminServlet.doReferenceList(InternalAdminServlet.java:160)
at com.scalix.sis.web.InternalAdminServlet.doGet(InternalAdminServlet.java:116)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Unknown Source)

Postby **ScalixSupport** » Tue May 08, 2007 3:45 am

Hi!

With reference to post:
viewtopic.php?t=6819
Try to set ulimit using the command below:

Code: Select all

ulimit -n 50000

if this does not help, try to stop and restart Indexer using the commands below:
Stop the indexer

Code: Select all

/opt/scalix/bin/omshutdm -s 35 -p -d 0

Start the indexer

Code: Select all

/opt/scalix/bin/indexer

See if this helps.

Thanks,
Subir

eyalm · Postby **eyalm** » Wed May 09, 2007 9:08 pm

Did what you said.
omshowlog -l3 output:

ERROR Indexer (Indexer ) 05.09.07 20:04:59
[OM 16082] Directory is unavailable at this time. Please retry later.

Current errno value: 4
-> sis_free
-> removeRequest
<- removeRequest
-> weedEntry
<- weedEntry
-> sis_create
-> ul_getBySXUID
-> dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- /build/11.0.3/src/lib/ombase/vi/vi_locks.c:352[3,16082]
<- /build/11.0.3/src/lib/ombase/vi/vi_locks.c:201[3,16082]
<- /build/11.0.3/src/lib/ombase/dr/dr_search.c:470[3,16082]

ERROR Indexer (Indexer ) 05.09.07 20:04:59
[OM 16082] Directory is unavailable at this time. Please retry later.

Current errno value: 4
-> sis_free
-> removeRequest
<- removeRequest
-> weedEntry
<- weedEntry
-> sis_create
-> ul_getBySXUID
-> dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- dr_ACISetDefaultContext
-> dr_ACIModContextFlags
<- dr_ACIModContextFlags
<- /build/11.0.3/src/lib/ombase/vi/vi_locks.c:352[3,16082]
<- /build/11.0.3/src/lib/ombase/vi/vi_locks.c:201[3,16082]
<- /build/11.0.3/src/lib/ombase/dr/dr_search.c:470[3,16082]

Search still doesnt work.

any other ideas?

Postby **ScalixSupport** » Thu May 10, 2007 6:55 am

Hi!

This seems to be a known issue:
http://bugzilla.scalix.com/show_bug.cgi?id=12081

It might possibly be due to high CPU usage and slow responses. You can try to monitor the
CPU usage when you expect this error, there has been a lot of improvements done to SIS
in 11.0.3, we are constantly trying to improve SIS more and more. I would suggest you to
upgrade the Scalix Server to Scalix 11.0.4 and search should work then.

Thanks,
Subir

eyalm · Postby **eyalm** » Thu May 10, 2007 9:55 am

I'm not able to view that bug, but I'll upgrade scalix this weekend to 11.0.4 and try it again.

eyalm · Postby **eyalm** » Sun May 13, 2007 3:37 pm

I just upgraded to 11.0.4 and search is still not working.
I tried with sxsearch and I do get results, but when searching from swa, it gives me no results.

Any ideas what else can be wrong? maybe some config files got changed when I upgraded from 11.0.2 to 11.0.3?

Thanks!

Postby **ScalixSupport** » Mon May 14, 2007 9:45 am

Please send the response with result of the following:
omstat -a
omstat -s
ps ax | grep index
ps ax | grep java
ps ax | grep post

Thanks,
Subir

eyalm · Postby **eyalm** » Mon May 14, 2007 10:17 am

Here is the output:

[root@scalix ~]# omstat -a
PC Monitor Started NON-STOP 0
Directory Relay Server Started 05.13.07
Notification Server Started 05.13.07 0
Shared memory daemon Started NON-STOP
Notification Monitor Started NON-STOP
Session Monitor Started NON-STOP
Indexer Started NON-STOP
Stats Daemon Started NON-STOP
Container Access Monitor Started NON-STOP
Item Structure Server Stopped 02.17.07
Database Monitor Started 05.13.07
Licence Monitor Daemon Started NON-STOP
LDAP Daemon Started 05.13.07
Queue Manager Started NON-STOP
Item Delete Daemon Started NON-STOP
IMAP Server Daemon Started 05.13.07
SMTP Relay Started 05.13.07
Mime Browser Controller Started 05.13.07
Event Server Started 05.13.07

[root@scalix ~]# omstat -s
Service Router Started 05.13.07 0
Local Delivery Started 05.13.07 0
Internet Mail Gateway Started 05.13.07 0
Sendmail Interface Started 05.13.07 0
Local Client Interface Enabled 05.13.07 0
Remote Client Interface Enabled 05.13.07 282
Test Server Started 05.13.07 0
Request Server Started 05.13.07 0
Print Server Started 05.13.07 0
Directory Synchronization Started 05.13.07 0
Bulletin Board Server Started 05.13.07 0
Background Search Service Started 05.13.07 0
Dump Server Started 05.13.07 0
CDA Server Started 05.13.07 0
POP3 interface Started 05.13.07 0
Omscan Server Started 05.13.07 0
Archiver Started 05.13.07 0

[root@scalix ~]# ps ax| grep index
2766 ? Ss 0:03 indexer
2767 ? S 0:09 index.catcher
2603 ? S 0:00 /opt/scalix/bin/index.browse
2702 ? S 0:00 /opt/scalix/bin/index.browse
2705 ? S 0:00 /opt/scalix/bin/index.browse
2762 ? S 0:00 /opt/scalix/bin/index.browse
2777 ? S 0:00 /opt/scalix/bin/index.browse
2845 ? S 0:00 /opt/scalix/bin/index.browse
23518 pts/0 S+ 0:00 grep index

[root@scalix ~]# ps ax| grep java
3267 ? Sl 38:09 ./jre/bin/java -Djava.compiler=NONE -cp /usr/RaidMan/RaidMan.jar com.ibm.sysmgt.raidmgr.agent.ManagementAgent
20450 ? Sl 9:16 /usr/java/jre1.5.0_06/bin/java -server -Djava.net.preferIPv4Stack=true -Xms1024m -Xmx1024m -Dscalix.instance=/var/opt/scalix/sx -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/opt/scalix/sx/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/scalix-tomcat/common/endorsed -classpath /usr/java/jre1.5.0_06/lib/tools.jar:/opt/scalix-tomcat/bin/bootstrap.jar:/opt/scalix-tomcat/bin/commons-logging-api.jar -Dcatalina.base=/var/opt/scalix/sx/tomcat -Dcatalina.home=/opt/scalix-tomcat -Djava.io.tmpdir=/var/opt/scalix/sx/tomcat/temp org.apache.catalina.startup.Bootstrap start
23612 pts/0 S+ 0:00 grep java

[root@scalix ~]# ps ax| grep post
2642 ? S 0:00 /usr/bin/postmaster -i -h scalix.int.cardonhealthcare.com -p 5733 -D /var/opt/scalix/sx/postgres/data -k /var/opt/scalix/sx/postgres/data
2644 ? S 0:00 postgres: stats buffer process
2645 ? S 0:00 postgres: stats collector process
4458 ? S 0:07 postgres: scalix scalix 10.1.1.33 idle
4459 ? S 0:07 postgres: scalix scalix 10.1.1.33 idle
6521 ? S 0:07 postgres: scalix scalix 10.1.1.33 idle
6522 ? S 0:08 postgres: scalix scalix 10.1.1.33 idle
7297 ? S 0:07 postgres: scalix scalix 10.1.1.33 idle
23795 pts/0 R+ 0:00 grep post

Thanks!

Postby **ScalixSupport** » Tue May 15, 2007 7:03 am

Hi!

Can you run the Scalix installer to re-configure the following components:
Scalix Search and Index
Scalix Messaging Services
Scalix DB

Thanks,
Subir

eyalm · Postby **eyalm** » Tue May 15, 2007 9:17 am

Subir,

I just installed those components last sunday.
I even had an error message while installing so I had to reconfigure all the components.

Any other ideas?

Postby **ScalixSupport** » Tue May 15, 2007 9:54 am

Hi!

What are the values for the following in /var/opt/scalix/??/webmail/swa.properties file:
swa.soap.soapRequestTimeout=
swa.platform.url=
swa.search.maxresults=
swa.user.locale=

If SIS is installed on a different machine, edit the following lines in the file, ~sys/general.cfg:
USRL_SIS_URL=sxidx://<FQDN:port>/%u
USRL_BB_SIS_URL=sxidx://<FQDN:port>/BB@<fqdn>
Where FQDN is the host that SIS is install on, and the port is optional, but almost always is
80. Restart SIS Tomcat and make sure it's initialized.

What is the value for the following in /var/opt/scalix/??/sis/sis.properties file:
index.max.queue.size=
search.max.results=
index.language=

Do the following have similar values in file /var/opt/scalix/??/sis/sis.properties:
search.client.whitelist=
index.client.whitelist=

Thanks,
Subir

Scalix Forums

Search stopped working in 11.0.3

Search stopped working in 11.0.3

Who is online