.. tomcat eating cycles

[carlPjohnson]

Anyone have a problem with Tomcat just eating cycles? I have been expeirncing this after the upgrade to 11 (3 days ago) on a test server during the weekend that has 10 users and about 20g of email. Any ideas? I am guessing it is the indexer as it is trying to index attactments (see all these errors)?

(yes that is 47% of the cpu on a 2xXeon 3G machine, v10 didn't ever have this issue)
root 5141 47.0 7.8 761232 606352 pts/0 Sl 16:09 9:01 /usr/java/jre1.5.0_06/bin/java -server -Djava.net.preferIPv4

(SIS Logs)
2006-12-24 16:35:00,343 ERROR [TP-Processor1] [InternalIndexerServlet.errorResponse:285] Index failed
com.scalix.index.api.IndexerException: Cannot get message body
at com.scalix.index.message.IndexableMimeMessage.generateDocument(IndexableMimeMessage.java:67)
at com.scalix.index.manager.IndexManager.createDocument(IndexManager.java:467)
at com.scalix.index.manager.IndexManager.indexMessage(IndexManager.java:212)
at com.scalix.index.web.InternalIndexerServlet.doIndex(InternalIndexerServlet.java:196)
at com.scalix.index.web.InternalIndexerServlet.doPost(InternalIndexerServlet.java:173)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:709)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at com.scalix.index.web.IndexerFilter.doFilter(IndexerFilter.java:39)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:754)
at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:684)
at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Unknown Source)
Caused by: javax.mail.internet.ParseException: Expected ';', got "filename"
at javax.mail.internet.ParameterList.<init>(Unknown Source)
at javax.mail.internet.ContentDisposition.<init>(Unknown Source)
at javax.mail.internet.MimeBodyPart.getDisposition(Unknown Source)
at javax.mail.internet.MimeBodyPart.getDisposition(Unknown Source)
at com.scalix.index.message.IndexableMimeMessage.addMimeHeader(IndexableMimeMessage.java:365)
at com.scalix.index.message.IndexableMimeMessage.addContent(IndexableMimeMessage.java:281)
at com.scalix.index.message.IndexableMimeMessage.generateDocument(IndexableMimeMessage.java:64)
... 24 more
(SKIP OVER SOME TIME)
2006-12-24 16:35:45,080 ERROR [QueueManager] [BatchUpdater.processMods:63] Failed to get reference modifier for user 2d5000009e776454-902.14.861.291
java.io.IOException: Lock obtain timed out: Lock@/var/opt/scalix/sx/tomcat/temp/lucene-8d19b1d040cb6fec1451209859f61cac-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:56)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:254)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:244)
at org.apache.lucene.index.IndexModifier.init(IndexModifier.java:148)
at org.apache.lucene.index.IndexModifier.<init>(IndexModifier.java:124)
at com.scalix.sis.util.IndexUtil.getReferenceIndexModifier(IndexUtil.java:62)
at com.scalix.index.manager.BatchUpdater.processMods(BatchUpdater.java:60)
at com.scalix.index.manager.QueueManager.processQueue(QueueManager.java:127)
at com.scalix.index.manager.QueueManager.run(QueueManager.java:73)
at java.lang.Thread.run(Unknown Source)
2006-12-24 16:35:46,785 WARN [TP-Processor1] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type image/bmp; name=ole0.bmp
2006-12-24 16:35:46,785 WARN [TP-Processor1] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type image/bmp; name=ole1.bmp
2006-12-24 16:35:46,786 WARN [TP-Processor1] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type image/bmp; name=ole2.bmp
2006-12-24 16:35:47,178 WARN [TP-Processor8] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; ^M
name=20-N3578-D050912_20050920T074414.991.htm
2006-12-24 16:35:48,274 WARN [TP-Processor12] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; name=Latitude.pdf
2006-12-24 16:35:48,441 WARN [TP-Processor3] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; name=Latitude.pdf
2006-12-24 16:35:48,697 WARN [TP-Processor10] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; name=Latitude.pdf
2006-12-24 16:35:48,781 WARN [TP-Processor11] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; name=Latitude.pdf
2006-12-24 16:35:48,948 WARN [TP-Processor8] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; name=Latitude.pdf
2006-12-24 16:35:49,596 WARN [TP-Processor11] [IndexableMimeMessage.addContent:299] No extractor could be found to process message content with type application/octet-stream; name=DellTCO.pdf

[heupink]

I'm seeing the same: since our upgrade to 11, we have a java process running at 99% cpu most of the time (at 58% memory usage)

sar looks like this nowadays:

13:20:01 CPU %user %nice %system %iowait %idle
13:30:01 all 30.67 0.00 6.23 4.55 58.55
13:40:01 all 45.74 0.01 9.25 7.01 37.99
13:50:01 all 51.61 0.01 5.05 0.97 42.36
Average: all 42.81 0.04 4.76 0.91 51.47

last entry in scalix-sis-indexer.log is from a few hours ago, so it can't be that..?
We're running suse enterprise server 9 here, with xfs as filesystem

Sorry I can't help anymore, just wanted to say you're not alone...

[florian]

how much memory do you have in your system?

Florian.

[heupink]

we have 1 gig, of which 512 mb is dedicated to java.

Rachel of scalix support has just taken a look at my system, and says that probably what we see is the initial processing of sis and api.

It has been running for 4 days at 99.9 cpu, for a 16 gb datastore. Over the last 6 hours 8 new lines have been added to scalix-sis-indexer.log today.

Our server is an IBM xSeries with an intel xeon processor, don't know it's speed, but it's three years old. (so not really old, slow and crappy)

[florian]

It has been running for 4 days at 99.9 cpu, for a 16 gb datastore. Over the last 6 hours 8 new lines have been added to scalix-sis-indexer.log today.

Good that you're working with Support on this. I can easily believe that indexing a 16 gb datastore initially can take a couple of days, this is what we expected to happen with the larger installations and when introducing Scalix 11.

However, your last statement indicates that this is now more the indexing of new incoming messages and the base run is finished, so that should bring the load down.

For everybody's benefit, as soon as support resolves this, maybe you can drop a few lines here on the solution.

Cheers,
Florian.

[carlPjohnson]

We had about 50gb of email and it was sparadic and took a few days to finally calm down.

[zolo]

Hello and Hey,

... can easily believe that indexing a 16 gb datastore initially can take a couple of days, this is what we expected to happen with the larger installations and when introducing Scalix 11...

Aha, you expected that ... :-(

Workaround 1?
I found the Options for throttling the Indexer and i tried some settings without luck, the Tomcat-Java-Process always stays at 99%.

What values should this options have to "calm down" the Java-Prozess? Do you mean CPU-Load in percentages or what? Dokumentaton says these are the default-values, is this right? But why do you set this defaults. The Server has ZERO idle time!

IDX_MAXLOAD=8.0
IDX_MINLOAD=4.0

Workaround 2?
How can we stop indexing while daytime and resubmit it at night, without to start it from beginning.

Maybe there is another "Solution"?

Regards,
Zoki

[florian]

well, sxmkindex can be called per-user, so you could divide your users into batches and distribute the load over a couple of days...

-- F.

[jorgen3048]

well, sxmkindex can be called per-user, so you could divide your users into batches and distribute the load over a couple of days...

-- F.

My datastore is just 2Gb but still java takes 97% of the CPU. I've been running since Dec 23.