Grails Searchable plugin: fighting down OOME when rebuilding the search index

In one of my projects we store blob data (word documents, pdfs,…) in a Grails domain class. This data should all be indexed by a search engine and provide a convenient search interface. Pretty easy to solve using the Grails Searchable plugin – customer satisfied.

The project went live, everything was fine the first time. Until a certain point, we found that rebuilding the index throwed OutOfMemoryException (OOME). Nothing unusual in the java world,  increasing -Xmx helped out for some time.

With a growing set of data we reached a point where -Xmx comes near the size of physical RAM in the machine. Plugging more RAM is not an option here, so I’ve decided to tackle down the cause of this.

After some debugging session I’ve found out that rebuilding the index is performed in batches. Grails Searchable uses the Compass Framework under the hood. Compass  uses a default batch size of 200. For whatever reason the Grails searchable plugin superseeds that to a fixed (!) value of 5000. This means that during indexing 5000 database rows (each one holding a megabyte-sized blob) are read into RAM and then stored in the index! As long as there are no blobs in the database 5000 might be a good value, the larger the batch size the faster indexing will work.

The best solution here would be a configuration parameter for the fetchCount, I’ve filed an JIRA for this. Until this is fixed, thers a workaround: reconfigure the compassGpsDevice bean in the application’s resources.groovy like this

compassGpsDevice(HibernateGpsDevice) { bean ->
    bean.destroyMethod = "stop"
    name = "hibernate"
    sessionFactory = ref("sessionFactory")
    fetchCount = 200
}

Mission Accomplished.