Grails Searchable plugin: fighting down OOME when rebuilding the search index

In one of my projects we store blob data (word documents, pdfs,…) in a Grails domain class. This data should all be indexed by a search engine and provide a convenient search interface. Pretty easy to solve using the Grails Searchable plugin – customer satisfied.

The project went live, everything was fine the first time. Until a certain point, we found that rebuilding the index throwed OutOfMemoryException (OOME). Nothing unusual in the java world,  increasing -Xmx helped out for some time.

With a growing set of data we reached a point where -Xmx comes near the size of physical RAM in the machine. Plugging more RAM is not an option here, so I’ve decided to tackle down the cause of this.

After some debugging session I’ve found out that rebuilding the index is performed in batches. Grails Searchable uses the Compass Framework under the hood. Compass  uses a default batch size of 200. For whatever reason the Grails searchable plugin superseeds that to a fixed (!) value of 5000. This means that during indexing 5000 database rows (each one holding a megabyte-sized blob) are read into RAM and then stored in the index! As long as there are no blobs in the database 5000 might be a good value, the larger the batch size the faster indexing will work.

The best solution here would be a configuration parameter for the fetchCount, I’ve filed an JIRA for this. Until this is fixed, thers a workaround: reconfigure the compassGpsDevice bean in the application’s resources.groovy like this

Mission Accomplished.

using Redmine with IntelliJ X EAP

Since the times when I’ve started using Grails some years ago, I became a passionate user of IntelliJ IDEA. Basically the single one thing I’ve been missing when moving over from Eclipse was the lack of something like Mylyn. Since version 9, IntelliJ comes up with basic task and context management features. Support for Jira has always been great by a plugin from Atlassian.  Redmine is the also a great issue tracker/wiki/project management application. But there was no IDE integration into IntelliJ so far. The upcoming IntelliJ X version has support for this in their latest EAP.

It took me some time to figure out how that works, so I want to share what I’ve found out so far:

  • go to File|Settings dialog, select Tasks|Servers
  • add a redmine server
    IntelliJ settings dialog

    The API Token must be taken from the “My account” page, link “API Access Key”. The project ID is the short project identifier found on Redmine’s project settings page

  • test and validate the connection
  • apply the changes in settings dialog
  • Open a new task (Alt-Shift-N)
  • Enter either the id or a word contained in the issue’s title and press Crtl-Space to find matching issues from redmine.
  • Select one of the issues being found and create a new context and changelist for that one.

Having the redmine tasks in the IDE will be a big time safer for me.

Update:

If your redmine server uses https with a self-signed certificate, you must append the certifacte to JVM’s keystore: http://www.chrissearle.org/node/260

Grails countries plugin 0.2 released

Thanks to the contributions of other hackers I’m announcing the 0.2 release of the Grails countries plugin. The changes:

  • added languages for country and continent names:
  • country list is now maintained outside CountryBootstrap.groovy in seperate csv files
  • reworked properties file to consistently use ISO-3166 3-letter-code as key

new Grails plugin: Countries

A common requirement in many applications is to deal with countries and/or continents. The plugin’s goal is to apply the DRY principle to these requirements.

In short, the plugin offers:

  • populated domain classes for countries and continents
  • i18n for continent and country names (for now, only en and de locales). Currently there are 190 countries available.
  • tags for simple usage of
    • localized names for countries and continents
    • select boxes: for all countries, for all continents, for all countries of a continent or for a defined set of countries. The select box contents are automatically sorted by the the localized country/continent name.

resources:

For bug reports please use http://jira.codehaus.org, project “GrailsPlugins”, component “countries”.

Update: since the initial release requires the usage of Grails 1.3.4-SNAPSHOT a bugfix release 0.1.1 has been release shortly afterwards. 0.1.1 should work with Grails >= 1.0.

Grails Neo4j plugin 0.3 released

Today I released an update of the Grails Neo4j plugin (http://www.grails.org/plugin/neo4j). The main changes are:

  • compatibility with Grails 1.3.x. Be aware, Grails 1.3 – 1.3.3 are suffering from http://jira.codehaus.org/browse/GRAILS-6427, so either use Grails 1.2.x, or be brave and use a recent git build of Grails 1.3.4.SNAPSHOT.
  • usage of Neo4j 1.1 (released today just a few hours ago, so get it while it’s hot).

All changes:

  • [GRAILSPLUGINS-2302] – “home” link broken in the org.codehaus.groovy.grails.plugins.neo4j.Neo4jController views
  • [GRAILSPLUGINS-2303] – Problems with annotation Neo4jEntity
  • [GRAILSPLUGINS-2345] – upgrade to Neo4j 1.1
  • [GRAILSPLUGINS-2346] – <domainclass>.get() throws exception if id is not invalid
  • [GRAILSPLUGINS-2347] – <domainClass>.findAllBy<Field>(value) fails
  • [GRAILSPLUGINS-2349] – provide compatibility for Grails 1.3.x

getting a list of all i18n properties used in a Grails application

You might know this situation: in a project you start by hacking code that uses i18n properties instead of fixed strings in order to support multiple languages. The normal process in Grails is to use the g:message tag in controllers or gsp templates. Side by side you append the new introduced i18n property with some value in your messsages.properties file.

When support for a new language is requested, all you have to do is translating messages.properties. So far so good – this make i18n really easy.

But: when the project evolves, there’s a good chance that some of your i18n properties in messages.properties gets orphaned. Assume you remove a block of code from a gsp. It happens often that the i18n properties used in this block are not removed from messages*.properties because at some point you are not sure if it is referenced elsewhere. So what would be really useful here would be a list of all referenced i18n properties from your *.groovy/*.gsp files.

Doing so is pretty easy, just add a new gant script to your Grails application’s script folder, let’s name it i18nList.groovy. This script basically contains:

(sorry the color coding seems to fail for some Groovy regexes) The script recursivly iterates over all *.groovy and *.gsp files in your project and extract the part after the ‘code’ attribute of the message tag using a regex. The regex result are collected into an array. This array is sorted, unique’d and printed to the console. That’s it.

One word of caution: this gant script ‘works for me’. So depending on your code, you might notice that the used regex are not sufficient or even fail. Feel free to modify them for your needs, even better send back your modifications.

new Grails plugin released: modproxybalancer

Last week I attended the gr8conf, a really great conference. One of my favorite event was the Hackergarden. In the evening after the regular talks ~30+ people came together (should I call them nerds?), split up into small groups and did some hacking on Groovy & Grails related topics. I found myself together with Davide Rossi to do some coding on deploying a Grails application to multiple tomcat nodes and manage a loadbalancer in front of them. The original goal was to finish the task at end of evening. Unfortunately we set our goal too high, and did not manage to release some code at that evening. During the last weekend I took some time to finish this and did a release today.

This plugin‘s goal is to simplify the deployment of a grails application to a cluster consisting of multiple tomcat instances and an Apache httpd mod_proxy_balancer.

It assumes you have an Apache mod_proxy_balancer running in front of multiple tomcat instances in order to provide a) high-availability and b) load-balancing to your Grails application. In such a scenario, upgrading the running application to a newer release is painful and error-prone when done manually. This plugin’s goal is to simplify that procedure by performing a “rolling upgrade”. Calling ‘grails tomcat redeploy’ performs these actions:

  1. take first node offline in loadbalancer
  2. undeploy app on first tomcat node
  3. deploy app on first tomcat node
  4. check if app responds on first tomcat node
  5. take first node online in loadbalancer
  6. proceed the same procedure with next node

To solve this, we’ve forked the 1.3.1 release of the tomcat plugin an added the following capabilities:

  1. deploying to multiple hosts and
  2. emitting lifecycle events: PreDeploy, PostDeploy, PreUndeploy, PostUndeploy
  3. provide support for ‘grails tomcat redeploy’

These changes have been included in a pull request to the upstream plugin, so I’m hoping they will find their way into Grails 1.3.2.

The rest of the work has been put into the modproxybalancer plugin, taking care of remote controlling the loadbalancer  by catching up these events. mod_proxy_balancer comes with a simple management frontend called balancer-manager. This consists of some simple HTML forms that can be easily triggered using htmlunit. When the forked tomcat plugin emits a “PreUndeploy” event, the loadbalancer disables the respective tomcat node, then a normal undeploy und deploy happens. After this, the freshly deployed tomcat is checked for availablity and taken back online in the loadbalancer.

The plugin is licensend under the WTFPL.

Side note: this is my first Grails plugin with it’s sources residing on github.com. Thanks to Peter Ledbrook’s excellent blog post, I was able to use GitHub pages for the documentation.

restrict a Grails controller to localhost access only

A common requirement for many web applications is that some parts (aka controllers) should only be accessible from specifc ip addresses. Typically controllers doing some administrative or maintenance work must be protected from non-authroized access. The most complete solution for this is using a full blown security framework like the Grails Acegi plugin. But there’s also a lean and quick solution for this in Grails: use a controller interceptor:

Grails calls the beforeIntereceptor closure prior every action in a controller. Only if it returns true, the action is executed. In the code above if the client has a non-local IP address, a 401 error is returned with an error message. Note that localhost in IPv6 is 0:0:0:0:0:0:0:1, so it work both in IPv4 and IPv6 networks.

Grails Neo4j plugin 0.2.1 released

Today I released a minor update of the Grails Neo4j plugin. The changes are:

  • performance improvement by no longer calling map constructor in createInstanceForNode
  • fixed transaction handling by replacing interceptor with a “real” servlet filter
  • support for primitve arrays as properties in domain classes
  • bugfix: handling of bidirectional many-to-many relationships
  • bugfix: setProperties does no longer null out properties that have not been set
  • support for encodeAsXXXX methods from CodecsGrailsPlugin. In previous versions encoding did not work, since the ‘node’
    property of the domain classes could not be encoded (it’s a neo4j internal class!). Workaround: add getNode()==null method
    in AST transformation.

Everyone using the 0.2 release is recommended to upgrade.