Troubleshooting guide

Tomcat memory

Typically, the default memory allowance for the Java Virtual Machine (JVM) is too low. The memory requirements for Web Apollo will depend on many variables, but in general, we recommend at least 1g for the heap size and 256m for the PermGen size as a starting point.

Suggested Tomcat memory settings

export CATALINA_OPTS="-Xms512m -Xmx1g \
        -XX:+CMSClassUnloadingEnabled \
        -XX:+CMSPermGenSweepingEnabled \
        -XX:+UseConcMarkSweepGC \

In cases where the assembled genome is highly fragmented, additional tuning of memory requirements and garbage collection will be necessary to maintain the system stable. Below is an example from a research group that maintains over 40 Apollo instances with assemblies that range from 1,000 to 150,000 scaffolds (reference sequences):

export CATALINA_OPTS="-Xmx12288m -Xms8192m \
        -XX:PermSize=256m \
        -XX:MaxPermSize=1024m \
        -XX:ReservedCodeCacheSize=64m \
        -XX:+UseG1GC \
        -XX:+CMSClassUnloadingEnabled \
        -Xloggc:$CATALINA_HOME/logs/gc.log \
        -XX:+PrintHeapAtGC \
        -XX:+PrintGCDetails \

To change your settings, you can usually edit the script in $TOMCAT_BIN_DIR/ where $TOMCAT_BIN_DIR is the directory where the Tomcat binaries reside. It is possible that this file doesn't exist by default, but it will be picked up when Tomcat restarts. Make sure that tomcat can read the file.

In most cases, creating the should be sufficient but you may have to edit a or another file directly depending on your system and tomcat setup. For example, on Ubuntu, the file /etc/default/tomcat7 often contains these settings.

Confirm your settings

Your CATALINA_OPTS settings from can be confirmed with a tool like jvisualvm or via the command line with the ps tool. e.g. ps -ef | grep java should yield something like the following allowing you to confirm that your memory settings have been picked up.

root      9848     1  0 Oct22 ?        00:36:44 /usr/lib/jvm/java-7-openjdk-amd64/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/current/conf/ -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Xms1g -Xmx2g -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled -XX:+UseConcMarkSweepGC -XX:MaxPermSize=512m -Dj

Re-install after changing settings

If you start seeing memory leaks (java.lang.OutOfMemoryError: Java heap space) after doing an update, you might try re-installing, as the live re-deploy itself can cause memory leaks or an inconsistent software state.

If you have named your web application named Apollo.war then you can remove all of these files from your webapps directory and re-deploy.

  • Run apollo deploy (or apollo release for javascript-minimization)
  • Undeploy any existing Apollo instances
  • Stop tomcat
  • Copy the war file to the webapps folder
  • Start tomcat

Tomcat permissions

Preferably, when running Apollo or any webserver, you should not run Tomcat as root. Therefore, when deploying your war file to tomcat or another web application server, you may need to tune your file permissions to make sure Tomcat is able to access your files.

On many production systems, tomcat will typically belong to a user and group called something like 'tomcat'. Make sure that the 'tomcat' user can read your "webapps" directory (where you placed your war file) and write into the annotations and any other relevant directory (e.g. tomcat/logs). As such, it is sometimes helpful to add the user you logged-in as to the same group as your tomcat user and set group write permissions for both.

Consider using a package manager to install Tomcat so that proper security settings are installed, or to use the jsvc

Errors with JBrowse

e.g. "Can't locate Hash/ in @INC" or "Can't locate in @INC"

If you are trying to run the jbrowse binaries but get these sorts of errors, try running which will initialize as many pre-requisites as possible including JBLibs and other JBrowse dependencies.

Rebuilding JBrowse

You can manually clear jbrowse files from web-app/jbrowse and re-run apollo deploy to rebuild JBrowse.

RequestError: Unable to load ... Apollo2/jbrowse/data/trackList.json status: 500

Apollo2 does fairly strict JSON validation so make sure your trackList.json file is valid JSON

If you still get this error after validating please forward the issue to our github issue tracker.

Complaints about 8080 being in use

Please check that you don't already have a tomcat running netstat -tan | grep 8080. Sometimes tomcat does not exit properly. ps -ef | grep java and then kill -9 the offending processing.

Note that you can also configure tomcat to run on different ports, or you can launch a temporary instance of apollo with apollo run-local 8085 for example to avoid the port conflict.

Unable to open the h2 / default database for writing

If you receive an error similar to this:

SEVERE: Unable to create initial connections of pool.
org.h2.jdbc.JdbcSQLException: Error opening database: 
    "Could not save properties /var/lib/tomcat7/prodDb.lock.db" [8000-176]

Then this is due to the production server trying to write an h2 instance in an area it doesn't have permissions to. If you use H2 (which is great for testing or single-user user, but not for full-blown production) make sure that:

You can modify the specified data directory for the H2 database in the apollo-config.groovy. For example, using the /tmp/ directory, or some other directory:

url = "jdbc:h2:/tmp/prodDb;MVCC=TRUE;LOCK_TIMEOUT=10000;DB_CLOSE_ON_EXIT=FALSE"

This will write a H2 db file to /tmp/prodDB.db. If you don't specify an absolute path it will try to write in the same directory that tomcat is running in e.g., /var/lib/tomcat7/ which can have permission issues.

More detail on database configuration when specifying the apollo-config.groovy file is available in the setup guide.

Grails cache errors

In some instances you can't write to the default cache location on disk. Part of an example config log:

2015-07-03 14:37:39,675 [main] ERROR context.GrailsContextLoaderListener  - Error initializing the application: null
        at grails.plugin.cache.ehcache.GrailsEhCacheManagerFactoryBean$ReloadableCacheManager.rebuild(
        at grails.plugin.cache.ehcache.EhcacheConfigLoader.reload(EhcacheConfigLoader.groovy:63)
        at grails.plugin.cache.ConfigLoader.reload(ConfigLoader.groovy:42)

There are several solutions to this, but all involve updating the apollo-config.groovy file to override the caching defined in the Config.groovy.

Disabling the cache:

    grails.cache.config = {
        cache {
            enabled = false
            name 'globalcache'

Disallow writing overflow to disk

Can be used for small instances

    grails.cache.config = {
        // avoid ehcache naming conflict to run multiple WA instances
        provider {
            name "ehcache-apollo-"+(new Date().format("yyyyMMddHHmmss"))
        cache {
            enabled = true
            name 'globalcache'
            eternal false
            overflowToDisk false   // THIS IS THE IMPORTANT LINE
            maxElementsInMemory 100000

Specify the overflow directory

Best for high load servers, which will need the cache. Make sure your tomcat / web-server user can write to that directory:

    // copy from Config.groovy except where noted
    grails.cache.config = {
        cache {
            maxElementsOnDisk 10000000
            // this is the important part, below!
                path '/opt/apollo/cache-directory'

Information on the grails ehcache plugin (see "Overriding values") and ehcache itself.