EPSG lookup failures caused by concurrent database shutdown in FactoryUsingHSQL finalizer

Description

Database shutdown in the FactoryUsingHSQL finalizer causes concurrent EPSG lookup via other FactoryUsingHSQL instances using the same database URL and thus the same org.hsqldb.Database instance to fail in, for example, GeoServer gs-main unit tests.

Note that ThreadedHsqlEpsgFactory.createDataSource() opens the database with "shutdown=true" so the database will be shutdown when the last session is closed. This suggests that an acceptable fix might be overriding FactoryUsingHSQL.shutdown() to disable it for use in ThreadedHsqlEpsgFactory. This would be backwards-compatible with all other uses.

There is one org.hsqldb.Database instance per JVM for each unique database URL. FactoryUsingHSQL really has no business shutting down a resource that it does not own. Furthermore, all synchronization on the parent DirectEpsgFactory dispose() and getConnection() is on the DirectEpsgFactory instance and does not protect against use by other DirectEpsgFactory instances of the same org.hsqldb.Database instance concurrent with its shutdown.

Failures are readily seen in about 8% of gs-main "mvn clean install" builds on a 2 CPU machine (e.g. Linux booted with maxcpus=2). Overriding FactoryUsingHSQL.shutdown() to disable it for use in ThreadedHsqlEpsgFactory seems to reduce the total failure rate of gs-main builds to about 0.5%.

Occurrences of the finalizer and related failure can be seen by rebuilding hsqldb 2.4.1 with

and ./gradlew -Dbuild.debug=true hsqldb. For example:

Environment

Linux booted with maxcpus=2 to reproduce GeoServer gs-main failures.

Status

Assignee

Ben Caradoc-Davies

Reporter

Ben Caradoc-Davies

Triage

None

Components

Fix versions

Affects versions

19.1
18.4
20-RC

Priority

Medium
Configure