According to our http://docs.geotools.org/latest/developer/conventions/test/online.html - documentation :
The default behaviour of OnlineTestCase is that, if connect() throws an exception, the test suite is disabled, causing each test to pass without being run. In addition, exceptions thrown by disconnect() are ignored. This behaviour allows tests to be robust against transient outages of online resources, but also means that local software failures in connect() or disconnect() will be silent. To have exceptions thrown by connect() and disconnect() cause tests to fail, set skip.on.failure=false in the fixture property file. This restores the traditional behaviour of unit tests, that is, that exceptions cause unit tests to fail.</blockquote>
After setting up online tests on the build server, using skip.on.failure=false I have observed that connection failures to the database still do not cause the test to fail. Looking at the code, I have observed that the connect() method is either not implemented (PostgisNGCreateDatabaseOnlineTest, ...), or does not actually attempt to connect to the database (JDBCTestSupport, OracleOnlineTestCase, ...) for many of the implementations of OnlineTestCase, meaning that connect() often does not throw an exception if a connection is not available.
Additionally, the OnlineTestCase.run() method (which is called before connect()) calls checkAvailiable(), which usually <b>does</b> attempt to connect to the database. A failure here will skip all tests and return success, regardless of skip.on.failure=false.
Between these two issues, skip.on.failure=false does not work as documented.
CodeHaus Comment From: bencaradocdavies - Time: Tue, 24 Mar 2015 13:40:54 -0500
I loathe the behaviour of OnlineTestCase and OnlineTestSupport. The current functionality makes it far too easy to silently disable tests. The implementation is super-fragile and relies on perfect understanding by subclass implementers. I would much rather that tests fail, but I was not permitted to implement this behaviour. See <a href="https://jira.codehaus.org/browse/GEOT-1951" title="Support test fixtures that fail on failed connection" class="issue-link" data-issue-key="GEOT-1951"><strike>GEOT-1951</strike></a> and <a href="https://jira.codehaus.org/browse/GEOT-2320" title="OnlineTestCase support for failure on failed connection" class="issue-link" data-issue-key="GEOT-2320"><strike>GEOT-2320</strike></a> (from 2008 and 2009) for discussion.
P.S. can we get our change proposals off codehaus?