Concurrent WFS 2.0 load can cause shared WFS/GML schemas corruption

Description

A simple load test like:

1 ab -n 10000 -c 8 "http://localhost:8080/geoserver/topp/wfs?service=WFS&version=2.0.0&request=DescribeFeatureType"

results in many stack traces like:

1 2 3 4 5 6 7 8 9 10 16 ago 14:53:30 WARN [geoserver.ows] - Error firing finished callback for class org.geoserver.wfs.xml.SchemaCleanerCallback java.lang.NullPointerException at org.eclipse.emf.ecore.util.EcoreEList.resolveProxy(EcoreEList.java:212) at org.eclipse.emf.ecore.util.EcoreEList.indexOf(EcoreEList.java:411) at org.eclipse.emf.common.util.AbstractEList.remove(AbstractEList.java:463) at org.geotools.xml.Schemas.dispose(Schemas.java:416) at org.geoserver.wfs.xml.SchemaCleanerCallback.finished(SchemaCleanerCallback.java:49) at org.geoserver.ows.Dispatcher.fireFinishedCallback(Dispatcher.java:304) at org.geoserver.ows.Dispatcher.handleRequestInternal(Dispatcher.java:280) at org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:147)

Loading with GetFeature:

1 ab -n 1000 -c 16 "http://localhost:8080/geoserver/topp/wfs?service=WFS&version=2.0.0&request=GetFeature&typeName=topp:states"

results instead in the following

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 16 ago 14:57:39 ERROR [geoserver.ows] - java.util.ConcurrentModificationException at org.eclipse.emf.common.util.AbstractEList$EIterator.checkModCount(AbstractEList.java:762) at org.eclipse.emf.common.util.AbstractEList$EIterator.doNext(AbstractEList.java:710) at org.eclipse.emf.common.util.AbstractEList$EIterator.next(AbstractEList.java:696) at org.eclipse.xsd.impl.XSDSchemaImpl.changeReference(XSDSchemaImpl.java:2558) at org.eclipse.xsd.impl.XSDConcreteComponentImpl.eNotify(XSDConcreteComponentImpl.java:1135) at org.eclipse.emf.ecore.util.EcoreEList.dispatchNotification(EcoreEList.java:255) at org.eclipse.emf.common.notify.impl.NotifyingListImpl.addUnique(NotifyingListImpl.java:310) at org.eclipse.emf.common.util.AbstractEList.add(AbstractEList.java:307) at org.eclipse.xsd.impl.XSDSchemaImpl.imported(XSDSchemaImpl.java:3126) at org.geoserver.wfs.xml.FeatureTypeSchemaBuilder$GML32.importGMLSchema(FeatureTypeSchemaBuilder.java:1018) at org.geoserver.wfs.xml.FeatureTypeSchemaBuilder.buildSchemaInternal(FeatureTypeSchemaBuilder.java:253) at org.geoserver.wfs.xml.FeatureTypeSchemaBuilder.build(FeatureTypeSchemaBuilder.java:154) at org.geoserver.wfs.xml.FeatureTypeSchemaBuilder.build(FeatureTypeSchemaBuilder.java:142) at org.geoserver.wfs.xml.ApplicationSchemaXSD2.buildSchema(ApplicationSchemaXSD2.java:74) at org.geotools.xml.XSD.getSchema(XSD.java:232)

This is happening because the shared schemas are not thread safe, while the Schemas.dispose does synchronize on Schemas itself, the FeatureTypeBuilder can build new schemas in parallel that are referencing the WFS and GML ones, thus modifying the list of back-references from WFS/GML to such schemas (EMF and XSD are structured in such a way that 2-way associations are always kept current, causing lots of synchronization and memory usage for us while building the temporary schemas used to respond WFS requests)

Environment

None

Status

Assignee

Andrea Aime

Reporter

Andrea Aime

Triage

None

Fix versions

Affects versions

None

Priority

Medium
Configure