The fid index file seems corrupted (some index are missing) after features deletion (using WFS-T call)

Description

I display my shapefile using Leaflet and I send a request to my geoserver in order to remove a feature (see the request below).
Everything is OK, my feature is well removed and my shapefile well updated.
But in fact, internally, the fid index file (XXX.fix) is corrupted because one index is now missing.
Therefore, if I select the feature that is missing in the XXX.fix file, I'm not able to update/delete it anymore: transactions are ignored and the WMS api returns OK but with wfs:totalInserted,wfs:totalUpdated and wfs:totalDeleted all equal to 0.

Technical investigations
The issue is coming from the geoserver sources (DeleteElementHandler.java), the instruction "store.getFeatures(filter).size()" returns 0 but should not.
After some digging, it's seems to come from geotools sources (IndexManager.java), the instruction "reader.findFid(fid)" should not return -1.
I provided a java file (IssueFidIndex.java) that helps me to investigate and to see what is in the file XXX.fix and wich index is missing.

Manual workaround
1) If I manually stop geoserver, remove the XXX.fix file and restart geoserver, then the index file is well regenerated with no more missing index.
2) Switch the value of fidIndexed in the file ShapefileDataStore.java (see details in comments)

Here comes the XML transaction sent through geoserver wms api in order to delete a feature (filtering by fid):

Environment

Using geoserver 2.11.1 and geotools 17.1
Using a "Directory of spatial files (shapefiles)" store
Using the shapefile provided by geoserver package (data_dir\data\shapefiles\states.shp)
OS: Windows 10

Activity

Show:
Ian Turton
March 29, 2019, 4:01 PM

see https://github.com/geotools/geotools/pull/2338 for a first look at fixing this

Alban44
September 26, 2017, 1:16 PM

We finally managed to find a "quick win" by changing the following line in the file ShapefileDataStore.java:

As the XXX.fix file is not used anymore, this issue is not occurring .

Andrea Aime
September 25, 2017, 7:48 AM

Yes, thank you. I've earmarked this one for the monthly bug sprint. Does not mean it will be solved "quickly", but raises the chances it will be addressed in "months" instead of "years". If are in need or a urgent for a solution I'd recommend trying a fix yourself and contributed it or look for commercial support (sorry, we just don't have enough core developers to quickly address tickets, many stay there for years just because core devs never stumble into them in the paid projects they're working on).
On the bright side, the project is open to you can help get a solution!
See these instructions about pull requests: http://docs.geotools.org/latest/developer/procedures/pull_requests.html

Alban44
September 25, 2017, 7:44 AM

Sorry if I was not clear enough, but I think you get it, here comes an example:
1) I sent a WFS-T deletion for the fid "8"
2) The deletion was OK
3) After that, I sent a new WFS-T deletion for the new fid "8" (because apparently, fid have been reassigned so fid "8" is exisiting and is different from the one I deleted before)
4) Nothing happen and wfs:totalDeleted is equal to 0, I'm not able to delete or modify this feature anymore
That behavior is due to the fact that the XXX.fix file seems to be corrupted as the fid "8" is not existing anymore in this file. And now, there is no way to retrieve it with a call to reader.findFid(fid)...
I hope it's clearer now?

Andrea Aime
September 24, 2017, 8:58 AM

Ah, your understanding of the count method also does not seems to be line with the documentation, a feature collection size method never returns -1, only a feature source size method can, and it does if the computation of size is not deemed to be fast.

Maybe I'm starting to understand what you actually meant... after the delete a different feature is at position "x", and it cannot be deleted or retrieved anymore?

Fixed

Assignee

Ian Turton

Reporter

Alban44

Triage

Components

Fix versions

Affects versions

Priority

High