When using a PostGIS datastore it is possible to easily invert a polygon (by using ST_Difference).
This can be useful for visualization issues (for example when computing reachable areas), but it does not work when WMS advanced projection handling is enabled and a very large "masking polygon" (the whole world) is used.
I described this issue already on stackexchange with some sample screenshots:
Some sample data to reproduce the issue using a PostGIS datastore (in a PostGIS enabled database):
OS: Fedora 24
Thanks for the detailed bug report.
Can I focus on a workaround using styles and layer composition and the composite-blend functionality.
As I understand it you would draw:
whole world polygon filled with transparency as the composite-base
draw your cookie cutter polygon using "destination-over"