I noticed I was getting a lot of 200 responses for tiles that are almost certainly in my GWC cache.
It seems the use of the Guava in memory tile cache may be the culprit. If my browser sends If-Modified-Since header, I get the normal 304 response with no body. If I clear my Guava cache (or in some cases the Guava cache automatically flushes itself), I get a new Last-Modified timestamp and a full 200 response. Even though the tiles exist on disk in the normal GWC directory. Its unclear if the disk cache is even being used.
The impact of this is that cache hit rates are very low causing poor performance of tiled layers. Turning off the in memory guava cache seem to restore the expected behavior.