Slow initial loading of layers in Geoserver from huge COGs stored in a AWS S3
Description
Environment
Geoserver version is: 2.22.3
Web server is: Tomcat 9
OS is: Ubuntu
Activity
Sina Abolhoseini June 19, 2023 at 2:09 PM
You are absolutely right. I assumed that it is in KB. Now it is fixed and the initial load is pretty quick.
Thanks.
Andrea Aime June 19, 2023 at 7:21 AM
Hum… you made the value smaller (I guess you assumed it was expressed in KB, but it’s in bytes).
That of course meant many more calls to the COG. Try with 262144 (256KB) or more instead.
Sina Abolhoseini June 19, 2023 at 2:31 AM
Maybe I did it wrong. I place the line below in /var/lib/tomcat9/conf/catalina.properties:
it.geosolutions.cog.default.header.length=512
I checked the System Properties after that in Geoserver server status panel, under Modules tab. The above parameter is mentioned there after adding it in the mentioned file.
I have also checked the Geoserver logs, after I clicked on previewing one of the huge COGs. I noticed that after calling establishing style it takes too long to call the request:
The COGs I am working on sometimes have 3 bands (RGB) and sometimes 4 or 5 (RGB + NIR). Any thoughts on that?
Andrea Aime June 17, 2023 at 3:38 PM
That’s weird… I was dealing with a large COG some time ago and increasing the read length, with that same property, made initial reads significantly faster. Ideally, when dealing with massive cogs, the code should try to load only the tile index portion that it needs, but doing so would require redesigning the underlying TIFF reader code (possible, not trivial)
Details
Details
Assignee
Reporter
Sina AbolhoseiniAffects versions
Components
Priority
Medium
Hello,
I am working with huge COGs (around 500GB) stored on my private S3 bucket. I am trying to publish these COGs as GeoTiff layers using COG community module in Geoserver. The Geoserver is deployed on an EC2 instance with 8 vCPUs and 16Gb of RAM. The projection system of COGs is WGS84 (EPSG:4326) and I want to access the WMS in WGS84.
The layers are published successfully, but when I want to access the WMS of these layers, the initial loading of the layers take about a couple of minutes. But when they are loaded, everything works smoothly and tiles are loaded pretty quick. Also, when the COG is smaller in size (less than 2GB), the initial loading is pretty good.
I realized that first thing Geoserver does, is to load the header of the COG and then access the inner tiles. By default, Geoserver reads first 16KB of COGs to access the header. I have tried to increase
it.geosolutions.cog.default.header.lengthso that a larger portion of the COG is read initially. But, it ended up slower loading of COGs.