The OpenLayers preview does not respect the URL scheme.
It creates URLs to http despite being accessed through https.
It's not enough to set the Proxy Base URL with https.
This will render security errors in both Chrome and Internet Explorer.
In Internet Explorer you can accept insecure content and it works anyway.
It would be better if GeoServer used the URL scheme the client is using.
An easy way is using
instead of http:
The browser will then use the scheme used for the containing page.