#Accessing stencil buffer in CompositorEffect

1 messages · Page 1 of 1 (latest)

light rock
#

I'm working on an outline effect similar to https://github.com/dmlary/godot-stencil-based-outline-compositor-effect, which needs to access the contents of the stencil buffer in its compute shader.

The effect on github does it by rendering a triangle that covers the entire screen, with a stencil test on a specific stencil value in order to create a mask of that stencil value. I could do something similar, but my effect requires checking multiple values in the stencil buffer instead of just one, so I'd have to do multiple passes of rendering full screen triangles just to recreate the stencil buffer that already exists. Surely there's some way to access the stencil buffer directly instead of wasting time remaking it?

#

My current thought is that the depth buffer and stencil buffer are stored together, so maybe it's possible to read the stencil values from the depth buffer that can be accessed with RenderSceneBuffersRD.get_depth_layer

#

When I do that I can get a depth texture with format 129, which includes stencil values

#

(for some reason, I only get this format when I'm looking at the camera preview, while running the game gives me DATA_FORMAT_R32_SFLOAT = 99 which doesn't have stencil, so I'd need to fix that if this method does turn out to work)

#

But where I'm stuck is actually accessing those stencil values from the shader

#

As far as I know there isn't any way to pass this depth/stencil texture format to the shader, or to sample from the stencil part rather than the depth part

#

Based on stuff I've found online (https://stackoverflow.com/questions/51855945/how-to-read-the-stencil-buffer-in-vulkan, https://stackoverflow.com/questions/27535727/opengl-create-a-depth-stencil-texture-for-reading), you need to create a texture view of the stencil part before sending it to the shader (?)

#

The godot texture view lets you pick a specific channel to sample, so maybe depth and stencil are treated as two different channels, and I just need to sample the stencil one, but how do I know what number the stencil channel is? Is there some way to check which channels exist? Do I just try numbers and see what happens?

#

Also, how do I know what format I'm working with? If the input texture is a DATA_FORMAT_D32_SFLOAT_S8_UINT, does that mean that my stencil texture view will also be DATA_FORMAT_D32_SFLOAT_S8_UINT? Wouldn't that mean you still can't access the "S8" part, just like with the regular depth texture?

#

The texture view lets you set a "format override" to any "shareable format", so maybe it's possible to override it to some kind of 8-bit integer format, but how do I know which formats are shareable? Is there a way to check from the code, or is there some kind of documentation for godot or vulkan or anything that covers this that I'm missing?

#

That's all I've really tried for now, but I'll update if I try anything else. Am I even going down the right path with the depth buffer + texture view stuff? Is reading the stencil buffer from a shader just not supported at all and the hack with rendering full-screen stencilled triangles is the only way to simulate it?

light rock
#

Not sure which of these could refer to the stencil buffer, unless it's just depth = R and stencil = G

#

I don't really think this is it at all but I'm going to try it since it's the only lead I have right now