How do spritemasks work with the stencil buffer?

[cobalt valve]

Hello everyone! Im trying to have a spritemask that masks out a particular thing. I made two shaders, one that writes to the stencil buffer and one that reads to it. This works great, but totally messes with existing sprites that use the spritemask system. It seems anywhere I write to the stencil buffer is considered “inside the mask”. If anyone knows how sprite masks actually work under the hood, that would be great. Its really frustrating how limited they seem to be.

[dull fjord]

Sprite masks have 2 modes: visible inside and visible outside. Depending on the selected mode it would show or hide pixels that have stencil value equal to 1.

[dull fjord]

You can also have a look at the frame debugger to see the stencil operations at each draw call.

[cobalt valve]

frame debugger gotcha

i tried renderdoc and couldnt make sense of it, im very unfamiliar with directx

its odd because im writing in a value of 80 and its acting as if thats a sprite mask

im also curious how the masked sprite interacts with the stencil buffer when the shader im using has no stencil block

is that added automatically in shader compilation?

[cobalt valve]

looking at the frame debugger this is not how it works, at least in unity 2021. Each sprite mask increments the value in the stencil buffer, i assume to keep track since they also decrement the value after the layer range has passed. As for the sprites themselves, it seems anything other than 0 is considered “in the mask”. Sprite set to render outside mask havr a ref value of 1 and use Greater

id be surprised if it did work differently in unity 6 considering that it needs to keep track of the amount of sprite masks rendered but yeah at least in 2021 it totally takes over the stencil buffer

[dull fjord]

Yeah. That aligns with my understanding.
To put it simply you can't use stencil buffer in your shaders, unless you render them in your own pass.

[dull fjord]

Stencil block is part of shader lab. Not the actual shader. The actual stencil settings are managed via the graphics api, which unity does on the C++ side it seems.