Custom Render Pass Help

[woven cloak]

Hello, I am trying to have a specific camera render just the area taken up by a certain mesh - i.e. a stencil buffer (I think) to render in a mask area defined by another mesh using custom render passes (in HDRP).

The image below is an example of what I am trying to do, where a purple cube acts as the masking area and when a green sphere is dragged in front of it partially, only the area of the purple cube is rendered.

I have looked through the documentation and downloaded some examples to try and understand how custom render passes work, but I'm still struggling to figure out how to do the basic steps to achieve this goal.

What I think I have to do:

1. Have a custom pass volume with the selected camera I want and a number of custom passes
2. On this custom pass volume, have a DrawRendersCustomPass to specify what mesh I want to have act as the mask
3. Another DrawRendersCustomPass to render only where this mask is for the specified camera

If anyone could advise me on whether this is the right way to think about or do this, I would really appreciate it!

And sorry if this is a basic thing to do - I'm relatively new and tried my best to understand the documentation and mess around with example projects but alas I am here - if there are any resources I should be familiar with please let me know!
Thank you!

[orchid mantle]

Just basic stenciling (or by depth depending on requirements). You can do it directly in the shader or by using URP render objects and render them on a per layer basis.
https://docs.unity3d.com/6000.3/Documentation/Manual/urp/renderer-features/how-to-custom-effect-render-objects.html

For stenciling you want your purple cube there to increment the stencil buffer by 1, then for the green object it will only draw if the current stencil buffer has that value currently set.

You can also include depth checks to only draw if the object is in front or behind the masking object in space.

[orchid mantle]

Well, there's also more information needed for how it should all render because you will run into sorting issues potentially until you know the exact requirements.

[woven cloak]

This is for HDRP - sorry, I should have made that clearer and have updated the post to reflect that

[orchid mantle]

https://docs.unity3d.com/Packages/com.unity.render-pipelines.high-definition@17.3/manual/Custom-Pass-Volume-Workflow.html
Check these out. Haven't used them but sounds like the equivalent

Implying you want something UI based, otherwise doing it via scripting works fine, or like I said you can do it all in the shader.

[woven cloak]

Conceptually, how can you do this all in a shader? I.e. how does one go about telling a camera to render only in an area defined by a shader on screen?

[thorny coral]

You do need a mask or stencil buffer for that.

[thorny coral]

You can't tell a camera to only render a specific area of the screen(you can if it's a rectangular area, but that's a different topic). During the pixel shader stage, when each pixel is rendered you can provide a condition for this pixel to be skipped. This is exactly why you need a stencil buffer or a mask.

[woven cloak]

Could the attached image be accomplished from the view of a camera then? Or, would the attached image need to be the output of a shader that takes in a render texture which a camera writes to, where the render texture originally has the fully sphere in view?

[thorny coral]

No, you don't need a separate camera/render texture setup.
You need to have several render passes. In one render your mask to a separate render target, and blit/copy it to a texture. In the other, render the object that needs to be masked. Bind the mask to the shader(could make it a global shader/material resource). sample the mask to see if the pixel should be drawn or not.

Check this page on how to set up a separate render pass with a different render : https://docs.unity3d.com/Packages/com.unity.render-pipelines.high-definition@17.3/manual/Custom-Pass-Scripting.html

Note, that unity calls it an "outline buffer" in that example, but it's really a render target(a texture you can render to). RTHandle stands for Render Target(Texture?) Handle.

It seems like unity also uses the word Render Texture interchangeably which is really confusing to be honest, as they refer to the Render Texture asset in the same way.

[orchid mantle]

Really you can do it all in the shaderlab code, but I prefer doing it via render objects / custom render passes because you can just specify a rendering layer instead of having to throw all the stencil logic inside of the shader code as you'd have to do it for each individual shader.

Like even the ordering of when the mask 'renders' you can specify in it too

For more common stencil effects you'd usually want the mask to be in the opaque queue though for things like creating a hole of an occluding wall

[woven cloak]

Ah, ok. In those cases, how does this impact performance? In the general case of the purple cube and green sphere again, would accomplishing what the attached image shows help performance? Like, does masking out objects = less pixels to render = better performance. Or does it not matter because the camera still actually captures the whole scene (with the whole green sphere) but then just applies the operations you describe (perhaps slightly decreasing performance?) to output the attached image?

[thorny coral]

It depends.
If the masked object uses a very heavy pixel shader, masking some of the pixels out might improve performance. It still depends on many factors.

If you're doing that just for performance reasons, then I'd step back and reconsider.
Implement your desired features and visual effects first, test and profile, and only if you detect performance issues, investigate and optimize accordingly.