Comparing to a depth texture value

[finite nexus]

I'm rendering geometry in the usual way to a target with a depth texture, then in a post-processing shader pass running an implicit surface ray tracer, comparing the ray-tracer depth value to the saved depth texture to discard fragments behind the existing rendered geometry. The depth comparison isn't right. It looks correct for an orthographic camera, but is slightly off for a perspective camera. I don't have a small working snippet, but here is the relevant code in case my math mistake jumps out at anyone:

Setting the uniforms:

const graphModelViewMatrix = new Matrix4().multiplyMatrices(
  this.camera.matrixWorldInverse,
  this.world.matrixWorld // coordinate system of the 1st pass geometry
);
const graphNormalMatrix = new Matrix3().getNormalMatrix(modelViewMatrix);

In the post-processing fragment shader:

uniform highp sampler2D depth;
uniform float cameraNear;      // this.camera.near
uniform float cameraFar;       // this.camera.far
uniform mat4 cameraProjection; // this.camera.projectionMatrix
uniform mat4 graphModelViewMatrix; // 
uniform mat3 graphNormalMatrix;

// ... ray-tracing runs for this fragment, finding a point at vec3 p

vec4 pNDC = graphModelViewMatrix * vec4(p, 1.);
float z = (cameraProjection * pNDC).w;
float zDepth = (z - cameraNear) / (cameraFar - cameraNear);
float d = texture2D(depth, vUv).x;
if (zDepth < d) { // set the fragment color for p

Or, more fundamentally, when reading a value from a depth texture, how do I convert that back to a z value in the coordinate system which wrote it?

[gilded charm]

vec4 projPt = (cameraProjection * pNDC);
float z =projPt.z / projPt.w; ?

[finite nexus]

That doesn't work. I haven't tried instrumenting to read out values in the depth texture yet, but to put some numbers on the uniforms going into the shader, doing the calculations on a single point, for a distant camera with a small fov:

near/far:  46.53572517978149 53.46427482021851
p = (0,-2,0) give pNDC:  -1.111140466039204 -0.6363792902864169 -48.46364448657717 1 
projectionMatrix * pNDC:  -18.51808497477784 -2.6710886067222925 -18.709998197271716 48.46364448657717

and for a closer camera with a large fov (after a dolly zoom which preserves size in the middle depth)

near/far:  4.535725179781489 11.464274820218511
p = (0,-2,0) give pNDC:  -1.111140466039204 -0.6363792902864169 -6.463644486577167 1
projectionMatrix * pNDC:  -2.962893595964454 -0.4273741770755667 -0.08360886257990074 6.463644486577167

[gilded charm]

🔧 Issues

1. Incorrect projection depth calculation
   You're doing this:

vec4 pNDC = graphModelViewMatrix * vec4(p, 1.);
float z = (cameraProjection * pNDC).w;

You’re extracting z from the .w component of a projected vector, which is incorrect.

You typically want to calculate gl_FragCoord.z-equivalent depth (after perspective divide). That means looking at .z / .w after projection.

✅ Correct approach:

vec4 clip = cameraProjection * graphModelViewMatrix * vec4(p, 1.0);
float ndcDepth = clip.z / clip.w; // range: [-1, 1]
float zDepth = (ndcDepth * 0.5 + 0.5); // range: [0, 1]

2. Incorrect linear depth conversion

float zDepth = (z - cameraNear) / (cameraFar - cameraNear);

This assumes z is linear depth, but the depth buffer stores nonlinear depth unless you wrote linear depth manually.

If your depth texture is from a default depth buffer, you need to linearize it for correct comparison:

float d = texture2D(depth, vUv).x;
float linearDepth = cameraNear * cameraFar / (cameraFar - d * (cameraFar - cameraNear));

[gilded charm]

Or better:

float linearizeDepth(float d) {
    return (2.0 * cameraNear * cameraFar) / (cameraFar + cameraNear - d * (cameraFar - cameraNear) * 2.0);
}

3. Matrix assumptions
   graphModelViewMatrix seems like a world-to-camera matrix (i.e. viewMatrix * modelMatrix), but you should make sure you're matching the same space used to write depth.

If the depth texture is from a screen-space render, you must match projection and modelView exactly.

✅ Corrected sketch:

vec4 clip = cameraProjection * graphModelViewMatrix * vec4(p, 1.0);
float ndcZ = clip.z / clip.w;
float zDepth = ndcZ * 0.5 + 0.5;

float d = texture2D(depth, vUv).x;
// Optionally linearize `d` if needed
if (zDepth < d) {
    // set fragment color, p is visible
}

Let me know if your depth texture is in linear or non-linear space and how it was generated (e.g., via render target, shadow pass, or default framebuffer), and I can tailor the logic more precisely.

(from chatGPT4o)

[finite nexus]

That looks helpful, I'll try that. (How much of that is chatGPT4o advice?) I'm pretty sure the depth texture is linear. It is from this, in which near everything uses the defaults:

   this.mainRenderTarget = new THREE.WebGLRenderTarget(
     this.canvas.width,
     this.canvas.height,
     {
       colorSpace: THREE.LinearSRGBColorSpace,
       samples: 4
     }
   );
   this.mainRenderTarget.depthTexture = new THREE.DepthTexture(
     this.canvas.width,
     this.canvas.height
   );

[gilded charm]

I pasted your snippet into gpt4o and asked if it saw any issues.

[finite nexus]

Confirmed!

      vec4 clip = cameraProjection * graphModelViewMatrix * vec4(p, 1.);
      float ndcZ = clip.z / clip.w;
      float zDepth = ndcZ * 0.5 + 0.5;
      if (zDepth < d) {

works beautifully. I did not understand what I was doing at all. That was extraordinarily helpful, particularly impressive by just looking at an incorrect snippet without a working fiddle. Thank you!!!

[gilded charm]

llms are getting goood.. and gpt4o knows threejs pretty well (but its knowlege is about a year or 2 behind current)

[finite nexus]

I am unused to LLM's being so helpful when I'm confused. Color me impressed.

[gilded charm]

I've been doing graphics programming for >30 years if its any consolation. 🙂
For something like this, where I didn't write the code, its easier to just paste into gpt and ask. I could have probably figured it out if you had provided a codepen or something that ran.. but hey! we're in the future now. 😄 (and even if you had provided a codepen, i probably would have just asked gpt:
"Give me glsl to reconstruct depth from NDC"

[finite nexus]

Nice! (I've been doing graphics programming for >30 years, also, with THREE being my 10th 3d API over that time, but it always feels like I have to relearn everything anew each time.) (Walk down memory lane: https://x.com/RonAvitzur/status/1452699865868222464) I've been resisting the lure of coding LLMs so far, but this may convince me to give them a try.

[gilded charm]

rad 😄 keep your expectations low.. but prepare for a few surprises 😄

ahh software renderers....

[finite nexus]

I didn't include the code to linearize d. It seems to work without it, but if that is a small correction, I may be overlooking a problem. How do I tell if my depth texture needs linearizing? Is it non-linear by default?