Calculating Screen Bounds of a Doorway (rectangle)

[winter sierra]

Given the 4 corners of a doorway in world space and a Camera, how would I go about calculating the tightest screen/viewport space rectangle I can wrap around the door? I'm implementing a modular building system where I only want to generate a room behind a door if the door is visible in the players view. The reason I need the bounds is that I can figure out whether a door on the next room can be seen through the first door (which simplifies to the intersection test between the two bounds). In edge cases it is totally fine the bounds to be too large but in no case should the bounds be smaller than the area the doorway actually takes on the screen because it can cause the player to see outside the generated area.

The approach that I tried was to convert each of the 4 corners into the viewport space of the camera (WorldToViewportPoint) and figure out the minimum and maximum values of each point on the XY plane. That worked wonderfully on most cases, but when I moved very close to the door, everything fell apart. From what it looks like the perspective projection seems to flip points behind the camera from left to right and from up to down. I could detect when the points are behind the camera by checking the Z coordinate of the projected (viewport) point but the problematic edge case is the one where only some of the points are behind the camera. For example, if I move very very close to the door and look straight up, the 2 upper corners of the door are in front of the camera and the 2 lower corners are behind it which causes a case where the returned bounds can be insufficiently small to capture the visible door.

The idea of finding a screen bounds of a rectangle in world sounds like there should be a relatively straight forward exact solution but I cannot think of one. Any ideas are appreciated.

[past nacelle]

Did you try screen space instead of view space? It might not have the inversion issue.

[winter sierra]

Screen space seems to suffer from the exact same problem. I think it might be some sort of inherent property of the perspective projection where the camera position is similar to a focal point where lines of same screen/viewport coordinate meet

[winter sierra]

I also thought I could fix these issues by taking the quad in the viewport space and do a boolean operation with the view frustum (which would just be a simple cuboid) but the viewport/screen space points seem to get so messed up in the edge cases that it would not be particularly useful either (that could still lead to underestimating of the bounds) + the projected points would not even form a flat quad which would make the boolean operator quite hard.

I am currently trying to find out how rasterizers choose which pixels are within a triangle and the screen to know which needs to be processed by the fragment shader. I haven't found any particularly good resources on the topic of clipping in rasterization yet but I think the idea should work the same. I could treat the rectangle as two triangles, transform them into a viewport/screen space, do the clipping and then iterate over the clipped point.

[winter sierra]

I'm also wondering whether the fact that WorldToViewportPoint returns the Z coordinate as the actual distance from the camera and not in the regular 0 to 1 range (near plane = 0, far plane = 1 or something similar) might be messing up with things

[winter sierra]

That might actually be big part of the problem since WorldToViewportPoint returns XY values which are projected using the perspective projection but the Z value is linear distance. I should probably try to transform the points into actual clip space positions with proper Z values and see if that gets me anywhere

[winter sierra]

Maybe it should be less surprising, but it seems like clip space is a great space to do clipping in (since it is the space right before perspective is applied). It turns out the only thing I need to do, is to transform all the corners into clipping space (by multiplying by the VP matrix) clipping the polygon they form in all the 6 directions using the Sutherland-Hodgman algorithm or similar, and then transforming all the points into normalized coordinates (which the X and Y represent in WorldToViewportPoint) by doing the perspective division (and finally finding the mins and max of them for the bounds). This I believe should give me the perfect bounds in all cases. The only unfortunate detail is that the clip space is a 4D space and I'm having tough time wrapping my head around it, but I guess it's the best we have.