Portal 3: No Render Targets Edition | Garry's Mod | Page 1

fair lake Nov 5, 2024, 7:25 PM

#

@deft mulch

deft mulch Nov 5, 2024, 7:26 PM

#

ty

fair lake Nov 5, 2024, 7:26 PM

#

So the way that portals work is that they do shockingly little "trickery"

#

When rendering the view through a portal, they calculate the player's camera position and angle relative to the portal entrance and then use that same position and handle relative to the portal's exit and render the view from that position

deft mulch Nov 5, 2024, 7:28 PM

#

ye I get that part of it

fair lake Nov 5, 2024, 7:28 PM

#

The stencil is really only used to crop that rendered view so that it only draws on top of the portal entrance

deft mulch Nov 5, 2024, 7:28 PM

#

yeah

#

I already have the general idea of what I need to do

fair lake Nov 5, 2024, 7:28 PM

#

So what's the problem

deft mulch Nov 5, 2024, 7:28 PM

#

flip the camera to point out the other side of the other portal + a clipping plane to avoid any issues

#

the issue is stencils

#

I cant crop the rendered view

#

when I try to mask the result of renderview with anything it just results in all black

deft mulch Nov 5, 2024, 7:45 PM

#

@fair lake basically (sorry if bugging just summarizing)

I know how to do the camera translation, I know how to make it not render stuff in front of the camera and behind the portal surface, i am pretty much ready... except that I dont know how to call RenderView, then stencil mask the result

I dont want to use rendertargets because

no way in hell I'm getting recursion with those (altho I prob wont with stencils either, we will see ig)
video memory concerns, rendertargets cant be freed
no antialiasing at all

#

and from what I can tell renderview will fuck up the stencil buffer

#

so unless theres some way to manipulate that to not be the case

fair lake Nov 5, 2024, 7:47 PM

#

I’ll admit I’m not sure if RenderView messes with Stencils

#

I would test it with something real simple

#

Draw a nice, simple rectangle on your screen and clip the view to it using stencils

#

A nice proof of concept

deft mulch Nov 5, 2024, 7:48 PM

#

I basically did that, using the stencil mask example i think you made

#

yk the one with like

#

a black and blue rectangle

fair lake Nov 5, 2024, 7:48 PM

#

Yeah, I made that

deft mulch Nov 5, 2024, 7:48 PM

#

I basically tried to take that and replace the part being masked with a renderview call

#

and it was just all black

fair lake Nov 5, 2024, 7:48 PM

#

Post the code

deft mulch Nov 5, 2024, 7:49 PM

#

I'll post it when I get to my pc

#

give me a few mins

deft mulch Nov 5, 2024, 8:24 PM

#

@fair lake

AddCSLuaFile()

ENT.Type = "anim"
ENT.Base = "base_anim"

ENT.Category     = "Portal Tests"
ENT.PrintName    = "Test 1: Camera"
ENT.Purpose      = "fuck"
ENT.Instructions = ""
ENT.Spawnable    = true

local removeTypes = {PhysCollide = "Destroy", Mesh = "Destroy"}

function ENT:AddSubcomponent(name, target)
    if not self.Subcomponents then self.Subcomponents = {} end
    if self:HasSubcomponent(name) then self:RemoveSubcomponent(name) end
    self.Subcomponents[name] = {target = target, remove = removeTypes[type(target)] or "Remove"}
    return target
end
function ENT:GetSubcomponent(name)
    if not self.Subcomponents then return end
    return self.Subcomponents[name].target
end
function ENT:HasSubcomponent(name)
    if not self.Subcomponents then return false end
    return self.Subcomponents[name] and IsValid(self.Subcomponents[name].target)
end
function ENT:RemoveSubcomponent(name)
    if not self.Subcomponents then return end
    local comp = self.Subcomponents[name]
    if not IsValid(comp.target) then return end
    comp.target[comp.remove](comp.target)
end
function ENT:RemoveAllSubcomponents()
    if not self.Subcomponents then return end
    for key in pairs(self.Subcomponents) do self:RemoveSubcomponent(key) end
end

function ENT:Initialize()
    self:SetSolid(SOLID_VPHYSICS)
    self:SetMoveType(MOVETYPE_NONE)
    self:SetCollisionGroup(COLLISION_GROUP_PLAYER)
    self:PhysicsInit(SOLID_VPHYSICS)
    self:SetModel("models/hunter/plates/plate2x2.mdl")

    if SERVER then
        local p = ents.Create("prop_physics")
        p:SetModel("models/dav0r/camera.mdl")
        p:SetPos(self:LocalToWorld(Vector(0, 0, 50)))
        p:Spawn()
        p:GetPhysicsObject():EnableMotion(false)
        self.Camera = self:AddSubcomponent("Camera", p)
        self:SetNWEntity("Camera", self.Camera)
    end
end


function ENT:OnRemove()
    self:RemoveAllSubcomponents()
end

function ENT:Draw()
    local camera = self:GetNWEntity("Camera")

    self:DrawModel()

    render.SetColorMaterial()
    render.DrawBeam(self:GetPos(), self:LocalToWorld(Vector(0, 0, 100)), 2, 0, 1, Color(100, 150, 255))
    render.DrawBeam(camera:LocalToWorld(Vector(0, 3.6, -4)), camera:LocalToWorld(Vector(100, 3.6, -4)), 2, 0, 1, Color(100, 150, 255))
end

local IN_RENDER = false

hook.Add("PostDrawOpaqueRenderables", "oakvcbn", function(origin, angles, fov)
    if IN_RENDER then return end

    cam.Start3D(origin, angles, fov)
    for _, v in ipairs(ents.FindByClass("part1_cameratest")) do
        local camera = v:GetNWEntity "Camera"
        render.SetStencilEnable(true)

        render.ClearStencil()

        render.SetStencilTestMask(255)
        render.SetStencilWriteMask(255)

        render.SetStencilPassOperation( STENCILOPERATION_KEEP )
        render.SetStencilZFailOperation( STENCILOPERATION_KEEP )

        render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_NEVER )

        render.SetStencilReferenceValue( 9 )
        render.SetStencilFailOperation( STENCILOPERATION_REPLACE )

        v:Draw()

        render.SetStencilFailOperation( STENCILOPERATION_KEEP )
        render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )

        cam.Start2D()
        local scrw, scrh = ScrW(), ScrH()
        IN_RENDER = true
        render.RenderView{
            origin = camera:GetPos(),
            angles = camera:GetAngles(),
            drawviewmodel = false,
            drawhud = false,
            x = 0,
            y = 0,
            w = scrw,
            h = scrh
        }
        cam.End2D()
        IN_RENDER = false
        render.SetStencilEnable(false)
    end
    cam.End3D()
end)```

#

ignore the hackiness of all of this

#

its only the stencil part

#

if you disable stencils it renders fine (ie. renders from the camera (the entity creates one) position/angles) to the full screen

#

but with stencils

#

I was basically just trying to get a basic asf "monitor" working

#

as a proof of concept of stenciling renderview result

fair lake Nov 5, 2024, 8:31 PM

#

You know the part there where you're doing some for loop business

#

Don't

#

Try to get it working just one time

#

Then scale it up

#

Expose yourself to one problem at a time

#

This nonsense is complicated enough that those kinds of things can make it nearly impossible to keep track of what's going on and what might be causing an issue where

deft mulch Nov 5, 2024, 8:37 PM

#

fair lake You know the part there where you're doing some for loop business

I mean to be fair there is only one entity out rn

#

so evne if I remove that

#

I'm only doing it once anyway

fair lake Nov 5, 2024, 8:37 PM

#

Fair enough

#

Why are you surrounding that in cam.Start3D ?

#

and why are you doing cam.Start2D when you render the scene view?

#

Is there a particular reason?

deft mulch Nov 5, 2024, 9:50 PM

#

fair lake Why are you surrounding that in `cam.Start3D` ?

I was doing that before when I was using old hooks

#

I don't think it matters in this case

#

I could try removing it but I really doubt it would change anything, because disabling the stencil makes it all work, and thats only there because I was previously using the render hooks that dont have a context set up

#

as far as cam2d, I thought I'd need to do that to render it to the screen

#

do I not need to..?

fair lake Nov 5, 2024, 10:54 PM

#

deft mulch do I not need to..?

Nah it’s gonna render without that

deft mulch Nov 5, 2024, 10:57 PM

#

fair lake Nah it’s gonna render without that


hook.Add("PostDrawOpaqueRenderables", "oakvcbn", function()
    if IN_RENDER then return end

    for _, v in ipairs(ents.FindByClass("part1_cameratest")) do
        local camera = v:GetNWEntity "Camera"
        render.SetStencilEnable(true)

        render.ClearStencil()

        render.SetStencilTestMask(255)
        render.SetStencilWriteMask(255)

        render.SetStencilPassOperation( STENCILOPERATION_KEEP )
        render.SetStencilZFailOperation( STENCILOPERATION_KEEP )

        render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_NEVER )

        render.SetStencilReferenceValue( 9 )
        render.SetStencilFailOperation( STENCILOPERATION_REPLACE )

        v:Draw()

        render.SetStencilFailOperation( STENCILOPERATION_KEEP )
        render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )

        local scrw, scrh = ScrW(), ScrH()
        IN_RENDER = true
        render.RenderView{
            origin = camera:GetPos(),
            angles = camera:GetAngles(),
            drawviewmodel = false,
            drawhud = false,
            x = 0,
            y = 0,
            w = scrw,
            h = scrh
        }
        IN_RENDER = false
        render.SetStencilEnable(false)
    end
end)```

#

still the same

#

#

if I disable stencil rendering before calling RenderView

#

#

I get this

#

@fair lake as a potential last resort, i was thinking about performing a renderview to a rendertarget but then stenciling that vs. doing some UV magic operations

#

only downside is antialiasing atp

#

but it could be one rendertarget then

#

since id just recycle a single one

#

yeah I got this with a single rendertarget

#

downside i cant reuse it with this approach after all

#

so yeah idk anymore

fair lake Nov 6, 2024, 1:03 AM

#

Why can't you re-use it?

deft mulch Nov 6, 2024, 2:00 AM

#

fair lake Why can't you re-use it?

because I have to render it in a separate place

#

local IN_RENDER = false
local tex = GetRenderTarget("Test", 1920, 1080)
local mat = CreateMaterial( "ExampleRTwithAlpha_Mat", "UnlitGeneric", {
    ['$basetexture'] = tex:GetName(),
} );

hook.Add("RenderScene", "bleh", function(ep, ea, fv)
    for _, v in ipairs(ents.FindByClass("part1_cameratest")) do
        local camera = v:GetNWEntity "Camera"
        render.PushRenderTarget( tex )
        cam.Start2D()
        local scrw, scrh = ScrW(), ScrH()
        render.RenderView{
            origin = camera:GetPos(),
            angles = camera:GetAngles(),
            drawviewmodel = false,
            drawhud = false,
            x = 0,
            y = 0,
            w = scrw,
            h = scrh
        }
        cam.End2D()
        render.PopRenderTarget()
    end
end)

hook.Add("PostRender", "oakvcbn", function(ep, ea, fv)
    if IN_RENDER then return end

    for _, v in ipairs(ents.FindByClass("part1_cameratest")) do
        local camera = v:GetNWEntity "Camera"

        IN_RENDER = true
        
        IN_RENDER = false
        
        render.SetStencilEnable(true)

        render.ClearStencil()

        render.SetStencilTestMask(255)
        render.SetStencilWriteMask(255)

        render.SetStencilPassOperation( STENCILOPERATION_KEEP )
        render.SetStencilZFailOperation( STENCILOPERATION_KEEP )

        render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_NEVER )

        render.SetStencilReferenceValue( 9 )
        render.SetStencilFailOperation( STENCILOPERATION_REPLACE )
        
        cam.Start3D(ep, ea, fv)
        v:Draw()
        cam.End3D()

        render.SetStencilFailOperation( STENCILOPERATION_KEEP )
        render.SetStencilCompareFunction( STENCILCOMPARISONFUNCTION_EQUAL )
        
        cam.Start2D()
        surface.SetDrawColor(255,255,255,255)
        surface.SetMaterial(mat)
        surface.DrawTexturedRect(0,0,1920,1080)
        render.SetStencilEnable(false)
        cam.End2D()
    end

    return true
end)```

#

excuse the mess

#

I think the issue is like

#

because renderview will fuck with all the shit ive set up for stencils already

#

and probably clears the stencil buffer

#

basically theres something in RenderView causing stencils to give me this headache

deft mulch Nov 7, 2024, 8:52 PM

#

@dense tangle I think something would have to be added in RenderView for this to be possible, the friend said
"render.renderview clears the stencil and depth buffers of the RT before rendering the scene". I dont know if you can maybe clone the stencil/depth buffer and then restore them after RenderView? Not sure how this is solved in graphics land

#

Basically the issue being; cant render a RenderView during stencil operations

#

since RenderView is gonna fuck with them

fair lake Nov 7, 2024, 8:53 PM

#

There are existing requests in GitHub for functions to copy the stencil and depth buffers from Render Targets

#

Or, rather, a single existing request

deft mulch Nov 7, 2024, 8:53 PM

#

Yeah but not just render targets, I'm talking main screen

#

Which ik is just a fancier RT but

#

I wanna make sure it's clear

fair lake Nov 7, 2024, 8:54 PM

#

The main screen isn't even a fancy Render Target

deft mulch Nov 7, 2024, 8:54 PM

#

I dont want to use render targets for this since they dont support anti aliasing

fair lake Nov 7, 2024, 8:54 PM

#

It's just a Render Target

#

Though, it does support anti-aliasing

deft mulch Nov 7, 2024, 8:54 PM

#

I know but fancy in the sense of "at least it has antialiasing" lol

fair lake Nov 7, 2024, 8:54 PM

#

So in that sense it's fancy

deft mulch Nov 7, 2024, 8:54 PM

#

ye

fair lake Nov 7, 2024, 8:54 PM

#

Some day, if we're good, we might get antialiasing on all kinds of Render Targets

deft mulch Nov 7, 2024, 8:55 PM

#

That wouldnt entirely solve my problem but would be very nice to get too

#

I dont want to use render targets for VRAM concerns too, cant destroy them or anything

#Portal 3: No Render Targets Edition