How to add specular highlight to a transparent shader?

[vital ingot]

.

[spice mason]

OK, cool.
Do you want the water to move? Like waves/ripples?

[vital ingot]

it already does that, im simply swapping the normal maps

[spice mason]

Not to distract, and we can continue along these same lines, but I'd like to introduce into the discussion some other factors. For example you'll need seemless normal textures and swapping them out requires you (well the engine) to upload new ones to the GPU each frame. What is often done is some noise and vertex displacement. Such as in:
https://lindenreidblog.com/2017/12/15/simple-water-shader-in-unity/
This doesn't have your specular highlights (yet) but it could.

[vital ingot]

if this is much more performant friendly than i guess its a far better solution

[spice mason]

Well, it may not be the effect you want though. If you want flat water, you can have that.

But if you want it to "wave" around rocks or whatever, it will have to change where the pixels are drawn and that's vetex displacement.

[vital ingot]

question if it will provide near the same visual fidelity as the hand drawn one, saddening that i drew them all for nothing

[spice mason]

You drew the normal map?

[vital ingot]

um yeah, its just pixel art

32x32

[spice mason]

Nice. Maybe we can do both! Don't want to waste effort. I mean we could use noise for vertex displacement and then size up/down your hand drawn ones for specular highlights (like mini flickers on the water).

[vital ingot]

but that would use double the resources

[spice mason]

32x32 isn't much. How many are there? Are they seamless? Do you have a height map related to them, or can you generate one?

[vital ingot]

its just 8 sprites on repeat, and there about 10,000 of them.. but i made it so they animate only when on screen

[spice mason]

10000 what? Quads?
How many actual normal map textures though, unique ones not counting repeated usage?

[vital ingot]

normal map textures? only 8 unique ones

but its repeated on all 10000 tiles with the shader

[spice mason]

How many FPS do you get?

[vital ingot]

106

[spice mason]

ok

And other than not having specular hightlights, is it doing what you want, and is it the look you're after?

[vital ingot]

this is how it looks

[vital ingot]

more or less i guess, i would like it to look more like it does in the editor but thats already a different issue i guess

but the only thing i want is the specular highlight

[spice mason]

I like it 🙂
And nice work on the normals. They look seamless to me.

So let's f around with this and add specular highlights.

[vital ingot]

glad to hear that 😁

[spice mason]

Can you please post for me your current shader file that's working after the fixes so I don't f it up, we'll start from the same file source.

[spice mason]

Thanks, let me load it up. I should probably start unity at this point too. lol. Takes me 10 minutes to make a new project, lol.

[vital ingot]

new project for a shader?

[spice mason]

Meh, sometimes. I've just got VS Code open right now.

I have a scratchpad project for each pipeline that I often use too.

I have your shader loaded in VSCode. Checking stuff out online.
I'm going to go with gut instinct for blinn-phong and not full blown pbr, but we don't have to stick with that if we don't like the results.

[vital ingot]

ill go with what is most cost efficient 😅

[spice mason]

OK.
What type of lights are we dealing with? One main DIRECTIONAL light, or do we have to worry about light attenuation, like from a torch/candle?

[vital ingot]

yeah its not only directional light 😅

[spice mason]

So if it's nighttime, we can just have a directional moonlight and don't have to worry about your guy/player carrying a torch.

Oh, misread

[vital ingot]

exactly the opposite

[spice mason]

We DO have to worry about it.

[vital ingot]

yeah

does it make it more troublesome?

didnt know there was difference between light sources

[spice mason]

Shaders are almost never easy.

[vital ingot]

i did notice that

[spice mason]

Are you in forward rendering mode, or deferred? Do you need to support both?

The reason is...you guessed it...lighting is different in each mode.

[vital ingot]

wait ill check

it on default

didnt change it

[spice mason]

You know what?
We might just want to try this with a surface shader. It does lighting automatically.

Just as an experiment.

And it will support both forward and deferred.

[vital ingot]

but you said its overkill

or its not that?

[spice mason]

You can tell a surface shader to use blinn-phong instead of standard-PBR

😄

[vital ingot]

ok so it is indeed a different thing

[spice mason]

It's a different lighting model.

[vital ingot]

this is what it shows on the camera

[spice mason]

OK

Give me a bit to f around.

[vital ingot]

this in graphics

oh its these

so its forward

whats the difference? between forward and deferred?

[spice mason]

Lighting mostly, and how it does things in what order. A lot to explain. Here, read this while I fire up Unity and screw with stuff: https://docs.unity3d.com/Manual/RenderingPaths.html

[vital ingot]

oh so in my case its better to use deferred, good to know

[spice mason]

Is it? Do you plan on using a lot of lights?...at one time?

[vital ingot]

for particles and other visual effect, from what it says, the trade off isnt too bad

oh but it limits hardware.. mm i see

[spice mason]

Just check the hardware requirements in that article, I'd avoid anything lower than high-end mobile if using deferred, IMO, but maybe not too bad. You'd have to test on your lowest expected device for performance.

What unity version are you using?

[vital ingot]

i see, my goal is to try to make good looking game that will run on most hardware, including my 10 year old laptop

2021.1.27f1

[spice mason]

I have a 2021.3.8 installed, so I'll go with that, it's close. Loading...converting BiRP scratchpad...

I'll slap your shader onto some quads and line them up to be kind of like water, but I don't want to spend time doing 10K of them. 😉 Might be close enough.

That's what you did to make that, right. Bunches of quads/squares.

Flat in a plane

[vital ingot]

yup, i mean they are randomly generated but yeah for our purpose i believe even 4 quads are enough

[spice mason]

Can you give me at least one of your normal maps to test with?

I know you posted one before but I don't feel like digging through all that on the main thread

[spice mason]

Thanks

[spice mason]

OK, digging up a skybox as the last thing before modifying shader, have a sample scene set up.

[spice mason]

OK, I totally get why the scene view is ticking you off, doesn't scale

[vital ingot]

hmm?

[spice mason]

Game view vs scene view differences

[vital ingot]

yes but you mentioned scale

[spice mason]

Yeah, well, the refraction amount (first slider) scales the effect well in edit/scene view, but is virtually useless in game view.

[vital ingot]

true

[vital ingot]

well it is already late in my place, so ill wish you good luck and again thank you very much 😁

[spice mason]

See ya on the flip side...I'm close, just have a wrong vector somewhere.

[spice mason]

A start. It assumes that the specular comes from the main directional light ONLY right now.

[vital ingot]

omg this looks absolutely stunning!!

This is truly incredible, thank you again so much!

[spice mason]

It was fun.
BUT...there's cautions (and no warranties).
Right now it assumes that the main light is a directional light.
And it only pays attention to that one light for the specular.

One thing I was going to experiment with is/was using a Surface Shader to try to duplicate this effect and let Unity deal with the lighting.

[vital ingot]

so making the shader to react to all light sources did not work 😅

[spice mason]

It CAN work, it just hasn't been done yet

But if I can get a Surface shader to do it, that's the easy way out.

[vital ingot]

How do I do that?

Or it's not something so simple? 😅

[spice mason]

IDK I haven't tried it yet, before I shot my mouth (fingers) off

lol

Might be able to use a surface shader, phong model, and do transparent water with distortion.

But it would take a long time to explain over discord, rather than show. But I have a busy day today, but might get a few hours.

[vital ingot]

Sure thing, I'm really grateful for your effort 😁

[spice mason]

It's fun, but I may not have the time right now to make promises....

[vital ingot]

It's alright, you did far more than I could hope, again thank you so much

[spice mason]

For now you can research surface shader examples in the docs.
https://docs.unity3d.com/Manual/SL-SurfaceShaders.html

You can also google around and find out how to read the back-buffer (background) in a surface shader, how to have a custom vert() function to do some calcs in, and maybe you can duplicate the functionality of your current distortion. But surface shaders generate lighting too...it is a code generator that makes a vert/frag shader with a bunch of lighting in it.

And for that reason, I want to experiment with using a surface shader. It generates all the lighting passes and code for you.

[vital ingot]

Alright, I'll try to mess around myself 😅

[spice mason]

Here's a good start at a surface shader.
It will receive shadows (in the opaque queue), accept multiple lights, and has environment reflections to react to the sky IF you want them.
It uses Unity's built-in BlinnPhong for quick lighting.

Since I cannot duplicate your scene, I can't test it well though.
And you'll probably want to turn off shadow casting on your water quads if you haven't already done so.

This doesn't support unity fog yet, I didn't bother to continue to add it in unless you need it.

Good example of a surface shader with a grab pass, and auto generated BlinnPhong lighting.

It happens to be running in the opaque queue, but you can change that near the top. Your other one had:
Tags { "Queue" = "Transparent+1" "LightMode" = "ForwardBase"} at the top, but I just forgot to copy it over to this one. lol
An alternative is
Tags { "RenderType" = "Opaque" "Queue" = "Geometry+1"}
for the opaque queue.

[spice mason]

I went ahead and moved it to the transparent queue, and I added a transparency slider for overall transparency.

Which makes it weird, but fun.

So now you can have two versions.

Really though, the refraction should be enough, the transparency makes for a double vision type of thing I suppose. Interestinly, the alpha fade (transparency) works in the opaque queue too, and you get shadows with that. There's the original tags line still in that "transparent version" above that you can uncomment to keep it in the opaque queue, just comment out the line below it that puts it in transparent. Or you can just override it all on the material's queue property.

[vital ingot]

Wow now it looks more like a sea, though the shadows are slightly strong and the distortion is completely gone 😅

also this happens when i add another light source

[spice mason]

Huh.
Try playing with the ranges in the shader for the sliders to get the distortion back. There's a slider(s) for refraction power, and check the queue that you're in per above discussion. Edit the shader to change the possible range values.

As far as light probes and light sources, that gets complicated in the settings, there's a lot of math behind all that and settings impact things greatly. I don't do any lighting in the shader, it is auto-generated by unity's surface shaders...that was the point of trying them. So whatever it generates is what we have to deal with.
Shadows: Turn shadow casting off for the quads, and as for shadow intensity for received shadows I think that's in lighting settings.

[vital ingot]

alright, well thank you again very much, you helped me greatly 😁

[vital ingot]

@spice mason hello again! may i ask for your assistance once more... 😅

[spice mason]

Sure 🙂

[vital ingot]

so.. as i continued developing the game, i noticed how bleak the water started to look at night in comparison with everything else. so i tried to learn shader coding myself, and well yeah it didnt go anywhere.. the most i actually managed to do is create a basic specular shader only to realize it did not fit the water shader which is vertex and fragment shader from what i understand

so i was searching if there is a way to let other light sources outside the directional light to affect the shader and i did found some but each lacked something specific.. and i tried different approaches and could not improve any of the variants..

first approach is this: https://en.wikibooks.org/wiki/Cg_Programming/Unity/Smooth_Specular_Highlights

but i could not understand how to make it transparent and distorting

so then i found this https://catlikecoding.com/unity/tutorials/flow/looking-through-water/

which is what im trying now and im not sure how to add normal map into its calculation

as you can see here it looks odd without some light reflection on the water from the player

[spice mason]

IDK, you could try adding some small-valued "ambient light" to all pixel calcs. Maybe vary that by depth so that you get less as it gets deeper. Or like he does in the article, add "underwater fog".

Not sure what you're saying "from the player". Does the player have a light source at the player position?

@vital ingot

[spice mason]

That ^^ answer is for this ^ comment.

[vital ingot]

yup the player glows, he has point light

[spice mason]

Oh, a point light. OK, not ambient light (but you might have that too if you have, say, moonlight reflecting around).

So you'll have to factor in player position as a point light, or pass in a point light located at the player.

I'm not sure what your exact question is, since you're following that tut. You'll have to add another light in, do you know how to do that?

[vital ingot]

the thing is there could be many light sources besides the player ones 😅 , how do i let the shader react to them all?

[spice mason]

Well, it isn't easy. That's why I was suggesting some form of surface shader, that auto generates lighting code. BTW - You can study what a surface shader does. You can view the generated code.

Anyway, there's different rendering paths (thinking deferred vs forward, https://docs.unity3d.com/Manual/RenderingPaths.html) and usually you want to have a manually written shader support both methods. But you can optionally decide to support only one of them to make your life easy.

So the first question is what do you want to do?

[vital ingot]

i just want transparent distorted specular shader 😅 , if surface shader can achieve that then im all for it

[spice mason]

Well, that's where we started with the surface shader. But I guess it didn't do what you want. I have to dig up what we came up with, I guess.

My problem was, IIRC, that I can't duplicate your scene conditions.

I just had a test scene with that surface shader.

[vital ingot]

mm the scene is mainly a grid of quad mesh with this texture, put some cube or something underneath to test refraction, and 2~3 point lights above to test specular reflection

[spice mason]

#1016065182314807456 message
OK, so let's start by going back to this surface shader from the post here, what do you want to change?

I'm having a hard time because you're bouncing around on approaches. No offense, I mean that's a great way to learn and a good idea, but it's hard for me to track what way you want to go, and it is hard to give advice too.

Lighting/shadows/refraction in shaders is a bit hard, it's not just some dirt-simple thing.

[spice mason]

That surface shader did do refraction/distortion in my test, and I had two lights and both showed up. It was placed on two quads over some "ground" at some arbitrary distance.

[vital ingot]

if we are talking about the surface shader then the main problem was with outside light sources, look i showed in this vid #1016065182314807456 message

[vital ingot]

im truly sorry 😅 , i am quite bad at explaining myself, though i still do not fully understand how shaders work so that adds up. the only thing i wish is as i said for the shader to distort image underneath it, its with "GrabTexture" if i understand correctly, next is for it to have specular light reflection from any light source, not only directional light, and also for the shader to be transparent

[vital ingot]

yeah i did see its more complicated than making regular diffuse shader, though again my knowledge of it is very little

This is the shader that i really liked #1016065182314807456 message but the only issue is that it only reacts to directional light and ignores every other light source

Here you can see example of both shader, on the left is the old vertex shader and on the right is the surface shader

[vital ingot]

it does do refraction but its kinda different from the one on the left, it looks like it doesnt even use the normal map for refraction

so to sum it up, the left shader is missing ability to react to every light and not only directional light

the right shader is a bit more opaque, it has shadows which i find unneeded, and it has different refraction that does not use normal map

these are the settings i used for each material: the left one

the right one:

[spice mason]

Don't/no-need-to apologize, you did nothing wrong, as I tried to say...I'd do what you're doing too if I were you. You're experimenting. I'm just challenged in keeping up with the changes/versions/different-approaches.

[spice mason]

OK, I'll have to fire up Unity and see.

How far above the ground is your water line? Like Y distance? in those examples. So I can recreate it...looks like a cube underneath.

The shadows can be turned off on the renderer/material/wherever.

Transparency is a slider

Let me dig at the others.

[vital ingot]

umm about 0.1.. the game is mostly flat but thanks to normal map it looks more 3d-ish

underneath is just a 32x32 texture of the ground

[spice mason]

Try a setting of 2 for refraction and 0.01 for refraction power. Just to see what happens. The refraction power slider works in reverse (we should fix that).

This is what I get, but maybe not enough?

Change the top two properties in the shader source file to:

_RefractionPower("Refraction Power", Range(0.001, 2.0)) = 1.0

We might be able to get rid of refraction power entirely.
The -40 to 40 on the refraction worked to change it quite a bit on game view. Or go -80 to 80, whatever.

This is with -80 to 80 range, setting is at -80.

[vital ingot]

it completely warps the cube away, leaving a "ghost" cube in the middle 😅

if i set the refraction to 0, it keeps the cube sort of in place but without the pixelated distortion that comes from normal map and also with the "ghost" cube

[spice mason]

Refraction Power = 0.01

[vital ingot]

and refraction to 40?

[spice mason]

yeah

or 80 (have to edit shader to allow range)

[vital ingot]

now the refraction is gone and only the "ghost" cube shows up

for starters, how to get rid of this ghost cube?

[spice mason]

I think that's due to transparency.

Put overall transparency to 1

Just to see

[vital ingot]

yup now its gone

as its completely opaque

[spice mason]

I think we talked about that above/before. I added transparency for fun. It was weird with ghosting but a "free" effect for doubling up stuff.

All this could be done in the opaque queue, with water a +1 in the queue so the background gets drawn.

[vital ingot]

is there no way to replicate the same refraction from the vertex shader?

[spice mason]

Maybe.
I need to get breakfast....AFK for a bit.

[vital ingot]

sure thing

[spice mason]

It's gonna be weird, number of interpolators may be a problem

IIRC

Also IDK why wherever you got that shader they do some of those things in the vert() the way they do, but never mind that now.

I'm also not going to be responsible for you using code that says copyright valve, all rights reserved. So it's up to you to verify they have released it into the public domain and/or you have some kind of rights to it.

[vital ingot]

hmmm did not consider the last part

[spice mason]

@vital ingotI've got a Surface shader version that I think looks code-similar to the vert/frag from your original code. Interpolators were a problem, it needs 14, so it requires shader target 3.5 which allows for 15 interpolators.
See here for target 3.5 implications:
https://docs.unity3d.com/Manual/SL-ShaderCompileTargets.html

You can give it a try in your setup and let me know. I restored the original code for using vertex colors although I don't think you or I used them. That added an interpolator.

It may not yet be at 100%

We'll see.

[vital ingot]

alright so um the outcome is kinda interesting 😅

but that is only if there is no render texture underneath, and i do actually need render texture for mirror effect

this is how it looks with render texture under

it completely ignores light and refraction, as if its completely transparent, as if there is no shader there

this is with the new shader

this is with the old one

this is very strange because the render texture is under everything

so your shader should atleast work where there is sand texture

yeah the surface shader does not seem to refract at all the quads underneath

this is without the render texture below

specular light does not seem to work either

if i put the shader above some other more complex mesh then yeah it reacts but also in a strange way

[spice mason]

I made a few changes, but it won't be identical...because...if it were, we wouldn't have to make it to begin with. It includes other lights, environmental reflections, and such.

I changed some ranges, and the transparency thing was a bit f-ed up. So another shot at it:

That does this

With these settings

Note that the specular color is NOT black. Try white on yours, or tint to whatever but it can't negate/zero the light color or it won't show up.

You can see both green and red showing up.

I have to hit the hay. Maybe more work can be done mid week.

[vital ingot]

alright! yes now it does work but not completely 😅

by that i mean, it reflects in editor, but not in game mode, unless i add some new shapes

this is in editor

this is in game

alright i figured it out.. this shader does not affect transparent materials 😅

yes so it does not affect transparent materials and render textures

[vital ingot]

i found some interesting shader, it almost achieves the desired effect in terms of refraction and lighting

but there are two issues with it: this is what happens when there is point light

and the second is this: the refraction sort of creates gaps between the quads below it

disabling the refraction fixes it

[vital ingot]

also just to be sure i checked and its a completely free to use shader 😅

[spice mason]

It CAN, I just didn't realize you needed it to. There's a comment in the source for moving it into the transparent queue. Right now it is in the opaque queue, so it is drawn before transparents are drawn.

See the source, and try editing things so it moves into the transparent queue and try it.

[spice mason]

That's a surface shader too, BTW. 😉
They're using a custom lighting model for "glass".
The Surface shader we've been using employs the Blinn-Phong specular model, but could do custom lighting too.

[spice mason]

The valve-specific code from the vert/frag isn't needed in the Surface shader we're writing and can be removed. 🙂

I forgot to delete it, but the compiler is stripping it out.

[vital ingot]

yes now it shows up, however the refraction mainly just duplicates and moves up the texture below instead of refracting like in the glass shader

[vital ingot]

is it possible to completely remove the dark shadow parts? so that it will only show highlights but no shadows. the mesh rendered has shadow recieve and cast off

[spice mason]

If receiving shadows is off, IDK.

You can remove the fallback at the end of the shader, that might do it, but it won't update the depth buffer, but then again if it is in transparent queue it shouldn't do that anyway.

[spice mason]

Any luck removing shadows?

[vital ingot]

no not really, i tried mixing with the glass material but it just doesn't want to work, rewritten it alteast 3 times..

the main issue is not the shadow though (even though i would like to remove it), im trying to implamanent the refraction from the glass material

the current refraction does not work well in my game

[spice mason]

I hope I can spend the necessary several hours this weekend.... 😉 Until then, please excuse the silence.

[vital ingot]

its alright, thank you regardless, meanwhile i will try to achieve it myself 😅

[vital ingot]

I can't believe i actually managed to do it

Removed shadow and fixed refraction

[vital ingot]

@spice mason hello 😁 , may i bother you once more?

[spice mason]

Hi 🙂
Not a bother at all, but IDK if I'll have an answer, will do my best. Go ahead and ask 🙂

@vital ingot

[vital ingot]

wonderful. thank you

so im trying to add POM to my current standard surface shader

but i get strange results

so this is my shader without POM

and this is with it:

now, it looks like the POM works but has no relation to normal and occlusion maps

those maps are like glass on top of the parallax map

also POM does not react to any lighting

[spice mason]

Huh.
This is a bit of a rough read, but it's one of my saved links that might help.
https://www.gamedev.net/tutorials/_/technical/graphics-programming-and-theory/a-closer-look-at-parallax-occlusion-mapping-r3262/

Also I saw some of your posts on the main thread. Remember that normals have to be decoded. There's a macro for that, lemme find it.

[vital ingot]

this is what i managed to do so far:

[spice mason]

Let me stuff this here and come back to it:
o.Normal = UnpackNormal (tex2D (_BumpMap, IN.uv_BumpMap));

[vital ingot]

oh yeah i do that

[spice mason]

I see that. You use a "with scale" version

OK. Hmm.

Does the height map change the pixel's normal? I mean logically.

[vital ingot]

height map does not seem to affect normals at all

[spice mason]

Right, but SHOULD it?

Right now you're "just" reading the normal map. And you also do separate height map calcs.

That might work, depending on your input data/textures. Or it might not, if you didn't allow for it.

[vital ingot]

should the height map change normal? mm no i dont think so, it would complicate stuff

[spice mason]

OK, fine. It's all part and parcel then.

[vital ingot]

i think the best solution would be to sort of parallax the result of normal and occlusion

[spice mason]

And you have an albedo texture, and read "color" but don't use it....right now. Correct?

[vital ingot]

yes its the previous texture befor the POM

[spice mason]

Well, if you don't plan to use it it can be deleted. Otherwise I'd set final color to that to start, rather than (1,1,1,1), I would think.

Anyway, not the main point I guess.

[vital ingot]

yes you can delete it, i left it just in case

[spice mason]

But.... IDK what I'm looking at in your pics. I would have thought it would look different. What do your normal map and height maps look like?

[vital ingot]

the occlusion map is used also for depth

[spice mason]

The last two look normal to me.
What is the first one, color?

Or occlusion? or what?

[vital ingot]

the orange is the maintex

the grey one is both occlusion and depth

and the bluish purple is the normal map

[spice mason]

OK, But you're not using that from what I see. (Color isn't being used, it's read, but not used)

[vital ingot]

then.. how do i get the texture to show up?

the main one

from what i see here they all show up

but used seperatly

[spice mason]

IDK, I'm not sure your code matches your pics. It is confusing me. How does it show up?

you read the color here:
fixed4 color = tex2D(_MainTex, IN.main_uv) * _Color;
that's line 138 in your file you sent me. But you never do anything with that variable....

You continue to use "finalColor" which came from some other calc unrelated to the variable "color".

Maybe I'm missing something.

[vital ingot]

yes this is not used

you can delete it, its from the previous version

before i added POM

[spice mason]

I don't want to delete it, I'm just saying...answering....this question ^^

[vital ingot]

the finalColor is from the POM, it replaces the Color

now, im not sure how to convert also normal and occlusion to it

[spice mason]

I'll have to think about it, now that you provided the code and the textures. Get back to you, because I have to leave right now.

[vital ingot]

sure no problem and thank you 😁

[spice mason]

When I use your shader (unchanged) on a cube, with the albedo set, and the normal set, and the height map set on both the depth and the occlusion... I get this:

[vital ingot]

oh yeah thats because you need to disable compression on those textures

[spice mason]

Which seems different than the pic you're showing me.
Also you're working with others on the main thread, so I don't want to waste my time if you're perusing another path.

[spice mason]

OK, IDK why you're using sprite rather than regular texture 2d,, but you have reasons so I copied your settings:

[vital ingot]

it still does not look correct

did you change compression to none on all of them?

[spice mason]

OK, I'm just letting you know where I'm at and also what I'm seeing.

My GF called and I have to go again, but I'll be back.

Yes, compression is NONE on all of them. One directional light and it is reacting to changes in the direction of the light.

I'm sorry I have to run, juggling many things here.

[vital ingot]

sure no problem 😅 good luck

[vital ingot]

check if filter set to point

[spice mason]

I had that same thought driving back from GF's.... 😉

It does react to lighting though, but probably not the way you intend.

You're trying to create a voxel/cube type of thing?

[vital ingot]

also change the normal scale to 0.25 in the material

[vital ingot]

yeah its completely unrelated to the POM

[spice mason]

do you have a script you're setting the instanced values from?

[vital ingot]

yeah, but the normal scale should be unlocked

[spice mason]

It is flagged as [PerRendererData]

[vital ingot]

you can temporarily remove the [perRenderData]

[spice mason]

so it comes from the Material property block.

Yeah, I know. Or I just set the default to .25 instead of 1.

Meh

I was just asking.

[vital ingot]

i mean i can give you the script but itll require more scripts for it and more componenets

so its best to just remove [PerRendererData] for now

[spice mason]

So it's kind of working....

[vital ingot]

yup, this is the result that i get

you can see now that the normal and oclussion arent related to the POM

its like POM is inside a glass made by normal and occlusion maps

if you remove the normal map and occlusion, youll see POM does not react to lighting at all

[spice mason]

OK, so I've recreated the problem. Now I just need enough screen time to dig in. Thanks for helping recreate it.

[vital ingot]

sure thing 👍

[vital ingot]

any luck so far?

[spice mason]

No, I have funerals to go to and stuff, haven't got to it, sorry.
I do suspect the normals don't match the calcs though. Although you CAN look up normals in a texture, the other way is to calc them somehow from the results, since the "new" surface may not have the same normals as the static texture if you adjust things. All depends on where you got that normal texture and if it matches up well or not.

There is a way to calc normal from depth, but you need 3 samples (the pixel, and two others slightly offset from it).

Also, depth of the pixel should probably be adjusted to match the new-result. But that's slow so if we can avoid it that'd be better. IDK if the lighting calcs even use or care about the depth buffer.

The other question I had was what if a pixel should actually be a "hole" and the skybox/background or other thing shows through it. Like a notch on the edge of the cube.

[vital ingot]

oh wow, im sorry i did not know 😟

[vital ingot]

i tried to play around with it and somehow got to the point where it crashed my unity unless i remove the shader script..

i dont think there is need for anything complicated, i mainly want to displace the result material

so for example i get regular material with normals and occlusion, so now i want to do parallax to this result instead of just doing parallax to only regular texture

[vital ingot]

@spice mason i do not understand why its doing this, now its like seperated layers one on another, ive been wracking my head around this several days and i cant understand what am i doing wrong

[spice mason]

IDK, what changed?

[vital ingot]

tried to rewrite everything but now this also crashes my unity, so i scrapped it aswell..

[spice mason]

I would revert back to the one above where I duplicated your results, then look at the code in the link I gave you whereby they calculated new surface normals or otherwise find a way to calc a normal rather than reading it from the normal texture. The reason is that I suspect the normal texture may not match up to the computed POM and thus we get lighting overlaid onto it f-ing up the results. Maybe.

#1016065182314807456 message
So this article says you can calc a new normal, but I haven't had time to dig into it. You don't need a normal map at all, just a depth map. You may have to research on the net.

[vital ingot]

but then if i do it without normal map, how will it react to light?

[spice mason]

You give it a normal...a calculated one. Have to figure out the calc.

Did you read that article? It says you don't need a normal map, they just use one for convenience. But they'll have to apply an object-to-world transform to it, and their normal map corresponds to their settings for POM. So it's a pre-calculated normal...a cheat.

@vital ingot

[spice mason]

What I just did was output the normal result....it's nearly all blue. On all sides, in all directions. We should end up with a world-space normal put into o.Normal, so we should end up seeing red if pointing along the x axis, green if pointing up, and blue if pointing toward the +z axis, for example.

So you can use a normal map, which is tangent space normals....but it has to correspond to the POM height map's resulting normals.

[vital ingot]

i read it and im not sure how i implement this to a standard surface shader...

[spice mason]

It should work as long as we get the normal correct (well, maybe some other details after that, like the occlusion, but that's for later.)

Did the normal map come from the height map's data? So that the normal for a pixel in the normal map corresponds to the normal for that pixel in the height map? Remember we're generating artificial geometry here.

[vital ingot]

Yes they come from the same place

[spice mason]

Which is where?

What software generated them?

Besides, if you adjust the height with the slider, that will move the geometry, so a pixel normal could go from being a wall with lowest level setting to a floor with higher level setting, and the normals would be, say, pointing to the right if a wall, and pointing up if a floor.

[vital ingot]

Me and a little help from Photoshop RGB seperation

[spice mason]

This is what I get when I rescale o.Normal to 0-1 range. It still looks like tangent space to me.

@vital ingot

Which makes sense from what the code is doing, because you/we unpack the normal and scale it, but we don't convert it to world space.

The vert() side calcs the tangent, bitangent and normal vectors, but it does this thing where it just stuffs some "tangentViewDir" into an interpolator which you use as your ray direction. But I think we need to pass those values and make a transformation matrix so we can convert the normal to world space.

But the thing is that reading a normal map probably won't work if you want to have a height adjustment slider. depends on if you support anything other than pure vertical and pure horizontal, I think.

So before I go off "on a tangent" (Ha!) I want to discuss whether or not you want that scalar for height. Because if you do we'll have to calc the normal, not get it from a normal map.

[spice mason]

OK, added some code to convert it, had to move to shader target 3.5 but now I'm getting what looks like world space normals:

[vital ingot]

im kinda having hard time understanding why normal map is at fault here, but will it give this type of result but in voxel style? https://www.youtube.com/watch?v=2JjjhMW_n9w

[spice mason]

That's a POM result, which is what you're after, so once it's working...sure.

The map may or may not be "at fault"...it depends. If it is pure horizontal and pure vertical, it might work. But otherwise, if you have "angles" like on those rocks in your vid....well....the scalar you have for height scale will change the resulting normal value.

Make sense?

[vital ingot]

so angles is an issue?, well since it needs to look like voxels so it should not have any angles

if i understood correctly

i do wish to retain normal map is it helps to imitate angles that arent actually there

[spice mason]

So do calculations....

[vital ingot]

so that every pixel looking upwards would not look flat, but slightly angled, but not by POM

[spice mason]

"so angles is an issue?"....well, any angle, including vert/horiz are what we need to light it properly. We need the proper resulting angle (normal) for each pixel.

Think about your normal map. Two pixels side by side.

They're both "floors" when viewed from above. Yet with height applied, they have depth, so that when viewed from the camera "eye", there's a wall that has "dropped down" for some magic pixels that need to be generated from the camera view.

[vital ingot]

what if the POM was not affected by normal map? the height did change, but when looked from above it would look angled as if regular normal map

im mainly searching for a good low cost depth illusion without creating additional polygons

is something like this in your opinion, expensive? https://www.youtube.com/watch?v=EmpjCMlB12Q

by expansive i mean, significantly impacts performance

[spice mason]

That's what POM is.

So you're on the right track.

It's a matter of implementation.

I just haven't done "voxel POM" before. But PinkPanzer gave you some example IIRC, but had a lot of artifacts.
I suspect we can sample something and calc a normal for a "wall" that appeared on a voxel due to displacement but isn't in the normal map. It's starting to make my head hurt.

Here's the current version with world space normals from the map

[vital ingot]

its the one that PinkPanzer made?

[spice mason]

No, that was shader toy

this is the one I fixed to use your normal map, but have proper world space normals. But I still don't think it works correctly, "because voxel walls".

[vital ingot]

oh, one sec ill check it out

[vital ingot]

im not a shader expert but.. if we are talking about voxel walls.. wouldn't it be just 4 fixed normal vectors? can't we just use it repeatedly when needed?

[spice mason]

I get your point, and I'm trying to think about it.
But like I said above...
take two side-by-side points in the normal map. Drop the right one down 10 feet. Now look at this cube from the side. There's a ton of points to fill in between those two that are now some vertical wall that appeared, and which wall it is depends on view direction.

[vital ingot]

what does the POM exactly do? it displaces for example those pixels to different height while remaining them flat, but how does it fill between the heights? does it stretch? or does it multiply itself many times like a stack of papers?

[spice mason]

That's what I'm thinking about.
I need to figure out how the height map's changes in exactly a vertical direction (displacement) impacts the tsNormal.

[vital ingot]

how does the one that i provided to you work?

from the looks of it, it duplicates it self and looks like voxel because it replicates the exact same color

[spice mason]

That one reads your normal map at the point where the pixel from the ray cast intersects it.

That what you see in this result, all I did is convert it to world space so Unity lighting calcs work.

But...

[vital ingot]

oh no no i meant the one that i gave you from the start

[spice mason]

Same answer

[vital ingot]

the one that completely ignored normal and occlusion

[spice mason]

? link

[vital ingot]

this one

[vital ingot]

this is the code

[spice mason]

That's the one I'm using, converted to world space normals.
And it doesn't ignore them, it sets occlusion and normal. So I don't know what you mean.

I didn't bother with later stuff since we discussed reverting back to this.

[vital ingot]

hmm, the POM kinda acts strange though

the darker one is the new one

[spice mason]

That's going to be lighting based on world space normals and what you have in your normal map

It will change depending on light angle and view angle.

[vital ingot]

no no thats not what i meant

one sec

its like its not exactly correct perspective depth, but something else

[spice mason]

I'm not seeing what you mean by a difference, other than lighting, but I'll have to play with it I guess. Or maybe find another example of POM textures like those rocks.

[vital ingot]

for example the blue lines represent correct perspective, while the magenta line show this strange like inverted perspective

[spice mason]

IDK, the only code I remember changing is the o.normal's calc into world space. You can compare versions, maybe I forgot something I did a while ago before I could get back to it.

[vital ingot]

but do you see what i mean?

[spice mason]

I'd have to switch shaders in the exact same perspective, not side by side perspective. That results in two different view angles.

Sec

[vital ingot]

found it

the normalize that you added here caused it

now the perspective is correct

[spice mason]

Cool, good work. I do see it....

But it is fixed. Don't remember doing that but it's been a bit.

I changed the default albedo to pink to see where the loop failed to find bottom. There is pink on the cube, so there's something to deal with that too.

[vital ingot]

alright yeah as i expected, the POM simply stacks on itself 300 times

so then.. how do you even make a wall normal vector if its just the edges of upward normal

[spice mason]

right, that's my question too.

In the rock example, they're "just" extruding the rocks higher or lower, but they still have sloped normals.

So our next task is to figure out how to calc that normal. You see, in their example their FIXED-height value matches their calculated normals in their normal map. But in your case, you have this slider and side-by-side normals I think.

[vital ingot]

so in other words its not possible to create fake walls? since its something that does not originally exist in the texture data

[spice mason]

I'm not saying "impossible" but I'm saying "harder".

[vital ingot]

so its basically moves verticies of each pixel up and down, so slopes and angles are much easier in that case

[spice mason]

It's all pre-calced per pixel. fixed height, fixed normals.

But not in your case

And even so, IDK how it handles verticals in their case. Maybe it doesn't, and they only have slopes.

Or maybe it does and I don't understand it.

yet

[vital ingot]

alright, in that case that would mean that we will have to sort of force it to make voxel like result, that will most likely result in perfomance impact...

so maybe lets try to make it regular angled slopes?

generated by normal map

[spice mason]

I thought you required voxels.

But if not, you have what you need I think.

We'll find out.

[vital ingot]

i would love it to be voxels

[spice mason]

Let's try and find some POM normal and height maps.

and try them with your shader

just for kicks

[vital ingot]

but its all mainly design preference, my top priority is performance

[spice mason]

Well, you can calc a new normal with THREE points...so you could sample your pixel and two others with slight offsets. But that's 3 x the raycasting budget.

so bad for perf

[vital ingot]

so what would be the better solution?

[spice mason]

if you want performance you don't want to do 3 ray casts.
but maybe POM is supposed to handle verticals in tangent space, and our implementation doesn't do that well.

[vital ingot]

there are many explanations on how to implement POM with shader graph

but little to none as a shader code

is it possible to translate shader graph to shader code?

[spice mason]

Sure, but POM has inherent issues.
Like steep height changes. Look what I found, this is worth the read:
https://learnopengl.com/Advanced-Lighting/Parallax-Mapping

It's openGL but meh, the concepts apply.

[vital ingot]

this looks like exactly what we need

[spice mason]

It's basically what we already have....I think. Would have to play more.

But note how the height maps and normal maps are, pay attention to the slop of the grade.

See the diff? So that's what we were talking about. And the article says that to deal with steep changes you do multiple samples, like we were discussing above.

Which sucks for performance.

[vital ingot]

can you point exactly what you mean?

[spice mason]

They don't have pure vertical drops

[vital ingot]

how much performance are we talking about?

[spice mason]

IDK, would have to try it and benchmark it.

reads like 2x the raycast budget

but maybe you can reduce iterations to compensate

[vital ingot]

yeah i guess thats what i would do

[spice mason]

I'm back to dealing with things IRL for a bit, so I'll leave you to experiment along the lines we've discussed and let me know how it turns out or what you hit for problems. I want to play with this too when I get more time.

Fun stuff.