Transparent Video Playback

[unkempt kiln]

What's your resolution? The video resolution I mean

[wary jolt]

@unkempt kiln Hello 🙂

My resolution is default 1280x720
but can be scaled in many different ways from phone screens in 21:9 to arcade cabinets in 4:3.

I think my only option is to write a custom control that will set the scale and position of the animated sprite as if it was inheriting from Control

[unkempt kiln]

I don't think that would change anything.
What you want to do is render the video on top of UI elements, while the UI elements show through the video?

[wary jolt]

I want to render the video on top of everything: UI, gameplay, etc.
@unkempt kiln

Godot doesn't current support transparent video without chroma-keying though

[unkempt kiln]

Well, design the UI and everything first as normal.
Then add a CanvasLayer to your root.
Add a Panel node to that CanvasLayer, set it's min_size and position accordingly.
Modify the panel's style to make it completely transparent.
Then add a shader to the Panel in order to render the video.

Don't forget to mix the video with the screen texture inside the shader.

[wary jolt]

Interesting, how would I get the data from an animated sprite onto my panel node using a shader? @unkempt kiln

I dont know if you can communicate across two nodes in a shader -- then again i dont really know a lot about shaders

[unkempt kiln]

You don't need an animated sprite, just load your video into the shader using a texture uniform

[wary jolt]

i can't use the video because it needs to be chromakeyed which gives artefacts like this

and doesn't allow for translucency

[unkempt kiln]

You could make 2 videos, one just normal, without chroma key or anything.
And another one for the transparency mask, since videos don't support alpha channel, you'd have to create a second video just with the alpha, then load both videos in a shader, and use the transparency one as alpha.

Honestly, I don't think using video files is a good idea, only when showing stuff in-game, for instance a TV in-game.
But for overlayed elements like these, nope.
I'd go for a png sequence and convert that into a 3D texture, then iterate through the layers inside the shader using the TIME constant

[wary jolt]

3d texture?

[unkempt kiln]

https://docs.godotengine.org/en/stable/classes/class_texture3d.html

[wary jolt]

I tried doing this by texture packing the animation for a animated sprite

[unkempt kiln]

it's not ideal, you end up with huge textures filling your ram just for a single element

[wary jolt]

what's not ideal, mine or yours?

mine isn't ideal fs I think

How can I create a texture 3d? Also, I see in 4.0 it has compressed texture 3d? Does that mean there won't be any compression in 3.6

[unkempt kiln]

yours

hang on, I'm reading the docs

[wary jolt]

I think the video one sounds maybe better, because i don't really want to write a custom flipbook animation playing shader

[unkempt kiln]

You can use a Texture3D or a TextureLayered, both will have the same functionality.
In the case of a Texture3D, you'd have to use the ImageTexture3D then, since it's meant for images.
you can create one by using it's constructor:
ImageTexture3D.new()
then initialize it using the create() function:
create ( int width, int height, int depth, Format format, int flags=4 )
then yuo can set each layer with the function:
set_layer_data ( Image image, int layer )

Of course, you have to iterate through all the frames, one by one. So import all the frames as transparent pngs, then load them in code one by one and set each layer in a for loop

TextureLayered is the same, but you initalize it by setting it's data property:
{"depth": 0,"flags": 7,"format": 37,"height": 0,"layers": [ ],"width": 0}

overal Texture3D is more practical

[wary jolt]

That definitely seems like more work than an alpha masked video

[unkempt kiln]

Yeah, that's the problem, there's just no other practical solution that's not overly computationally expensive for something like this.

[unkempt kiln]

The problem with the videos is that they will use more cpu power, and maybe more vram, but i'm not really sure

[wary jolt]

I think I can write a custom ffmpeg script to convert the original video into a masked one

[unkempt kiln]

if that's not a problem, then okay

[wary jolt]

eh ¯⁠\⁠_⁠(⁠ツ⁠)⁠_⁠/⁠¯

How would I be able to access one video from another via a shader though?

[unkempt kiln]

you code your shader with two inputs, say:
uniform Texture2D video_color
uniform Texture2D video_alpha

then access both separately in the fragment shader
like:
vec3 color = texture(video_main, UV).rgb
float transp = texture(video_alpha, UV).r

[wary jolt]

Also, wait, if I use this method, I'm not sure it'll work. Because OGV requires solid colors, therefore there needs to be a background color to the main video, therefore if there's any aliasing in the exported animation, then it'll still be present in the final version

it'll mix the background colors into any aliased section

[unkempt kiln]

The background color should be the same as the foreground, in your case, pink, and also contain any shading, just not transparency at all (or what looks like trasnparency)

the transparency video will be a grayscale containing the alpha values

[wary jolt]

Well there's also gold stars, which would then theoretically have a pink border

[unkempt kiln]

They should have a gold border, because it's the color that you want to bleed through the transparency

[wary jolt]

But how would the transparent aliasing work?

because to generate the alpha mask you'd need to I guess capture the aliasing in the alpha mask?

[unkempt kiln]

the shader system loads any texture whilst applying a bilinear filter, so it would not have any aliasing at all

[wary jolt]

look back at this texture

see how it's black and jagged at the edge

it would be pink and jagged

instead of aliased

[unkempt kiln]

That's alpha clipping, it has nothing to do with aliasing, aliasing is the term used to describe those rigged artifacts, but in this case, they're expected, and not really artifacts

When using a shader, the texture would be filtered, having a nice falloff or gradient as it approeaches the extreme border

[wary jolt]

hmmm

this might be the way

Because when exporting it should be aliased in alpha, then, I can extract that aliasing into the mask, then I can re-export the video with enabling the layer that has the color expansion that gets rid of the aliasing, and then the mask will reapply the alpha gradient aliasing

I might not be using aliasing right, but that's the way I'm imagining it.

[unkempt kiln]

I don't really get what you're trying to say, but I guess that's alright

[wary jolt]

Lemme try to illustrate it

it fades to transparent, which is shown by the green artefacts

which would probably be captured by the alpha mask

and gotten rid of

[unkempt kiln]

yes, exactly

but the color should remain pink

as if it didn't have any transparency at all

they it would create a nice pink transparency blending falloff

[wary jolt]

great

thanks so much

can i dm you if i have more questions later about this?

[unkempt kiln]

sure, no problem

[wary jolt]

Trying this now. Praying for myself 🤣

got the alpha masks

[wary jolt]

I'm going to write a reddit post about this after this.

Others deserve not to have to go through this in the way i did 😆

@unkempt kiln I'm not understanding how I could get a Texture2D from videos into the shaders

Am I basically putting one shader on two videos, and then trying to figure out how to have them each output different colors to the texture?

Okay so I need to get the video player shader to communicate to a viewport somehow, and maybe both the videos need to contribute to the same viewport with different, but similar, shaders (ex one copies the rgb to the viewport, the other copies the alpha to the viewport).

[wary jolt]

welp i tried this and it didn't go so well, takes like a minute to process a frame lol

[wary jolt]

next iteration:

(testing values right now)

This doesn't seem to be working

I'm not sure why

heres if you want to try it

Heres that at text so you can copy paste

using Godot;

public class ResultingVideo : TextureRect
{
    public override void _Ready()
    {
    }

    public override void _Process(float delta)
    {
        var alpha = GetNode<VideoPlayer>("../TestAlpha").GetVideoTexture();
        var rgb = GetNode<VideoPlayer>("../TestRGB").GetVideoTexture();

        var shaderMaterial = (ShaderMaterial)Material;
        shaderMaterial.SetShaderParam("rgb", rgb);
        shaderMaterial.SetShaderParam("alpha", alpha);
    }
}

shader_type canvas_item;

uniform sampler2D rgb : hint_white;
uniform sampler2D alpha : hint_white;

void fragment() {
    // Sample the color from the texture
    vec2 uv = FRAGCOORD.xy / SCREEN_PIXEL_SIZE;
    vec4 rgbTex = texture(rgb, uv);
    vec4 alphaTex = texture(alpha, uv);

    // Output the sampled color
    COLOR = vec4(rgbTex.r, rgbTex.g, 255, 255);
}

This should be working but isnt . Thanks for all your help so fa @unkempt kiln

[wary jolt]

note: this has the 255's just to try to demo it and see it overlay bluely on top of everything to prove it works

Verified that assigning the texture property the rgb texture works properly. So in theory it should be working to set the shader param. But it feels liek the shader param isnt being set correctly, I think thats the part that isnt working

This doesn't work either. Getting the texture from the TEXTURE property 🤔

i fuckin got it working

the uv variable didnt work

replaced it wtih UV and it works

holy shit

😄

further proof of working

Thanks a bunch. It works so well. Couldn't have done it without your advice dude :DDD

[wary jolt]

even made a custom object to share with the community 😄

Going to post the scripts as well 🙂

Im literally so happy

I've been working on this issue for months

[wary jolt]

it looks great

[unkempt kiln]

Holy moly! All of this happened while I was sleeping, glad to know you figured it out and got it working!
Congratulations on your endeavors.

[wary jolt]

😁😁😁😁😁😁😁😁😁😁😁

only took 5.5 hours 🤣

[unkempt kiln]

Welcome to gamedev

[ripe bolt]

Im gonna need to do this on godot 4

[wary jolt]

I can help bud it should be the exact same thing

going to post on GitHub today

[wary jolt]

😭