WGSL Frag shader behaving very strangely

[atomic hare]

Hey, Im trying to make a shader akin to this video: https://youtu.be/nYch_TIkq6w?t=482 . The TLDW is: Take two textures, one a source with RG pixel values that are UV coordinates to a second texture, which is the palette map. For instance RGBA value of (13,1,0,1) would map to the pixel position x=13 y = 1 in the second map/palette texture.

I tried implementing this in WGSL like so, but I get very strange behaviour, and usually end up getting rendered pixels that are all 0,0,0., or all map to the 4 top left pixels in my palette, indicating that the palette_uv values are incorrect. The sampled values inside uv_map are correct, bc when i return them, the image is the same as my source image.

a minimal working example is here: https://github.com/RaminKav/test_shaders

@group(1) @binding(0)
var source_color_texture: texture_2d<f32>;
@group(1) @binding(1)
var source_texture_sampler: sampler;
@group(1) @binding(2)
var lookup_color_texture: texture_2d<f32>;
@group(1) @binding(3)
var lookup_texture_sampler: sampler;


@fragment
fn fragment(
    #import bevy_sprite::mesh2d_vertex_output
) -> @location(0) vec4<f32> {

    let uv_map_dims = vec2<f32>(textureDimensions(source_color_texture));
    let palette_dims = vec2<f32>(textureDimensions(lookup_color_texture));

    // snap value algorithem to remove weird fuzzy pixels/interpolation
    let uv_map_uv = uv - (uv % (1f/uv_map_dims)) + (1f / (uv_map_dims * 2f));

    let uv_map = textureSample(
        source_color_texture,
        source_texture_sampler,
        uv_map_uv
    );

    let palette_uv = ((uv_map.rg * 255f) + vec2<f32>(0.5)) / palette_dims;

    let color = textureSample(
        lookup_color_texture,
        lookup_texture_sampler,
        palette_uv
    );

    return vec4<f32>(color.rgb, uv_map.a);
}

Ive tried playing around with the math in the let palette_uv = ... line, but i can never get the result I want. Sometimes I can see some of the colors in my palette show up, but never the correct result.

[queen quest]

maybe use textureLoad() instead of the sampler

[atomic hare]

im actually trying that now, gonna see if it works

[queen quest]

also not sure what uv_map.a actually is, maybe just try setting the alpha to 1f

[atomic hare]

its so it only renders pixels that are not transparent in the source

but in this case, all source pixels have a = 1

but i can try that anyway

[queen quest]

also might be worth making the color "map" he shows in the other video so you can see whats mapping to what

[atomic hare]

So i managed to resolve it... as i suspect, I needed to convert Linear values to sRGB ones before using them on the second texture as a UV coordinate!

let uv_map_dims = vec2<f32>(textureDimensions(source_color_texture));
    let palette_dims = vec2<f32>(textureDimensions(lookup_color_texture));

    let uv_map_uv = uv - (uv % (1f/32f)) + (1f / 64f);

    let uv_map = textureSample(
        source_color_texture,
        source_texture_sampler,
        uv_map_uv
    );

    // S = sRGB, L = Linear 
    // ((S+0.055)/1.055)^2.4
    // 1.055×L^1/2.4 − 0.055
    var u: f32;
    var v: f32;
    if (uv_map.r<= 0.00313)
    {
         u = uv_map.r * 12.92;
    } else {
     u = (1.055 * pow(uv_map.r, 1./2.4)) - 0.055;
    }
    if (uv_map.g<= 0.00313)
    {
         v = uv_map.g * 12.92;
    } else {
     v = (1.055 * pow(uv_map.g, 1./2.4)) - 0.055;
    }
 
    let palette_uv = vec2<f32>(u*255f + 0.5, v*255f + 0.5) / palette_dims;


    let color = textureSample(
        lookup_color_texture,
        lookup_texture_sampler,
        palette_uv  
    );

    return vec4<f32>(color.rgb,1f);```

What i dont understand is, why did i need to do this. Is there a way i can specify the second texture to be linear?

[queen quest]

did textureLoad not work for you? I had a similar issue with sRGB but i was using Image::from_dynamic cuz i was making my image from data, but i dont think thats the case here.

[atomic hare]

I dont think so, i couldnt get it to work right. This is what i used

    let uv_map_uv = uv - (uv % (1f/32f)) + (1f / 64f);
    

    // This one produced values that looked right when i returned them
    let uv_map = textureLoad(
        source_color_texture,
        vec2<i32>(i32(uv_map_uv.x*uv_map_dims.x),i32(uv_map_uv.y*uv_map_dims.y)),
        0
    );

    //exp:  10/255 * 255 = 10
    let palette_uv = uv_map * 255f;

    // integer coordinate goes into the map, but i get all 0,0,0 pixels
    let color = textureLoad(
        lookup_color_texture,
        vec2<i32>(i32(palette_uv.x),i32(palette_uv.y)),
        0
    );

    return vec4<f32>(color.rgb,1f);

[queen quest]

you dont need the "half pixel" thing with texture load, b/c it gets the value at the texel value at the location, thats why it asks for a i32 rather than a float, so think of it like a 2d array

im gonna assume that your uv goes from 0 to 1, so you just need to multiply by the texture dimension

[atomic hare]

do you mean uv - (uv % (1f/32f)) + (1f / 64f); this part?

[queen quest]

yeah

[atomic hare]

ok let me try

i was getting correct outputs from uv_map though, which is why i was confused

yeah still all black output

I suspect its an issue with my second textureLoad call? am i correct to expect uv_map is 0-1 representing the sRGB value of the pixel? something like 10/255 if the R value was 10?

[queen quest]

maybe try changing the image format to rgba instead of srgba? when you load the image initially

[atomic hare]

Ahh, this is the part i couldnt figure out how to do. In my test im simply doing this to get my image:

CustomMaterial {
            source_texture: Some(asset_server.load("source_head.png")),
            lookup_texture: Some(asset_server.load("lookup-head.png")),
        })

Should I be creating an Image struct manually and adding it to the Image resource? I think I saw somewhere do something like this

this probably has some default behaviour in how it loads the image, which is causing my issues?

[queen quest]

not sure what the default for loaded images is, so you could change it to the image asset resource

when you made the images, did you save them as rgba png? depending on the software you used it might add that to the metadata of the image?

[atomic hare]

ahh, i used aseprite and just hit export PNG, ill take a look to see if there are any options for that, but i dont remember.

the metada for both source and palette images says color space: RGB, color profile: sRGB IEC61966-2.1

[queen quest]

not sure why its on sRGB 🤷

if you switch to using the asset server you can print the texture_descriptor.format to see what it is

the default is wgpu::TextureFormat::bevy_default() -> Rgba8UnormSrgb when a new image is made, but im not sure what data is made when the image is loaded

[atomic hare]

ahh, so i can change it here, but these are the only values i get to pick from lol

[queen quest]

what does none do?

[atomic hare]

ahh, am i not using the asset server, when i call asset_server.load()?

[atomic hare]

let me see

lmao so None literally removed the metadata for color profile

not sure what that means exactly

[queen quest]

i think that just loads the image. but if you did
mut textures: ResMut<Assets<Image>>, in the setup and inside textures.add(image); you can use that in systems, so its not necessary, you can also just load the image twice just for this debugging purpose

[queen quest]

so i did a test on a png and it does load it in srgb format

[atomic hare]

The None setting, or the sRGB profile setting? Thanks for testing that, i cant get around to testing until after work