UV mapping doesn't stretch across triangles

[boreal nebula]

Introduction

(you can skip this part)

It's been three days I've been trying to implement a "portal" system, think stuff like mirrors, portals, or the anti-chamber magic cubes (which is actually closer to what I want to make): https://www.youtube.com/watch?v=lFEIUcXCEvI&t=540

I don't know how shaders and stencil buffer work so I took another approach from the video above, I first tried doing my own projection to project exactly the view through the square from the camera and use that as a texture. After many many many attempts and getting a result that is close to what I want but not perfect ( https://assets.octodon.social/media_attachments/files/110/131/577/276/727/398/original/dca92e31c6baf718.mp4 ) I realised what I wanted to do was a projection on a near plane that is tilted and not orthogonal to the camera direction. There is no shorthand for that and it would have involved me messing with 4x4 matrixes (something like this I guess, but not only for the frustum, for the whole projection: http://www.terathon.com/lengyel/Lengyel-Oblique.pdf ), which I considered for a short time but I was like "nah"

Then I took a different approach, the one described in this video/article:
https://www.youtube.com/watch?v=cWpFZbjtSQg
https://tomhulton.blogspot.com/2015/08/portal-rendering-with-offscreen-render.html
It involves rendering everything and then cropping the result using UV-mapping

I arrived again at a result that is close to what I want but not exactly, I tried many different variations, I found the result that I had before with the projection even, but when I did things properly (either calculating things myself or using camera.world_to_ndc) I had a weird artifact: https://assets.octodon.social/media_attachments/files/110/137/465/669/386/893/original/25b308cc270d1ac4.mp4 (ignore the fact that the cube texture looks crispier, I just changed linear interpolation to nearest in between)

.
After some testing I realised the image was deformed at one of the diagonals of the square, and after a lot more testing and trying a bazillion different things, noticing my code was a mess, I tried making a simple code that reproduces my problem and confirmed UV-mapping doesn't work the way I thought it did.

Current Problem: UV-mapping

Here is a simple code to demonstrate my problem. A camera is facing a Quad (two triangles), a texture is mapped on this Quad (this one: https://assets.octodon.social/media_attachments/files/110/137/483/398/681/885/original/24e171530794c3a2.png ), and I change uv-mapping

use bevy::{
    prelude::*,
    render::mesh::VertexAttributeValues
};
use std::f32::consts::PI;

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_startup_system(setup)
        .run();
}

fn setup(
    mut commands: Commands,
    assets: Res<AssetServer>,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>
) {
    //out plane
    let plane_mesh = Mesh::from(shape::Quad::new(Vec2::new(10.0, 10.0)));
    let mut plane_transform = Transform::from_xyz(
        0.0,
        0.0,
        0.0,
    );
    plane_transform.rotate_local(Quat::from_axis_angle(Vec3::X, -PI/2.));
    let plane_mesh_handle = meshes.add(plane_mesh);
    commands.spawn(PbrBundle {
        mesh: plane_mesh_handle.clone(),
        material: materials.add(StandardMaterial {
            base_color_texture: Some(assets.load("checkered.png")),
            perceptual_roughness: 1.0,
            alpha_mode: AlphaMode::Mask(0.5),
            cull_mode: None,
            ..default()
        }),
        transform: plane_transform,
        ..default()
    });

    commands.insert_resource(AmbientLight {
        color: Color::WHITE,
        brightness: 0.5,
    });

    let mut transform_window_plane = Transform::from_xyz(
        0.0,
        15.0,
        0.
    );
    transform_window_plane.look_at(Vec3::ZERO, -Vec3::Z);
    commands.spawn(
        Camera3dBundle {
            transform: transform_window_plane,
            ..default()
        }
    );```

(had to cut the code in half sorry)

    let plane_mesh = meshes.get_mut(&plane_mesh_handle).unwrap();
    let attribute_uv = Mesh::ATTRIBUTE_UV_0;
    let uv_value = plane_mesh.attribute(attribute_uv.id).unwrap().clone();
    plane_mesh.remove_attribute(attribute_uv.id);
    if let VertexAttributeValues::Float32x2(mut uv_values) = uv_value {
        uv_values.get_mut(0).unwrap()[0] = 0.;
        uv_values.get_mut(0).unwrap()[1] = 1.;
        uv_values.get_mut(1).unwrap()[0] = 0.2;
        uv_values.get_mut(1).unwrap()[1] = 0.2;
        uv_values.get_mut(2).unwrap()[0] = 1.;
        uv_values.get_mut(2).unwrap()[1] = 0.;
        uv_values.get_mut(3).unwrap()[0] = 1.;
        uv_values.get_mut(3).unwrap()[1] = 1.;
        dbg!(&uv_values);
        plane_mesh.insert_attribute(attribute_uv, VertexAttributeValues::Float32x2(uv_values));
    }
}```
In the example above I actually didn't change the UV-mapping, here is the result:
https://assets.octodon.social/media_attachments/files/110/137/502/626/280/505/original/c83873915eff519f.png

If I want to take only the central part of the image, I can do this:

        uv_values.get_mut(0).unwrap()[0] = 0.25;
        uv_values.get_mut(0).unwrap()[1] = 0.75;
        uv_values.get_mut(1).unwrap()[0] = 0.25;
        uv_values.get_mut(1).unwrap()[1] = 0.25;
        uv_values.get_mut(2).unwrap()[0] = 0.75;
        uv_values.get_mut(2).unwrap()[1] = 0.25;
        uv_values.get_mut(3).unwrap()[0] = 0.75;
        uv_values.get_mut(3).unwrap()[1] = 0.75;

I would get this, so far so good:
https://assets.octodon.social/media_attachments/files/110/137/514/760/830/593/original/898507387031ea59.png

Okay now let's go back to the full image, except for the top-right corner, where we'll stop half-way:

        uv_values.get_mut(0).unwrap()[0] = 0.;
        uv_values.get_mut(0).unwrap()[1] = 1.;
        uv_values.get_mut(1).unwrap()[0] = 0.;
        uv_values.get_mut(1).unwrap()[1] = 0.;
        uv_values.get_mut(2).unwrap()[0] = 0.75;
        uv_values.get_mut(2).unwrap()[1] = 0.25;
        uv_values.get_mut(3).unwrap()[0] = 1.;
        uv_values.get_mut(3).unwrap()[1] = 1.;

The image stretches, as we expect, nice !
https://assets.octodon.social/media_attachments/files/110/137/527/282/644/790/original/4fddc81594d8fa12.png

.
Now let's do the exact same thing but with the top-left corner!

        uv_values.get_mut(0).unwrap()[0] = 0.;
        uv_values.get_mut(0).unwrap()[1] = 1.;
        uv_values.get_mut(1).unwrap()[0] = 0.25;
        uv_values.get_mut(1).unwrap()[1] = 0.25;
        uv_values.get_mut(2).unwrap()[0] = 1.;
        uv_values.get_mut(2).unwrap()[1] = 0.;
        uv_values.get_mut(3).unwrap()[0] = 1.;
        uv_values.get_mut(3).unwrap()[1] = 1.;

This is the result, oh no!
https://assets.octodon.social/media_attachments/files/110/137/534/579/437/411/original/27e55ad9d21ab2be.png

As you can see, it stopped stretching at the diagonal, which is a similar problem I had in the introduction. The reason is that my Quad is made of two triangles, and each triangle interpolate its own UV thing independently, which is not what I want

Do you have any idea how I can solve this problem? Is there a magic configuration that doesn exactly what I want? I tried making the Quad a PrimitiveTopology::TriangleStrip instead of TriangleList, and I tried making it four triangles, but it doesn't solve the issue.

[calm fox]

Barycentric interpolation will always be per-triangle. You need to modify the diagonal uv values as well.

Not only the corner.

If you want to modify only the corner for a stretchy effect, you can look into uvw/bilinear Quad Interpolation.

Simple barycentric interpolation will always result in linear changes within a triangle, not exponential or anything that is affected by something beyond that triangle.

What you are seeing is intended behavior.

[boreal nebula]

@calm fox Thank you for your answer 💜
Is there a way to not do barycentric interpolation then? Is OpenGL able to do biliear interpolation? From what I saw on Bevy's side what I have with the mesh is just a uv Vec<f32[2]>, is there an option to change or should I put more points in that uv Vec than there are vertices, how would that work?
What do you mean when you say "you need to modify the diagonal uv values as well", should I have more triangles to approximate it?
How come it work with the guys in the video/blog post without even mentioning this problem, did they have a plane with more triangles and just brushed of the problem as trivial?

[calm fox]

You need to program bilinear yourself in the shader, it is not hardware-based.

[boreal nebula]

Ah okay

[calm fox]

you need to modify the diagonal uv values as well - you modified only one vertex, you needed to modify all vertices somehow.

[boreal nebula]

I guess I would need to learn how to program a shader then ><"

Yes I see what you mean

[calm fox]

How come it work with the guys in the video - they probably used linear functions for their projection (before the rasterizer) and then divided by w in the fragment shader or something.

[boreal nebula]

They divided by w and I did too but only for those 4 corners

I didn't understand they were doing this for all vertices

[calm fox]

Yes, so your biggest problem is that you need to correct your vertex modification things first.

[boreal nebula]

They use Unity and shader and I don't understand the code

Okay, thanks !

[calm fox]

You can send me the code but I can not look into it rn.

I am at work.

[boreal nebula]

No problem

It's in the blog post if you're interested

But I kinda see what I need to do now

[calm fox]

Anyways, you need to transform all vertices first, then do perspective divides in the fragments.

I have done portals before btw.

I remember having no issues when done that way.

And no bilinear interpolation requirement.

[boreal nebula]

I'm kinda short on time for the bevy jam though I kinda want to approximate it by having more triangles

I mean I never learned to do a shader or anything

Thanks a lot for your help anyway
I naively though (before starting, I saw it was not the case after testing) openGL would do bilinear interpolation by default

[calm fox]

Bilinear is not going to give you the portal effect you want exactly btw.

[boreal nebula]

Hmmm, I think I see what you mean yes

[calm fox]

Okay the shader in that blog is very simple.

If you copied the code it should have worked.

[calm fox]

Oh I am lucky I will never participate in a jam oof.

[boreal nebula]

I didn't copy the code I understood what he did and did it, but only on the corners of my cube, hence why it doesn't work

I'll try to reproduce it on bevy

[calm fox]

I always copy code lol.

[boreal nebula]

If I don't finish before the end it's cool, but it allowed me to motivate myself and learn a lot honestly, I didn't know how projection, uv mapping and shaders worked before starting four days ago x)

[boreal nebula]

I never copy code I don't understand ^^'

[calm fox]

We are like super opposite lol.

[boreal nebula]

And also I would have to learn how to compile a shader in C#

And like that seems way more of a hassle than learning how to do it with bevy

[calm fox]

Bevy is an engine, which does it for you.

Unity is also an engine, which does it for you.

Compiling a shader in Vulkan is going to be maybe 8 lines of code.

At least that was for me in C++.

[boreal nebula]

Yeah I thought Bevy was able to compile shaders itself but apparently not x)

[calm fox]

Well once you load them as an asset they get compiled, no?

At least they are definitely compiled when the material i constructed.

Meanwhile you have to manage all of that in Vulkan yourself.

[boreal nebula]

looks like it yeah

[boreal nebula]

Just in case anyone is interested I finally did it ><'

Here is my portal.wgsl

#import bevy_pbr::mesh_view_bindings
#import bevy_pbr::mesh_bindings

@group(1) @binding(0)
var texture: texture_2d<f32>;
@group(1) @binding(1)
var texture_sampler: sampler;

struct FragmentInput {
    @builtin(front_facing) is_front: bool,
    @builtin(position) frag_coord: vec4<f32>,
    #import bevy_pbr::mesh_vertex_output
};

@fragment
fn fragment(in: FragmentInput) -> @location(0) vec4<f32> {
    let dimensions = textureDimensions(texture);
    let dimension_x: f32 = f32(dimensions.x);
    let dimension_y: f32 = f32(dimensions.y);
    let uv: vec2<f32> = vec2(in.frag_coord.x/dimension_x, in.frag_coord.y/dimension_y);// / in.world_position.w;
    let color = textureSample(texture, texture_sampler, uv).rgb;
    return vec4(color, 1.0);
}```

I load it similar to this: https://bevyengine.org/examples/shader/shader-material/

Thanks a lot for your help @calm fox 💜

[calm fox]

Oh wow great you have done it!

Also looks like the shader is very simple, too.

I hope you get what is going on now.

Wait, that is not the same shader @boreal nebula

Your shader is literally just a "calculate uvs and sample from a texture" shader.

What has changed?

[boreal nebula]

That is what I wanted?
It works for my usage, idk, @builtin(position) frag_coord gives me the position of each fragment on screen so I don't need the vertex shader in the original.
The original was in GLSL I used WGSL instead

Here is the result: https://assets.octodon.social/media_attachments/files/110/146/911/672/700/864/original/749c84c0eab3cb1f.mp4