Way to rebuild tangent for blendshape in vertex shader?

[vagrant schooner]

Hi, I am trying implement blend shape animation in vertex shader, but I have little idea how to rebuild the correct tangent after the blend. (The tangent is required to be mikktspace)

What I have:

• base mesh's positions + normals + tangent passed in as shader input
• multiple target meshes' position + normal (can be in delta or absolute form) and blend percentage accessed from shader resources

As vertex shader lack adjacent vertex info, do I have no choice but to retrieve blended result from shader to C++ side in order to rebuild tangent?

Any advice is appreciated.

[final python]

Why would you need adjacency?

Is blend shape just weighted sum of vertex attributes?

Do you also have/need bitangent? For that you need at least one bit somewhere to tell the direction.

I've never done this, but I think this would give you something: Assuming you have initial tangent, bitangent and normal, and updated normal, you keep either T or B, and compute the other using crossproduct with the new normal.

You probably want to throw away the one of T vs. B based on which one is closer to the new normal.

You can add orthonormalization at the end, keeping the new normal unchanged (other than normalization which you might do already after interpolation).

[vagrant schooner]

@final python Thank you for the replies! The orthonormalization way looks great,
But I still have doubts: is it legit that new T calculated in this way corresponds to the new geometry surface?

For example, the mesh on the left lies on xz plane, and assuming the blended result is the right, before and after blend, N is unchanged so that new calculated T (or B) will also keep the same, yet mesh is skewed.
(That may explain why I need adjacency)

[final python]

That is a really good picture. I don't have an answer yet but I'll think about it. This is an interesting question.

Perhaps you could ask from Khronos / glTF how to deal with tangent frame when mesh has this kind of transforms. There is https://www.khronos.org/news/permalink/new-gltf-slack-channel and https://github.com/KhronosGroup/glTF/issues. Also Morten S. Mikkelsen probably has good understanding and could help if you ask him, https://github.com/mmikk/MikkTSpace/issues for example.

Oh and please do ping me when you find something - thanks!

I suppose you might be able reconstruct tangent frame from texture coordinate derivatives and/or normal derivatives (in the fragment shader).

[vagrant schooner]

@final python

be able reconstruct tangent frame from texture coordinate derivatives
yes, things should be eaiser in fragment shader, but blend shape has to be calculated before any skinning which happens in vertex shader so we're out of luck.

and thanks for following up! I will keep you updated.

[final python]

What exactly do you mean by "has to be calculated before any skinning?"

Isn't skinning kind of part of vertex shading (or compute), so that you can do whatever is left in the fragment shader?

[vagrant schooner]

Sorry for the lack of explanation.
Currently, the skinning is done in vertex shader, and the requirement is to apply blend shape first, then do the skinning based on the blended mesh.

this restricts the place that blend shape tangent rebuild happens.

[final python]

✅