STALE ISSUE: Making sense of GLTF animations and models

[granite brook]

Hello,

recently I was trying to implement animations using GLTF, C++ and fastgltf to load the model.

I seem to have a peculiar issue with the program: it sometimes works, and sometimes it does not.
A simple cube with 7 bones: works. Model from Mixamo: works. My own model of a tram (and parts of it) for some reason don't.
The Mixamo model has the armature as a parent of the mesh, the cube and the tram's armatures are at the same level as their meshes. All models were created in Blender, all models' animations work in Blender.

I tried to diagnose the issue using RenderDoc, however unsuccessfully. The cout'ted output looks alright. I mostly followed the GLTF reference and spec, as well as example code from Sascha Willems.

https://www.khronos.org/files/gltf20-reference-guide.pdf
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html
https://github.com/SaschaWillems/Vulkan/blob/master/examples/gltfskinning/README.md
https://github.com/SaschaWillems/Vulkan/blob/master/examples/gltfskinning/gltfskinning.cpp

Is there some issue with parenting armatures/meshes I should be aware about? Is the issue in some part of my code? Any help is appreciated.

Vertex shader code: (uJoints are the joint matrices)

#version 450 core

layout(location = 0) in vec3 Position;
layout(location = 1) in vec2 TexCoord;
layout(location = 2) in vec3 Normal;
layout(location = 3) in float MaterialId;
layout(location = 4) in vec4 BoneIds;
layout(location = 5) in vec4 BoneWeights;

layout(location = 15) uniform mat4 uMatrix;

out vec2 pTexCoord;
flat out float pMaterialId;

layout(std430, binding = 51) readonly buffer sJoints {
    mat4 uJoints[];
};

void main() {
    mat4 skinMatrix;
    float totalWeight = BoneWeights.x+BoneWeights.y+BoneWeights.z+BoneWeights.w;
    //as reasonably close to 1 as per GLTF
    if(totalWeight > 0.95) {
        skinMatrix =
        BoneWeights.x * uJoints[int(BoneIds.x)] +
        BoneWeights.y * uJoints[int(BoneIds.y)] +
        BoneWeights.z * uJoints[int(BoneIds.z)] +
        BoneWeights.w * uJoints[int(BoneIds.w)];
    }
    else {
        skinMatrix = mat4(1.0);
    }

    gl_Position = uMatrix * skinMatrix * vec4(Position, 1.0);
    pTexCoord = TexCoord;
    pMaterialId = MaterialId;
}

CPU skinning matrices calculation:

glm::mat4 Model::getNodeMatrix(Node& node) {
    glm::mat4 nodeMatrix = node.localMatrix;
    int64_t currentParent = node.parent;
    while (currentParent != -1) {
        nodeMatrix = this->mNodes[currentParent].localMatrix * nodeMatrix;
        currentParent = this->mNodes[currentParent].parent;
    }
    return nodeMatrix;
}

void Model::updateJoints(Node& node, std::vector<glm::mat4>& aJointMatrices) {
    if (node.idOfSkin > -1) {
        glm::mat4              inverseTransform = glm::inverse(node.transformMatrix);
        Bone                   skin             = this->mBones[node.idOfSkin];
        size_t                 numJoints        = (uint32_t)skin.joints.size();
        std::vector<glm::mat4> jointMatrices(numJoints);
        for (size_t i = 0; i < numJoints; i++) {
            //do NOT set transform matrix anew
            jointMatrices[i] = getNodeMatrix(this->mNodes[skin.joints[i]]) * skin.inverseBindMatrix[i];
            jointMatrices[i] = inverseTransform * jointMatrices[i];
        }

        aJointMatrices.insert(aJointMatrices.end(), jointMatrices.begin(), jointMatrices.end());
    }

    for(auto& child : node.children) updateJoints(this->mNodes[child], aJointMatrices);
}

https://github.com/MegapolisPlayer/GLTFAnimationSample Repository

[feral crescent]

Im looking into it rn

Graphics API is OGL correct?

[granite brook]

yes, OpenGL 4.5

videos of models (one of them works, the other... doesnt)

the transparency is the base model without any transformations or skinning matrices applied

[feral crescent]

Unfortunatley, I'n not insanely experienced with gltf loading (I primarily use assimp because it can load a lot of different model formats) but from what I'm seeing, you might need to check stuff like the bone weight and index calculations to make sure every thing can be normalized and is within bounds.

[granite brook]

Thanks for your time, I will try to check this.
I am mostly confused whether or not the issue lies in the code or in my subpar modelling skills.

[feral crescent]

you metioned other models work, I would try and use some models from the gltf sample model github repo: https://github.com/KhronosGroup/glTF-Sample-Models there are some animated ones, but they are pretty simple. Check if those work, and if they do, its either a code issue with complexity of the animation/model, or your subpar modelling skills

[granite brook]

BoxAnimated already threw a fastgltf error, Fox does so too, but when I import the model into blender and then export it again it magically works...

[feral crescent]

sry

thats we

ird

w

ok

ay

my

k

eyboard

wont

[feral crescent]

work

[granite brook]

RE is the reexported one

[feral crescent]

okay sry about that

yeah thats weird

maybe blender export settings??

[granite brook]

should be the default

it throws a segfault when reading normals with the non-reexport

[feral crescent]

here lemme open belnder rq to check animation defaults for export

yeah so under te animation checkbox for the export, one of the settings is to optimize the animations (for me its checked by default) this could be why, but getting a seg fault could mean that something is potentially out of bounds when being calculated

[granite brook]

are the NORMAL attributes of a vertex required?

if not i found the reason for the fault...

okay I found the reason for the fault :)

adding a simple check if it is present fixed it

[feral crescent]

noice, i have that in my buffer creation code // Set up the attributes based on flags if (attributes & POSITION) { glVertexAttribPointer(0, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Position)); glEnableVertexAttribArray(0); } if (attributes & NORMAL) { glVertexAttribPointer(1, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Normal)); glEnableVertexAttribArray(1); } if (attributes & TEXCOORD) { glVertexAttribPointer(2, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, TexCoords)); glEnableVertexAttribArray(2); } if (attributes & TANGENT) { glVertexAttribPointer(3, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Tangent)); glEnableVertexAttribArray(3); } if (attributes & BITANGENT) { glVertexAttribPointer(4, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, Bitangent)); glEnableVertexAttribArray(4); } if (attributes & BONE_IDS) { glVertexAttribPointer(5, MAX_BONE_INFLUENCE, GL_INT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, m_BoneIDs)); glEnableVertexAttribArray(5); } if (attributes & BONE_WEIGHTS) { glVertexAttribPointer(6, MAX_BONE_INFLUENCE, GL_FLOAT, GL_FALSE, sizeof(Vertex), (void*)offsetof(Vertex, m_Weights)); glEnableVertexAttribArray(6); }

can you now load it correctly without the seg fault?

[granite brook]

yup

[feral crescent]

nice

[granite brook]

now im trying to move the camera properly to see if the animations are (not) fine

[feral crescent]

does it look okay so far??

[granite brook]

no..

the issue is therefore in my code

but the model loading itself (the greyed out fox) works

[feral crescent]

which one did you load, RE or normal

[granite brook]

normal

[feral crescent]

gotcha

[feral crescent]

are you on linux?

[granite brook]

yup

but on X11

[feral crescent]

nice

[granite brook]

so wayland issues should bother us

[feral crescent]

gotcha

[granite brook]

but again the head turns correctly, it is just moved weirdly

just like the tram

[feral crescent]

did you try the RE yet?

[granite brook]

the RE works (at least paritally)

[feral crescent]

what problem occurs with RE

[granite brook]

tbf the RE seems fine

[feral crescent]

yeah it doesnt look messed up

[granite brook]

i commited the changes to github

[feral crescent]

kk ill check it out

[granite brook]

so now we have validation during loading and i only apply the first animation (the fox has 3)

boxAnimated also now loads, sadly without animation

[feral crescent]

yeah im checking it out rn

i might spin up a vm and comp the code on linux

[granite brook]

the build system is CMake so you should be able to compile on Windows

i dont use any POSIX calls

[feral crescent]

kk

sry this is off topic but what linux distro do you use?

[granite brook]

but the dependencies of GLFW are linux only so you will have the download the DLLs

[granite brook]

fedora

[feral crescent]

yeah ive got glfw installed so we should be good

[feral crescent]

nice

never used it but it seems pretty good

[granite brook]

you are running it on ubuntu i guess?

[feral crescent]

no my main pc uses windows (i also game), but i have a laptop that i put arch on

sry its taking a second justs had to deal with a complier error

i got a debug assertion: Expression: vector subscript out of range

[granite brook]

i did add asserts when loading the models

assert(initialId+aId < vertices.size());

[feral crescent]

got it. yeah terminal printed: gltf model Error: GLTF2 model load failed! i dont have the model in the correct dir, so ill add it

[granite brook]

ok

i could have added colors highlighing in hindsight

[feral crescent]

okay so it ran

the fox animation did pop up

[granite brook]

nice

[feral crescent]

do you have the ablitliy to use wasd to move the camera?

[granite brook]

ye

its slightly weird but yes

R/F is up, down

C is reset to 0,0,0

X is close, ESC is undock mouse (click on window with RMB to redock)

this program is based on a simple OpenGL test program and i didnt really bother to update the camera since then :)

[feral crescent]

no its all good

im working on an ogl application rn

yeah lemme test with the normal one

yeah normal is messed up

was there any difference in terminal output for you?

[granite brook]

for which model?

[feral crescent]

comparing normal fox and RE fox

[granite brook]

will check

logically there should be

[feral crescent]

yeahn

[granite brook]

the RE has less bones it seems

[feral crescent]

could that be becasue of blender export optimization??

[granite brook]

probably is

[feral crescent]

maybe try and export it as a .gltf then check the gltf file to see any differences

idk at this point

blender is magic to me

[granite brook]

we have this information from the terminal, the bones are ordered differently and the offsets are slightly different

[granite brook]

first 2 months of learning it were the same

The program is divided into 3 parts: the Mesh class, the Animation class and the Model class

We know that Mesh and Animation is ok

We know that the load itself is too

so the issue is somewhere here:

glm::mat4 Model::getNodeMatrix(Node& node) {
    glm::mat4 nodeMatrix = node.localMatrix;
    int64_t currentParent = node.parent;
    while (currentParent != -1) {
        nodeMatrix = this->mNodes[currentParent].localMatrix * nodeMatrix;
        currentParent = this->mNodes[currentParent].parent;
    }
    return nodeMatrix;
}

void Model::updateJoints(Node& node, std::vector<glm::mat4>& aJointMatrices) {
    if (node.idOfSkin > -1) {
        glm::mat4              inverseTransform = glm::inverse(node.transformMatrix);
        Bone                   skin             = this->mBones[node.idOfSkin];
        size_t                 numJoints        = (uint32_t)skin.joints.size();
        std::vector<glm::mat4> jointMatrices(numJoints);
        for (size_t i = 0; i < numJoints; i++) {
            //do NOT set transform matrix anew
            jointMatrices[i] = getNodeMatrix(this->mNodes[skin.joints[i]]) * skin.inverseBindMatrix[i];
            jointMatrices[i] = inverseTransform * jointMatrices[i];
        }

        aJointMatrices.insert(aJointMatrices.end(), jointMatrices.begin(), jointMatrices.end());
    }

    for(auto& child : node.children) updateJoints(this->mNodes[child], aJointMatrices);
}

void Model::draw(const glm::mat4& aProjectionView) noexcept {
    for(uint64_t i = 0; i < this->mTextures.size(); i++)
        this->mTextures[i].bind(i);

    std::vector<glm::mat4> jointMatrices;
    for(size_t id : this->mRootNodes) {
        this->updateJoints(this->mNodes[id], jointMatrices);
    }

    glBindBuffer(GL_SHADER_STORAGE_BUFFER, this->mJointMatrixBuffer);
    glBufferSubData(GL_SHADER_STORAGE_BUFFER, 0, jointMatrices.size()*sizeof(glm::mat4), jointMatrices.data());

    glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 51, this->mJointMatrixBuffer);
    glBindBufferBase(GL_SHADER_STORAGE_BUFFER, 50, this->mMaterialBuffer);

    for(Mesh& m : this->mMeshes) m.draw(aProjectionView);
}

9/10 this will be some exteremely stupid mistake on my part

as is usual for coding bugs

[feral crescent]

im going to debug the code to see whats going on

okay im apparently missing a file called asan_poisoning.cpp for gcc. idk if your code uses asan, but when debugging the functions, (i partially got done deubgging them, just not the draw functiopn) they seemed okay, but i think to get more info out of them, we need to setup debug statements (via cout) in order to really see whats going on

[granite brook]

i do use asan (forgot to mention, sry), you can disable it in CMake

[feral crescent]

no your good, ill disable it

[granite brook]

draw is guaranteed fine, Animation was tested (when disabling it nothing changed expect for it being slightly choppy which makes sense)

you can enable the rendering of the base model in the "gui"

it uses the same draw and load functions

[feral crescent]

kk

is asan this param in the cmake -fsanitize=address

[granite brook]

yes

[feral crescent]

got it

[granite brook]

it is also in the linker

[feral crescent]

yeah comp and linker

okay so the first thing i see in in the updateJoints function is if (node.idOfSkin > -1) {, but node.idOfSkin is -1. could that maybe be a problem?? idk if it is or not

okay now im confused because it still executes the code in the if statement

[granite brook]

idOfSkin should be set in the Node constructor

which is set at Model.cpp line 104

last param

node.skinIndex.value()

[feral crescent]

got it. it just confused me a little bit because it was -1 one but the if statement had greater than -1

[granite brook]

-1 just means that the node has no skin/bone

[feral crescent]

kk

also my lord do you have that many matrices in the vector + aJointMatrices { size=233741081077576703 } std::vector<glm::mat<4,4,float,0>,std::allocator<glm::mat<4,4,float,0>>> &

[granite brook]

uh that is not supposed to happen

it seems alright when i cout the size in the draw function

[feral crescent]

wait nvm that was a problem on my part

sry

[granite brook]

its ok

[feral crescent]

yeah im not really seeing any problems.

[granite brook]

but still a massive thanks for your time

[feral crescent]

no problem! sry we werent able to fix it.

[granite brook]

the issue is probably in the matrix being applied where its not supposed to or an extra matrix being added somewhere

but its peculiar that this only affects some models

[feral crescent]

yeah that is pretty weird that only certain models are being affected, especially if we have to export them through blender just to get them to work

[granite brook]

but some blender-exported models dont work anyway

[granite brook]

The search goes on... when changing the animations, everything breaks...

[feral crescent]

weird....

[feral crescent]

okay even weirder (this is collada, also developed by khronos group) but assimp (or at least wiht my implementation) requires that i export the model trough blender to get skeletol animation to work. i think this might be a thing with model structure in the formats khronos has developed, even though glTF is different than collada

[granite brook]

it could be that we have the same issue here

I would suspect (in my implementation at least) the issue is with the joint offsets since the bind pose is ok

[feral crescent]

problem is what secret sauce is blender using when optimizing the animation??? are they really optimizing the bone structure and related parts? is it just the mesh? idk what they are really doing. btw this iss where i got my implementation from when i wrote it (i changed some stuff but its teh sssame core functoins for the animation) https://learnopengl.com/Guest-Articles/2020/Skeletal-Animation

i know its assimp but you kind of have to do the same thing for animation

[vocal oxide]

Sorry I didn't read everything, but from the looks of the video which shows the animation issue, it seems you're not handling the model node hierarchy correctly

Did you fully implement the gltf node hierarchy or are you just displaying the model meshes?

[granite brook]

i am reading the bones and animations from the heirarchy and then storing them per mesh node

every mesh also knows its amount of joints and based on that the joint matrices are computed and stored in a buffer

[vocal oxide]

But as i understand it, there's no mesh node, but rather nodes which can contain meshes
And also animations are applied to arbitrary nodes
So I don't think you can assume an animation will only touch a subset of nodes unless you know the source file, right?

[granite brook]

yea, i load all of them (i have the source code https://github.com/MegapolisPlayer/GLTFAnimationSample here) and transform all of them, according to the spec (or what i read from it) every node with mesh has some nodes with joins assigned to them and from the entry data it would make sense

the program then transforms the vertices based on the skinning matrices for said mesh

the peculiar thing is that some animations on some models do work, so this rules out the issue with matrix calculations (all models would not work). I know the linear interpolation calculations are also OK (when i disabled/enabled then it did nothing except what it was supposed to).

so basically: I have a model with N meshes, each mesh has N joints, all of this (nodes, meshes, joints) are stored in an array and all of the nodes know their parents and their children, which I use to calculate the matrices
The matrices are stored in a SSBO and each node has a joint offset so I can render the mesh with a single draw call from a single array. This system seems to work since some animations work and the debug output seems OK.

[lucid flax]

i had a problem once with exporting from blender. it was some kind of checkbox about the inclusion of an empty root node or something for unreal

ticking it off fixed it, but it was my fault, since it was my code not handling it, i guess

[granite brook]

going to try

although i cant find any such option in blender

[granite brook]

i frankly think that this is related to the issues

it indicates a larger problem that just some wrong matrices

[granite brook]

when I do not call any animation-related infrastructure it also doesn't work..., but it doesn't work less

(for the people reading the source: I do not call Model::setStateAtTime)

Also another weird thing is that then i don't set the uMatrix uniform to the transform matrix in the Mesh::draw function the result is the same - that matrix has no effect.

these two functions give the exact same effect

[lucid flax]

did you make a model with, like, two or three bones and rotations only of 45/90 degs and inspected the chain and values (also inside the ssbo)?

[granite brook]

yup

and it works

but another one i made doesnt

[lucid flax]

hehe

it breaks only on larger bone counts?

[granite brook]

not exactly

[lucid flax]

you should have two separate arrays for final matrices and for hierarchy

[granite brook]

my testing model #1 is a cube with 7 bones, this works
my testing model #2 is a door from my tram model with 4 bones, it doesn't work
the gltf sample fox model doesn't work, but when I run it though Blender it does work, but only for a single animations (tbf the doors do this too)

[lucid flax]

i peeked at the code and you're doing it in one, it seems, at first glance

[granite brook]

I have a SSBO for the joint matrices and a separate array (Model.mNodes) for the hierarchy

there isn't an array for the joint matrices per se

[lucid flax]

yeah, and they should have different matrices

not just duplicated

[granite brook]

yes

I calculate them using a function from a Vulkan sample

And considering some animations do work I severely doubt that the sample code is wrong, the issue is probably in the pieces I personally wrote

generation of joint matrix array (and its storage in the SSBO)

[lucid flax]

and you clear weights array to 0s

[granite brook]

Could I please know where exactly?

That would be a mistake

[lucid flax]

sorry, that was supposed to be a question :D

[granite brook]

oh okay :D

no I do not, the weights and bone ids seems fine in RenderDoc though

[lucid flax]

they should add up to 1 and unused should be 0s

just a sanity check

[granite brook]

I know for a fact that the model loading itself if fine - when I render the model without any extra transforms everything works fine and I know that the linear interpolations are also ok (I disabled them and nothing, except for the chopiness changed)

[granite brook]

from the renderdoc data they seem to do that

I will add a sanity check tho

Sanity checks pass ;)

[lucid flax]

can't really compile the code on win10/visual studio

ah, cause of the .so files

[granite brook]

yeah you need to download the GLFW dlls from their website

sry

I develop on Linux and I usually don't bother all that much with Windows

[lucid flax]

you could just have glfw as git submodule and let the cmake compile it for the platform

[granite brook]

also an option and when/if I will intergrate animations into a serious project I will do just that

the other libraries are added as source files

I pushed some changes in CMake to git, should be a bit more Windows-friendly now

[lucid flax]

vs msvc doesn't have /wextra buut

ah, you're doing the vector + index for bones

i gave up with that. tree structure is so much better and you still have random accesses

[granite brook]

okay...

I have a slight idea what could be wrong

should i multiply the matrices of all parent nodes or just the joints and skins?

[lucid flax]

yeah, seeing that too while reading the code

you should build up a chain

passing matrices in recursion call

[granite brook]

that part (updateJoints) was copied straight from the vulkan sample code

that should be functional code

[lucid flax]

okay, so what part of the code pertained to your question whether you should multiply matrices of all parent nodes

[granite brook]

Model.cpp - around line 353 to 379 - the updateJoints and the getNodeMatrix functions

[lucid flax]

if it's straight from the vulkan repo - no modifications - then it should work, really

[granite brook]

that is the whole issue

before this I spent 3 weeks trying to implment this, gave up, and came back to this a month and a half later

Model loading - it loads the model with all attributes, I guess it is fine?
Mathematics - some animations work, I guess it is OK then
Animation linear interpolation - works, tested
Draw code - works with other models

Maybe some issue with passing the data to the shader?

[lucid flax]

you can inspect all the values in rdc

and i wholeheartedly can recommend using DSA in opengl

so much nicer

[granite brook]

thought about that, will get around to it later :)

VS In is fine, VS Out is a mess

[lucid flax]

show vertex buffer

[granite brook]

the VS in data?

[lucid flax]

yeah

[granite brook]

or the joint matrices?

[lucid flax]

with all them attributes

joints too

[granite brook]

now to just format this

[lucid flax]

weird that it doesn't

[granite brook]

it does sometimes

perhaps due to ungrateful closing of the app

will fix and rerun wait a sec please

it does throw some issues even though my vertex attribute setup code is fine and the opengl debug callback doesn't get called...

okay here we go

[lucid flax]

i managed to run your code

yeah

[granite brook]

great

VS OUT data

VS IN data

[lucid flax]

hmmm

looking at your mats

they should not have that values in 'em

[granite brook]

I do know that the issue is NOT in the Animation class

that is its not in the linear interpolation, I tested that part thoroughly and disabling it did not fix anything

so the issue is somewhere in the Model.cpp file (Mesh does nothing except send the matrix to the shader)

[lucid flax]

you should also add this, imo

    glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 4);
    glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 6);

[granite brook]

I use 4.5 (since 4.6 is not supported but all our cards currently being sold) but yea

not that this will help the issue but this is good practice

[lucid flax]

so these are your joints

yeah, and i look the right column and see some big values

not that they are wrong. but look suspicious

[granite brook]

that seems wrong

those are the joint ids in the VBO?

or the matrices?

[lucid flax]

nah, they are actual joint matrices

just some big translation they have

[granite brook]

those values are certainly wrong and they would explain the issue

[lucid flax]

i am hesitant since your fox is quite large, so it could explain them

well, not 30-60 large

[granite brook]

try some other models perhaps

[lucid flax]

should really be scaled, not moved

so big values should be in 1-3 cols

for scale/rotation

[granite brook]

there are a few other (simpler) models

line 182 in GL3D.cpp - just change the path

Fox.glb is the GLTF sample fox, FoxRE is the same fox run through blender, T3.glb is a model of a tram, Untitled is just a cube with bones and Untitled2 is the Mixamo model (which works!)

[lucid flax]

can you confirm that you're doing animatino composition in the correct order?

scale first, translation last

[granite brook]

it should (i think even is by the spec) be enforced when i load the animations

it could also be an issue if this is not enforced and the mixamo model just so happens to provide them correctly

debug output confirms: everything is in order TRS

[lucid flax]

fixed it

i was right with that checkbox xd

[granite brook]

i got here so far

[granite brook]

oh

[lucid flax]

export the fox with "remove armature"

[granite brook]

happens :)

[granite brook]

or try to handle the armature

fastgltf does allow this

[granite brook]

still a MASSIVE thank you for your time and dedication :)

[lucid flax]

oki, so now one is good, one is bad. only to find a difference

[granite brook]

i commited some changes to github, now it still does not work but it does not work less

the animations almost work

but are transformed wrong

[lucid flax]

prob hierarchy is screwed somehow because of that one root node

that checkbox removes root node, that is either empty or i don't know

do let me know, cause last time i just settled with ticking that checkbox

[granite brook]

but this time something else doesn't work

when i recalculate the animations -

• if i set it to identity - it kinda works
• if i set it to original local matrix - that happens
• if i dont set it - the animations go off the deep end

[lucid flax]

maybe bone ids should be offset by 1, because of that faulty root node xd

[granite brook]

might be

[lucid flax]

still not a solution

[granite brook]

GLTF doesn't specify anything about armatures tho

[granite brook]

I read through the fastGLTF code and outputted the root matrices

The node matrices seem OK, there aren't any armatures defined

[granite brook]

It seems I calculate the joint matrices wrong, as suggested by Karol here.

[lucid flax]

did you check whether you have any unused bones?

[granite brook]

there arent any in blender and that shouldn't affect the model

the spec doesnt say anything

3.7.3. unused joints have 0 weight

[granite brook]

The search goes on - I have now exported the working matrices (a.csv) and the non working matrices (b.csv). Now to figure out what is happening.

[granite brook]

Also the animation switching seems to work now.

[granite brook]

Also could the small box (another mesh) have something to do with the issue? Blender did add it for some reason...

[oak osprey]

Hey, I'm new here in search of graphics programming knowldege help.

So am currently trying to apply vertex skinning to a model and so far the model is darting all over the screen instead of animating the walk its supposed to.

I'm not sure what I'm doing wrong but, I'll give you a brief overview of my animation rendering pipeline and hopefully somebody can see what I'm doing wrong.

So as you read this know I using a gltf 2.0 file type

1. jointLocalTransform - this is the matrix I'm creating using keyframe interpolated translation, scale and rotation data

2. jointGlobalTransform = parentJointLocalTransform * jointLocalTransform
   Except for the root joint I first recursively calculate the jointGlobalTrasnform like this

3.animationTransform = jointGlobalTransform * jointInverseMatrix

4. sending an array of animationTransform to vertex shader for vertexSkinng

Here is my vertexShader code in Opengl

#version 300 es

const int MAX_JOINTS = 50;//max joints allowed in a skeleton
const int MAX_WEIGHTS = 4;//max number of joints that can affect a vertex

in vec3 position;
in vec2 tex;
in vec3 normal;
in ivec4 jointIndices;
in vec4 weights;

out vec2 oTex;
out vec3 oNorm;

uniform mat4 jointTransforms[MAX_JOINTS];

uniform mat4 model;
uniform mat4 projection;
uniform mat4 view;

void main(){

vec4 totalLocalPos = vec4(0.0);
vec4 totalNormal = vec4(0.0);

for(int i=0;i<MAX_WEIGHTS;i++){
    mat4 jointTransform = jointTransforms[jointIndices[i]];
    vec4 posePosition = jointTransform * vec4(position, 1.0);
    totalLocalPos += posePosition * weights[i];
    
    vec4 worldNormal = jointTransform * vec4(normal, 0.0);
    totalNormal += worldNormal * weights[i];
}

gl_Position = projection * view * (model * totalLocalPos);
oNorm = totalNormal.xyz;
oTex = tex;

}

[granite brook]

Frankly it seems correct, but considering we are both on this forum it isn't ;)

Did you base it off the code here or us it your own?

[oak osprey]

I didn't use any forum(here) code its all 'my' code

[oak osprey]

If that's the case though I wouldn't mind a second opinion then maybe my problem might be how I am arriving at jointLocalTransform. There's a vulkan example on github I'm looking at and instead of TRS its like SRT or maybe I'm seeing my own things

https://github.com/beaumanvienna/vulkan/blob/master/engine/renderer/skeletalAnimation/skeleton.h

struct Joint
{
std::string m_Name;
glm::mat4 m_InverseBindMatrix; // a.k.a undeformed inverse node matrix

        // deformed / animated
        // to be applied to the node matrix a.k.a bind matrix in the world coordinate system,
        // controlled by an animation or a single pose (they come out of gltf animation samplers)
        glm::vec3 m_DeformedNodeTranslation{0.0f};                            // T
        glm::quat m_DeformedNodeRotation = glm::quat(1.0f, 0.0f, 0.0f, 0.0f); // R
        glm::vec3 m_DeformedNodeScale{1.0f};                                  // S

        glm::mat4 GetDeformedBindMatrix()
        {
            // apply scale, rotation, and translation IN THAT ORDER (read from right to the left)
            // to the original undefomed bind matrix
            // dynamically called once per frame
            return glm::translate(glm::mat4(1.0f), m_DeformedNodeTranslation) * // T
                   glm::mat4(m_DeformedNodeRotation) *                          // R
                   glm::scale(glm::mat4(1.0f), m_DeformedNodeScale);            // S
        }

        // parents and children for the tree hierachy
        int m_ParentJoint;
        std::vector<int> m_Children;
    };

[granite brook]

How exactly does it not work? Does it not work at all (a garbled mess) or does it sort of work (like this)

[craggy plaza]

Do you have join visualization?

If not, make that first

Here is mine, if you find reference useful: https://github.com/tksuoran/erhe/blob/main/src%2Feditor%2Ftools%2Fdebug_visualizations.cpp#L217

[granite brook]

The best I can do is this since GLTF doesnt store the full skeleton data.... The lines point from the node's transform to the final joint transform

glLineWidth(10);
    glPointSize(20);
    for(Node& b : this->mNodes) {
        if (b.idOfSkin > -1) {

            glm::mat4              inverseTransform = glm::inverse(b.transformMatrix);
            Bone                   skin             = this->mBones[b.idOfSkin];
            size_t                 numJoints        = (uint32_t)skin.joints.size();
            std::vector<glm::mat4> jointMatrices(numJoints);
            for (size_t i = 0; i < numJoints; i++) {
                //do NOT set transform matrix anew
                jointMatrices[i] = getNodeMatrix(this->mNodes[skin.joints[i]]) * skin.inverseBindMatrix[i];
                jointMatrices[i] = inverseTransform * jointMatrices[i];

                glBegin(GL_LINES);
                glm::vec3 boneStart = getNodeMatrix(this->mNodes[skin.joints[i]])*glm::vec4(1.0f);
                glColor3f(1.0f, 0.0f, 1.0f);
                glVertex3fv(glm::value_ptr(boneStart));
                glm::vec3 boneEnd = jointMatrices[i]*glm::vec4(1.0f);
                glColor3f(1.0f, 1.0f, 1.0f);
                glVertex3fv(glm::value_ptr(boneEnd));
                glEnd();
            }
        }
    }

It is possible to see that something is really wrong with the matrices

idk if it could be my packing of the joints into a flat array...

    //root nodes (children of non-node root)
    std::cout << "Root nodes ";
    for(size_t nid : model->scenes[model->defaultScene.value()].nodeIndices) {
        this->mRootNodes.push_back(nid);
        std::cout << nid << "\n";
    }
    std::cout << std::endl;

    this->getNodeJointAmount();
    //get offsets for joint matrices (ids bound to node)
    uint64_t curOff = 0;
    for(size_t nid : this->mRootNodes) {
        this->getNodeJointOffset(nid, &curOff);
    }

    std::cout << "Joint offsets ";
    for(Node& n : this->mNodes) std::cout << n.jointsIdOffset << " ";
    std::cout << std::endl;

    std::cout << "Joint amounts ";
    for(Node& n : this->mNodes) std::cout << n.amountOfJoints << " ";
    std::cout << std::endl;

    uint64_t jointMatricesAmount = 0;
    for(Node& n : this->mNodes) jointMatricesAmount += n.amountOfJoints;

        std::cout << "Total joint amount: " << jointMatricesAmount << '\n';
    glGenBuffers(1, &this->mJointMatrixBuffer);
    glBindBuffer(GL_SHADER_STORAGE_BUFFER, this->mJointMatrixBuffer);
    glBufferData(GL_SHADER_STORAGE_BUFFER, jointMatricesAmount*sizeof(glm::mat4), nullptr, GL_DYNAMIC_DRAW);
    this->mJointsAmount = jointMatricesAmount;

glm::mat4 Model::getNodeMatrix(Node& node) {
    glm::mat4 nodeMatrix = node.localMatrix;
    int64_t currentParent = node.parent;
    while (currentParent != -1) {
        nodeMatrix = this->mNodes[currentParent].localMatrix * nodeMatrix;
        currentParent = this->mNodes[currentParent].parent;
    }
    return nodeMatrix;
}

void Model::updateJoints(Node& node, std::vector<glm::mat4>& aJointMatrices) {
    if (node.idOfSkin > -1) {
        glm::mat4              inverseTransform = glm::inverse(node.transformMatrix);
        Bone                   skin             = this->mBones[node.idOfSkin];
        size_t                 numJoints        = (uint32_t)skin.joints.size();
        std::vector<glm::mat4> jointMatrices(numJoints);
        for (size_t i = 0; i < numJoints; i++) {
            //do NOT set transform matrix anew
            jointMatrices[i] = getNodeMatrix(this->mNodes[skin.joints[i]]) * skin.inverseBindMatrix[i];
            jointMatrices[i] = inverseTransform * jointMatrices[i];
        }

        aJointMatrices.insert(aJointMatrices.end(), jointMatrices.begin(), jointMatrices.end());
    }

    for(auto& child : node.children) updateJoints(this->mNodes[child], aJointMatrices);
}

https://github.com/MegapolisPlayer/GLTFAnimationSample I have uploaded the code sample to the repository if someone wants to experiment with it. GL3D.cpp is the main function (guaranteed OK), Animation.cpp handles animations (almost certainly OK, underwent extensive testing and the feared COUT spam), Model.cpp is probably wrong, Mesh.cpp just draws stuff, Shader.cpp and Texture.cpp unrelated to issue and tested thoroughly. The model itself loaded without any issues, the issue therefore lies somewhere in the matrix calculation or the packing system.

[oak osprey]

The armature is in it bind pose, its not distorted, or stretching in any way, the problem is when I apply the animation Transform instead of the armateur moving like its supposed to in its animation its like doing a rotation as a whole object in its armateur bindpose state

[craggy plaza]

First make sure animations work, especially hierarchical aninstions, without skinning involved.

Test with minimal test assets, starting with one node getting transforms right, then two nodes wirh parenting

You probably should check first that hierarchical transforms work without animation

[oak osprey]

I've used the Windows 3D viewer and the animation works well. I also ported it into Unreal to check it and it worked.
Is that a valid way to confirm the animation works?

[granite brook]

I would say yes

[granite brook]

STALE ISSUE: Making sense of GLTF animations and models

[granite brook]

This is fascinating - this simple model works without any issues.....

Log for this simple one

Material: Material
Bone.001 0) [1;2;1;1]
Bone 1) Bone.001 [1;1;1;1]
Cube 2) [1;1;1;1]
Armature 3) Cube Bone [1;1;1;1]
Bone.001(0)->Bone(1)
Cube(2)->Armature(3)
Bone(1)->Armature(3)
Skin Armature
Root nodes 3

Joint offsets 2 2 0 0 
Joint amounts 0 0 2 0 
Total joint amount: 2
Mesh name: Cube
Found animation: ArmatureAction

Log for the non working model

Material: fox_material
root 0) _rootJoint [1;1;1;1]
fox 1) [1;1;1;1]
_rootJoint 2) b_Root_00 [1;1;1;1]
b_Root_00 3) b_Hip_01 [1;0.999996;-1;1]
b_Hip_01 4) b_Spine01_02 b_Tail01_012 b_LeftLeg01_015 b_RightLeg01_019 [-1;27.328;44.2282;1]
b_Spine01_02 5) b_Spine02_03 [14.1065;-0.65023;1;1]
b_Spine02_03 6) b_Neck_04 b_RightUpperArm_06 b_LeftUpperArm_09 [22.6203;1.03409;1;1]
b_Neck_04 7) b_Head_05 [25.8869;1.39408;1;1]
b_Head_05 8) [14.7901;-0.0540924;1;1]
b_RightUpperArm_06 9) b_RightForeArm_07 [19.6621;-5.31166;7.96883;1]
b_RightForeArm_07 10) b_RightHand_08 [23.9682;1.07144;1;1]
b_RightHand_08 11) [19.0834;1.26135;0.973813;1]
b_LeftUpperArm_09 12) b_LeftForeArm_010 [19.6617;-5.31082;-5.9655;1]
b_LeftForeArm_010 13) b_LeftHand_011 [23.9682;1.07144;1;1]
b_LeftHand_011 14) [19.1501;1.2309;1.0316;1]
b_Tail01_012 15) b_Tail02_013 [2.96014;15.4026;1;1]
b_Tail02_013 16) b_Tail03_014 [13.5411;0.85141;1;1]
b_Tail03_014 17) [25.342;0.886695;1;1]
b_LeftLeg01_015 18) b_LeftLeg02_016 [4.4035;3.80032;-5.96842;1]
b_LeftLeg02_016 19) b_LeftFoot01_017 [20.3577;0.0446672;1;1]
b_LeftFoot01_017 20) b_LeftFoot02_018 [17.7077;1.39405;1.00067;1]
b_LeftFoot02_018 21) [15.2648;1.31705;1;1]
b_RightLeg01_019 22) b_RightLeg02_020 [4.40406;3.80046;7.96756;1]
b_RightLeg02_020 23) b_RightFoot01_021 [20.3577;0.044667;1;1]
b_RightFoot01_021 24) b_RightFoot02_022 [17.7091;1.39507;0.999587;1]
b_RightFoot02_022 25) [15.2648;1.31705;1;1]

_rootJoint(2)->root(0)
b_Root_00(3)->_rootJoint(2)
b_Hip_01(4)->b_Root_00(3)
b_Spine01_02(5)->b_Hip_01(4)
b_Tail01_012(15)->b_Hip_01(4)
b_LeftLeg01_015(18)->b_Hip_01(4)
b_RightLeg01_019(22)->b_Hip_01(4)
b_Spine02_03(6)->b_Spine01_02(5)
b_Neck_04(7)->b_Spine02_03(6)
b_RightUpperArm_06(9)->b_Spine02_03(6)
b_LeftUpperArm_09(12)->b_Spine02_03(6)
b_Head_05(8)->b_Neck_04(7)
b_RightForeArm_07(10)->b_RightUpperArm_06(9)
b_RightHand_08(11)->b_RightForeArm_07(10)
b_LeftForeArm_010(13)->b_LeftUpperArm_09(12)
b_LeftHand_011(14)->b_LeftForeArm_010(13)
b_Tail02_013(16)->b_Tail01_012(15)
b_Tail03_014(17)->b_Tail02_013(16)
b_LeftLeg02_016(19)->b_LeftLeg01_015(18)
b_LeftFoot01_017(20)->b_LeftLeg02_016(19)
b_LeftFoot02_018(21)->b_LeftFoot01_017(20)
b_RightLeg02_020(23)->b_RightLeg01_019(22)
b_RightFoot01_021(24)->b_RightLeg02_020(23)
b_RightFoot02_022(25)->b_RightFoot01_021(24)
Skin 
Root nodes 0
1

Joint offsets 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Joint amounts 0 24 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 
Total joint amount: 24
Mesh name: fox
Found animation: Survey
Found animation: Walk
Found animation: Run

Log for the working one

Material: fox_material
Material: Material
b_Head_05 0) [14.7901;-0.0540847;1.00001;1]
b_Neck_04 1) b_Head_05 [25.8869;1.39409;0.999998;1]
b_RightHand_08 2) [19.0834;1.26135;0.973811;1]
b_RightForeArm_07 3) b_RightHand_08 [23.9682;1.07148;1.00004;1]
b_RightUpperArm_06 4) b_RightForeArm_07 [19.6621;-5.31165;7.96883;1]
b_LeftHand_011 5) [19.1501;1.23091;1.03162;1]
b_LeftForeArm_010 6) b_LeftHand_011 [23.9682;1.07144;0.999977;1]
b_LeftUpperArm_09 7) b_LeftForeArm_010 [19.6617;-5.31082;-5.9655;1]
b_Spine02_03 8) b_Neck_04 b_RightUpperArm_06 b_LeftUpperArm_09 [22.6203;1.0341;1.00002;1]
b_Spine01_02 9) b_Spine02_03 [14.1065;-0.650266;1.00001;1]
b_Tail03_014 10) [25.3421;0.886718;1.00001;1]
b_Tail02_013 11) b_Tail03_014 [13.5411;0.851429;1;1]
b_Tail01_012 12) b_Tail02_013 [2.96017;15.4025;1;1]
b_LeftFoot02_018 13) [15.2648;1.31704;0.999996;1]
b_LeftFoot01_017 14) b_LeftFoot02_018 [17.7077;1.39408;1.00067;1]
b_LeftLeg02_016 15) b_LeftFoot01_017 [20.3577;0.0446816;1.00002;1]
b_LeftLeg01_015 16) b_LeftLeg02_016 [4.40351;3.8003;-5.96842;1]
b_RightFoot02_022 17) [15.2648;1.31703;0.999996;1]
b_RightFoot01_021 18) b_RightFoot02_022 [17.7092;1.39507;0.999567;1]
b_RightLeg02_020 19) b_RightFoot01_021 [20.3577;0.044654;0.999995;1]
b_RightLeg01_019 20) b_RightLeg02_020 [4.40407;3.80045;7.96757;1]
b_Hip_01 21) b_Spine01_02 b_Tail01_012 b_LeftLeg01_015 b_RightLeg01_019 [-1;27.328;44.2282;1]
b_Root_00 22) b_Hip_01 [1;0.999996;-1;1]

_rootJoint 23) b_Root_00 [1;1;1;1]
fox 24) [1;1;1;1]
root 25) fox _rootJoint [1;1;1;1]
Cube 26) [1;1;1;1]
b_Head_05(0)->b_Neck_04(1)
b_RightHand_08(2)->b_RightForeArm_07(3)
b_RightForeArm_07(3)->b_RightUpperArm_06(4)
b_LeftHand_011(5)->b_LeftForeArm_010(6)
b_LeftForeArm_010(6)->b_LeftUpperArm_09(7)
b_Neck_04(1)->b_Spine02_03(8)
b_RightUpperArm_06(4)->b_Spine02_03(8)
b_LeftUpperArm_09(7)->b_Spine02_03(8)
b_Spine02_03(8)->b_Spine01_02(9)
b_Tail03_014(10)->b_Tail02_013(11)
b_Tail02_013(11)->b_Tail01_012(12)
b_LeftFoot02_018(13)->b_LeftFoot01_017(14)
b_LeftFoot01_017(14)->b_LeftLeg02_016(15)
b_LeftLeg02_016(15)->b_LeftLeg01_015(16)
b_RightFoot02_022(17)->b_RightFoot01_021(18)
b_RightFoot01_021(18)->b_RightLeg02_020(19)
b_RightLeg02_020(19)->b_RightLeg01_019(20)
b_Spine01_02(9)->b_Hip_01(21)
b_Tail01_012(12)->b_Hip_01(21)
b_LeftLeg01_015(16)->b_Hip_01(21)
b_RightLeg01_019(20)->b_Hip_01(21)
b_Hip_01(21)->b_Root_00(22)
b_Root_00(22)->_rootJoint(23)
fox(24)->root(25)
_rootJoint(23)->root(25)
Skin root
Root nodes 25
26

Joint offsets 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 24 0 0 24 
Joint amounts 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 24 0 0 
Total joint amount: 24
Mesh name: fox
Mesh name: Cube
Found animation: Run
Found animation: Survey
Found animation: Walk

[granite brook]

here we can see the pantograph animation working but transformed wrong

[craggy plaza]

I can check what it looks like in my code tomorrow

Or you could try compare yourself

[granite brook]

If you could I would be extremely grateful

Thanks in advance

[craggy plaza]

Which glb asset are you testing?

Finding simplest asset that has issue would be useful

[granite brook]

in the repo T3.glb, the fox is Fox.glb (doesnt) and FoxRE.glb (works)

[granite brook]

the BoxAnimated.glb is the smallest non working one

[granite brook]

Some more breadcrumbs of information: using the tram model is probably better than the fox since the issue is more visible - the skinning matrices seem alright, I would suspect that the issue is with calculating the transform of the parent node perhaps?

[craggy plaza]

I run into assert when testing with my code

So this is actually a good test asset 🙂

Report for glTF Tools VSCode extension:

{
    "uri": "t3d1.glb",
    "mimeType": "model/gltf-binary",
    "validatorVersion": "2.0.0-dev.3.10",
    "validatedAt": "2025-05-30T06:44:25.667Z",
    "issues": {
        "numErrors": 0,
        "numWarnings": 1,
        "numInfos": 1,
        "numHints": 0,
        "messages": [
            {
                "code": "UNUSED_OBJECT",
                "message": "This object may be unused.",
                "severity": 2,
                "pointer": "/meshes/0/primitives/2/attributes/TEXCOORD_0"
            },
            {
                "code": "NODE_SKINNED_MESH_NON_ROOT",
                "message": "Node with a skinned mesh is not root. Parent transforms will not affect a skinned mesh.",
                "severity": 1,
                "pointer": "/nodes/4"
            }
        ],
        "truncated": false
    },
    "info": {
...

I can see where the issue is in my code:

• I have parse_node() where I process node. It is recursive by nature, since node can have child nodes
• The issue is that parse_node() processes also mesh and skin for that node. But skin in a node can point to joints / nodes that have not yet been processed
• Solution in my case would be to process skins in a separate pass, not from parse_node()
• In my original pass, I process gltf/scenes/[0]/nodes recursively. In my new pass for node mesh and skin, I don't do recursion, I collect node indices during the first pass to a vector and in second pass I process exactly those nodes. You could also use recursion in the second pass I guess.

[craggy plaza]

Looks like my animation playing has rotted, I can see there is animation but scrubbing the slider does not play it

[craggy plaza]

Yay,

I pushed fixes to my code in here https://github.com/tksuoran/erhe/commit/926526e0f0db222651d368f7a489749db144246e - The gltf fixes were in gltf_fastgltf.cpp. I have no idea if your issues are related, but that seems plausible.

Thanks for the test content 👍 , my code still has issues with some of these

[craggy plaza]

Using vertex colors instead of texture would probably work better here

[oak osprey]

Solved the issue. I was using the wrong data type for jointIndices, which lead to all my jointIndices being read as 0 and that's why it was stuck on bind-pose.

[granite brook]

Has anyone encountered such an issue?

It seems that the skinning matrices are OK, and the animations themselves are too, everything is just transformed and scaled wrong

I would guess that I am incorrectly multiplying the parent matrices but I really don't know anymore