I am trying to render quads that always face the camera. I currently extract the rotation part of the 4x4 view matrix like this:
glm::mat3 rotationOnly = viewonly;
QuadParticleData quadData{
.viewProj = viewproj,
.billboard = glm::transpose(rotationOnly)
};
Then, in the shader, I take the quad point (a vec2), scale it, apply the bilboard matrix, and finally apply the billboard matrix:
vec3 localPoint = vec3(inVertex * data.scale, 0);
localPoint = matrices.billboard * localPoint;
vec4 vert = vec4(localPoint + data.pos, 1);
vert = matrices.viewProj * vert;
However, the quads do not face the camera. If I remove the bilboard operation, then the quads are aligned to the XY axes. How can I fix my math so that the billboarding works?