Need info about GPU/vertex animation

[regal sierra]

Are there any examples of animation implementation via vertex shader? I will be glad of any helpful information

[barren gorge]

Could you elaborate a bit more? This is awfully little detail about what you actually want to do. "animation" could be anything from: "simply offsetting the meshes vertices slightly based on some noise seeded with time" which is like, 4 lines of shader code and really simple, to "full skinned mesh animation with bones, IK, weight-painting, etc" and knowing which you mean dramatically changes the resources and examples that will be useful to you

[regal sierra]

Yes, I'm sorry, perhaps I really should have been more informative. I need to animate the movement of the limbs of a large number of characters (running, walking, etc.) while ensuring maximum performance. I think I need something like interpolation between morph targets at the vertex shader level, but actually I would listen to advice in this direction

[barren gorge]

let me ask the obvious question: have you tried the built-in animation playback in your usecase and found it lacking?

[regal sierra]

yeah, I tried and yes, in my case I found it lacking

[barren gorge]

How did you profile it? What was slow?

what i mainly wonder is whether the animations themselves were performing badly, or some other factor related to the number of entities, especially moving entities

[regal sierra]

I tried to use the methods from the examples, but they did not include the possibility of using custom materials and instancing. Maybe I don't need gpu animation, but a way to use animation by creating an entity not as a SceneBundle, but by adding everything individually. I hope I made myself clear.

[barren gorge]

could you elaborate some more what you mean by "adding everything individually"? whats the goal of that - it seems unrelated to animation performance?

[regal sierra]

if I'll spawn a few thousand MaterialMeshBundle, which will contain mesh and material from gltf, then these objects will have almost no effect on performance, but if I'll spawn these objects as a SceneBundle, then everything will become bad

I need to animate objects while still being able to customize the material and use instancing

I'm asking about animation because I haven't found any examples of animation without loading an object as a SceneBundle

[barren gorge]

This is the example for manually doing skinned mesh animation: https://bevy.org/examples/animation/custom-skinned-mesh/

[barren gorge]

when performance is bad, are you spawning all versions of your mesh as a single scene or are you spawning individual versions from their own scene? Also, you mentioned SceneBundle and MaterialMeshBundle - which version of bevy are you using?

[regal sierra]

I upload a mesh/scene to a resource, and every time I create an entity, I use an instance from that resource. I'm using 0.14

[barren gorge]

firstly, i highly recommend upgrading. we're on bevy 0.17 now, its getting close to 2 years since 0.14 came out. I dont remember the state of... most things from 0.14 so i can't help you with that.

If its slower when you copy a scene from a resource, i suspect the cause is that its somehow duplicating the mesh and material assets which leads to zero instancing. Bevys built-in instancing works primarily by instancing entities using the same mesh and material handles.

that last point also means that if you want to change something about each instance, doing so through changing the material will require separate material assets and therefore break instancing

if you want to change how your instanced objects show up, you should have a look at this example: https://bevy.org/examples/shaders/automatic-instancing/

there, an image is used to change the instances color, but you could of course use some other set of data or even just randomize colors for example

[regal sierra]

all models with same mesh and team is sharing the same extended materials that simply change color of mask color to team color

[barren gorge]

but the technique shown there requires MeshTag which was added 0.16 iirc

[barren gorge]

could you show the code which changes the color? mainly the CPU side where you set up the different colors/teams/whatever.

[regal sierra]

let material;

if let Some(mat) = instanced_materials.team_materials.get(&(b.model.mesh.id(), *team)) {
    material = mat.clone();
} else {
    if let Some(original) = materials.get(b.model.material.id()) {
        material = extended_materials.add(ExtendedMaterial {
            base: original.clone(),
            extension: TeamMaterialExtension {
                team_color: color,
            },
        });
    } else {
        material = extended_materials.add(ExtendedMaterial {
            base: StandardMaterial{
                ..default()
            },
            extension: TeamMaterialExtension {
                team_color: color,
            },
        });
    }

    instanced_materials.team_materials.insert((b.model.mesh.id(), *team), material.clone());
}

new_unit = commands.spawn((
    MaterialMeshBundle{
        mesh: b.model.mesh.clone(),
        material: material.clone(),
        transform: Transform::from_translation(new_unit_position),
        ..default()
    },

[barren gorge]

okay, so you do set up different material extensions per team color. I assume then that the number of units spawned per team vastly outnumbers the amount of teams?

[regal sierra]

#import bevy_pbr::pbr_fragment::pbr_input_from_standard_material
#import bevy_pbr::pbr_functions::alpha_discard

#ifdef PREPASS_PIPELINE
#import bevy_pbr::prepass_io::{VertexOutput, FragmentOutput}
#import bevy_pbr::pbr_deferred_functions::deferred_output
#else
#import bevy_pbr::forward_io::{VertexOutput, FragmentOutput}
#import bevy_pbr::pbr_functions::{apply_pbr_lighting, main_pass_post_lighting_processing}
#endif

@group(2) @binding(120) var<uniform> white_replacement: vec4<f32>;

@fragment
fn fragment(
    in: VertexOutput,
    @builtin(front_facing) is_front: bool,
) -> FragmentOutput {
    var pbr = pbr_input_from_standard_material(in, is_front);

    var base_color = pbr.material.base_color;

    let is_white = 
        base_color.r > 0.95 &&
        base_color.g > 0.95 &&
        base_color.b > 0.95 &&
        base_color.a > 0.95;

    if (is_white) {
        base_color = white_replacement;
    }

    pbr.material.base_color = base_color;

    var out: FragmentOutput;
    out.color = apply_pbr_lighting(pbr);
    out.color = main_pass_post_lighting_processing(pbr, out.color);

    return out;
}

[barren gorge]

also: is this the version that works efficiently, or the version that works inefficiently?

[regal sierra]

2 teams, thousands of units

[barren gorge]

.

[regal sierra]

this works efficiently

[barren gorge]

okay, thats what i would expect

now what about the version that works inefficiently?

[regal sierra]

the same but instead of spawning this

MaterialMeshBundle{
      mesh: b.model.mesh.clone(),
      material: material.clone(),
      transform: Transform::from_translation(new_unit_position),
      ..default()
  },

there will be spawning of scene from resource

and also I do not know how to control the material used in the scene in this case.

[barren gorge]

exactly what you do for "spawning of scene from resource" is what i'm curious about

[regal sierra]

need a little bit of time for this

[barren gorge]

Also, to be clear: There is one example of an animation implementation using vertex shader: bevys built-in PBR material. The skinning/morphing is done on the GPU, CPU-side its basically just computing the joint matrix/morph weights

[regal sierra]

when I spawned the scenes, I took the bundle with the unit from the producer entity component

pub struct UnitProductionBuildingComponent {
    pub available_to_build: HashMap<String, (UnitBundles, ProductionData)>,
    pub build_order: Vec<(i32, UnitBundles, ProductionData, CompanyTypes, (i32, i32, i32, i32, i32, i32, i32), String)>,
    pub spawn_point: Vec3,
    pub elapsed_time: u128,
}

the entity had this bundle from this resource when it was created

#[derive(Resource)]
pub struct ProducableUnits {
    pub barrack_producables: HashMap<String, (UnitBundles, ProductionData)>,
    pub factory_producables: HashMap<String, (UnitBundles, ProductionData)>,
}

and the bundle was added to this resource in the startup system. This bundle stores the scene uploaded during the startup

[barren gorge]

What does UnitBundles look like? What does the code look like that that actually does the spawning?

[barren gorge]

wait no, nevermind @regal sierra, i dont think i need to see it

you just have a Handle<Scene> you insert repeatedly, yes?

[regal sierra]

bundle is always something like this

#[derive(Clone)]
pub struct AssaultBundle {
    pub scene: Scene,
    pub lod: PbrBundle,
    pub unit_component: UnitComponent,
    pub combat_component: CombatComponent,
    pub supplies_consumer: SuppliesConsumerComponent,
    pub controller: KinematicCharacterController,
}

and spawning is something like this

match &infantry_producer.1.build_order[0].1 {
    UnitBundles::Shock(b) => {
        unit_type = b.combat_component.unit_type.clone();

        new_unit = commands.spawn((
            SceneBundle{
                scene: b.scene.clone(),
                transform: Transform::from_translation(new_unit_position),
                ..default()
            },
            b.unit_component.clone(),
            CombatComponent {
                team: *team,
                current_health: b.combat_component.current_health,
                max_health: b.combat_component.max_health,
                unit_type: b.combat_component.unit_type.clone(),
                attack_type: b.combat_component.attack_type.clone(),
                attack_animation_type: b.combat_component.attack_animation_type.clone(),
                attack_frequency: b.combat_component.attack_frequency,
                attack_elapsed_time: b.combat_component.attack_elapsed_time,
                detection_range: b.combat_component.detection_range,
                attack_range: b.combat_component.attack_range,
                enemies: b.combat_component.enemies.clone(),
                is_static: b.combat_component.is_static,
                unit_data: (
                    tile,
                    (
                        infantry_producer.1.build_order[0].3,
                        infantry_producer.1.build_order[0].4,
                        infantry_producer.1.build_order[0].5.clone(),
                    ),
                ),
            },
            b.supplies_consumer.clone(),
            b.controller.clone(),
            SelectableUnit,
            LOD{
                detailed: (b.model.mesh.clone(), material.clone()),
                simplified: b.lod.clone(),
            },
        )).id();
    },
}

[barren gorge]

yeah i think i understand the issue

To be sure: could you write a systems that logs out the number of Image, StandardMaterial, and Mesh assets every once in a while?

you should be able to see that with SceneBundle the at least one of those numbers grows a lot, while with the other way with team materials it doesn't

[regal sierra]

The problem is that it will take me some time to do it exactly as I have shown

[barren gorge]

i dont think you need to, you can honestly just test it with any scene in an emtpy bevy app

if you repeatedly clone and spawn SceneBundle, do the asset counts go up? if they do, the issue is relatively clear

[regal sierra]

ok, ill check this

[barren gorge]

Note that i suspect your "how do i update the material after spawn" woes are likely nicely solved by updating to newer bevy by using component hooks to react to the insert events for StandardMaterial (filtering on something like CombatComponent so you dont overwrite the materials of other entities)

[regal sierra]

its with "team materials" way with something like 3k entities

[barren gorge]

wait is this with you spawning or with scene? i dont understand what "tam materials" means

[regal sierra]

not with scene

i'll check for scene now

so i need to clone a scene multiple times?

[barren gorge]

i mean, thats what you were doing no? you were spawning tons of entities by cloning the scene

[regal sierra]

yeah i guess

[barren gorge]

for this to actually work and show anything make sure your scene actually contains like, a mesh with a material. ideally the same thing you use in your game

[regal sierra]

it was spawned like this

for _ in 0..2000 {
    commands.spawn(SceneBundle{
        scene: unit_assets.test.clone(),
        ..default()
    });
}

and test was loaded like this

let test: Handle<Scene> = asset_server.load(GltfAssetLabel::Scene(0).from_asset("units/soldier.glb"));

[barren gorge]

hmmm, strange

[regal sierra]

these scenes didn't really cause any problems, I had problems when I tried to use skeletal animation. The fact is that this assumes that each unit will be a bundle of many entities, and this caused lags.

I guess I can use shape keys instead of skeleton and it will work better but I will have to solve the problem of replacing the material of these entities

[barren gorge]

wait i think i missed an important step

besides the different spanwing, what are the differences between the efficient and the inefficient versions?

[regal sierra]

"inefficient" version had to deal with spawning scenes with bones besides meshes and materials

[barren gorge]

okay wait

so

the efficient version just... didn't animate at all?

am i understanding that correctly?

[regal sierra]

yeah but just having not animated scenes with bones caused issues

[barren gorge]

i'm really struggeling to understand what you actually mean, what the differences are

[regal sierra]

in one case it's just mesh and material and in other case it's the same with ready to animate armature

[barren gorge]

okay so, please respond with yes/no: Did the efficient version have any animation?

[regal sierra]

no

[barren gorge]

okay.

this whole time i thought both versions had animations...

but the one spawned with scene was slower

Okay, next question: Please explain exactly how you determined the skeletal animation version is too slow for you.

Please tell me:

• did you compile with --release when testing the performance?
• did you have any opt-level config in your Cargo.toml at the time?
• approx what number of entities with skeletal animation were being spawned?
• what did you use to check the performance?

[regal sierra]

yes
yes
2000-3000
FrameTimeDiagnosticsPlugin it was less than 10 fps

[barren gorge]

thank you

What i would recommend:

• update to bevy 0.17 (there were a bunch of optimisations since then)
• check performance with animations again
• if its still bad, read profiling.md from the bevy repo and set up Tracy for profiling, then send the saved trace here