How would I get information from an image?

[obsidian carbon]

I would like to be able to setup a level from an image file. Maybe even while the game is running.

[quiet cosmos]

Image has a public field data: Vec<u8>

[obsidian carbon]

let data = player_texture.data;```

[plush panther]

You'll want to interpret this based on the format: the exact byte order will vary

[obsidian carbon]

Are there any simple examples?

[main canopy]

Handle<Image> is just a handle. Actual images will live in Res<Assets<Image>>

[obsidian carbon]

Kay.

I think that I need a simple example.

[main canopy]

Here's how I edit my minimap image:

fn update_minimap(maze:Res<Maze>, minimap:Res<Minimap>, player:Query<&Transform, With<Player>>,mut textures:ResMut<Assets<Image>>) {
    let maze_pos = Maze::pos_to_cell(player.single().translation);
    let cell = maze_pos.round();
    let image = textures.get_mut(minimap.image.id()).unwrap();

    let dim = maze.size as usize * 2;

    for dx in -1..1 {
        for dy in -1..1{
            let cell = Vec2{x:cell.x + dy as f32, y:cell.y + dx as f32};
            if cell.x >= 0.0 && cell.y >= 0.0 && cell.x < maze.size as f32 && cell.y < maze.size as f32 {
                let maze_index = maze.cell_to_index(cell);
                let pixel_index = (cell.x as usize * 2 * 16 + dim*16 * (cell.y as usize * 2));
                
                // ... populate aa..bb
    
                for channel in 0..4 {
                    let aa = aa[channel].to_ne_bytes();
                    for byte in 0..4 {
                        image.data[pixel_index + channel*4 + byte ] = aa[byte];
                    }
                    let ab = ab[channel].to_ne_bytes();
                    for byte in 0..4 {
                        image.data[pixel_index + 16 + channel*4 + byte ] = ab[byte];
                    }
                    let ba = ba[channel].to_ne_bytes();
                    for byte in 0..4 {
                        image.data[pixel_index + dim*16 + channel*4 + byte ] = ba[byte];
                    }
                    let bb = bb[channel].to_ne_bytes();
                    for byte in 0..4 {
                        image.data[pixel_index + dim*16 + 16 + channel*4 + byte ] = bb[byte];
                    }
                }
            }
        }
    }
}

So the key lines there are

let image = textures.get_mut(minimap.image.id()).unwrap();
//... and then
image.data[pixel_index + channel*4 + byte ] = aa[byte];

My image is f32_RGBA so there are 4 bytes per channel, and 4 channels per pixel

You can get the 4 f32 of a pixel with Color::as_rgba_f32()

Then you can get 4 bytes from f32 with f32::to_ne_bytes()

[obsidian carbon]

How did you get the image file in the first place?

And what parts are x and y?

[main canopy]

I don't have that one in a file, it's a minimap so it's generated, rather than the other way around.

[obsidian carbon]

Kay.

Thanks.

[main canopy]

look at let pixel_index = ...

[obsidian carbon]

let pixel_index = (cell.x as usize * 2 * 16 + dim*16 * (cell.y as usize * 2));

I did try looking this up though.

[main canopy]

yes, that's how to convert x and y to index in the image data. Well it's x + image_width * y

[obsidian carbon]

Kay,

I will look through this then.

I think I can figure it out from here.

Thanks!

[main canopy]

Good luck!

[quiet cosmos]

now im curious, how will you use an image to setup a level?
is it a grid level where you spawn things depending on the pixel color?

[obsidian carbon]

That is the idea.

[quiet cosmos]

cool!

glhf

[obsidian carbon]

I still can't figure out how to get the image pixels.

I am going to try https://crates.io/crates/bevy-mutate-image.

[quiet cosmos]

you store the Handle<Image> from AssetServer.load(...)
get the image from ResMut<Assets<Image>> (images) using your handle like: let image = images.get_mut(handle.id()).unwrap()
and now you can operate on image.data, which holds all of the pixels, u8 for each channels in some order, for example RGBARGBARGBA...

[quiet cosmos]

here is a simple example of using image's pixel data to setup a level

use bevy::prelude::*;

fn main() {
    App::new()
        .add_plugins(DefaultPlugins)
        .add_systems(Startup, setup)
        .add_systems(Update, spawn)
        .run();
}

#[derive(Resource)]
struct Level(Handle<Image>);

fn setup(mut commands: Commands, assets: Res<AssetServer>) {
    commands.spawn(
        Camera3dBundle {
            transform: Transform::from_xyz(0.0, 3.0, 5.0).looking_at(Vec3::new(1.0, 0.0, 1.0), Vec3::Y),
            ..default()
        }
    );

    commands.spawn(
        PointLightBundle {
            transform: Transform::from_xyz(1.0, 7.0, 1.0),
            ..default()
        }
    );

    commands.insert_resource(
        Level(assets.load("level.png"))
    );
}

fn spawn(
    mut commands: Commands,
    mut meshes:    ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>,
    mut images:    ResMut<Assets<Image>>,
    level: Res<Level>,
    input: Res<ButtonInput<KeyCode>>
) {
    // just to give AssetServer time (avoids unwrapping None)
    if !input.just_pressed(KeyCode::Space) {
        return;
    }

    let image = images.get_mut(level.0.id()).unwrap();

    for y in 0..image.height() {
        for x in 0..image.width() {
            let i = ((x + y * image.width()) * 4) as usize; // `* 4` because of the assumed RGBA format

            let red   = image.data[i]     == u8::MAX; // u8::MAX = 255
            let green = image.data[i + 1] == u8::MAX;
            let blue  = image.data[i + 2] == u8::MAX;

            if red {
                commands.spawn(PbrBundle {
                    mesh:      meshes.add(Cuboid::new(0.5, 0.5, 0.5)),
                    material:  materials.add(Color::RED),
                    transform: Transform::from_xyz(x as f32, 0.25, y as f32),
                    ..default()
                });
            }

            if green {
                commands.spawn(PbrBundle {
                    mesh:      meshes.add(Rectangle::new(1.0, 1.0)),
                    material:  materials.add(Color::GREEN),
                    transform: Transform {
                        translation: Vec3::new(x as f32, 0.0, y as f32),
                        rotation:    Quat::from_rotation_x(-std::f32::consts::FRAC_PI_2),
                        ..default()
                    },
                    ..default()
                });
            }

            if blue {
                commands.spawn(PbrBundle {
                    mesh:      meshes.add(Sphere::new(0.3)),
                    material:  materials.add(Color::BLUE),
                    transform: Transform::from_xyz(x as f32, 0.3, y as f32),
                    ..default()
                });
            }
        }
    }
}

please ask if anything in the code is unclear

running this code on a 3x3 image like this

produces

[obsidian carbon]

Whoa.

[obsidian carbon]

let level_image = asset_server.load("levels/test_level.png").id();Will this work?

(It does not)

Never mind.

I figured it out.

let level_image = images.get_mut(asset_server.load("levels/test_level.png").id()).unwrap();

Never mind.

It panics.

called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace```

[quiet cosmos]

yes, thats why i made it on button input and not instant

asset loading happens in the background asynchronously

[quiet cosmos]

i did this, so the AssetServer will have time to load it

even if the Handle<Image> exists, doesnt mean its loaded

you can keep checking it's load state if you want

but i went with a simple delay for this example

[quiet cosmos]

if you try to do it instantly like this

you will try to .unwrap() on None (because its not loaded yet)

thats a big nono

kinda like dereferencing a NULL in C, or nullptr in C++

[obsidian carbon]

Kay.

[obsidian carbon]

How do I check the loading state?

[quiet cosmos]

this is a snippet from one of my projects, where i needed to insert a skybox into a camera, but i could only do that after AssetServer has loaded it
this function is registered in Update:

fn insert_skybox (
    mut commands: Commands,
    assets:       Res<AssetServer>,
    skybox:       ResMut<SkyBox>,
    mut images:   ResMut<Assets<Image>>,
    q_camera:     Query<Entity, With<SpawnCamera>> // <-- SpawnCamera is a temporary placeholder
) {
    if let Ok(entity) = q_camera.get_single() {
        if assets.load_state(&skybox.handle) == Loaded { // <-- the important line 1
            let image = images.get_mut(&skybox.handle).unwrap();

            image.reinterpret_stacked_2d_as_array(image.height() / image.width());

            image.texture_view_descriptor = Some(TextureViewDescriptor {
                dimension: Some(Cube),
                ..default()
            });

            commands.entity(entity).insert(Skybox { image: skybox.handle.clone(), brightness: 250.0 });
            commands.entity(entity).remove::<SpawnCamera>(); // <-- the important line 2
        }
    }
}

i commented the lines you should focus at

to check the state, you can have a placeholder component, in my case SpawnCamera

as long as it exists, i keep checking if the Handle<Image> from AssetServer is loaded (important line 1)

and when the state is bevy::asset::LoadState::Loaded, i work with the Image

and then delete the placeholder SpawnCamera (important line 2)

_ _
_ _
_ _
or you could have a state instead of a placeholder component, that you turn to some value, and run it in Update, but with a condition
for example you would register the function with .run_if(in_state(MyImage::NotYetLoaded)), and instead of my "important line 2", you could change the state to MyImage::WasLoaded

[obsidian carbon]

I don't get it.

I have an idea.

[quiet cosmos]

as long as the placeholder component exists, because we delete it when the asset was loaded, therefore you dont need to keep checking for the state after

its like having a sticky note, you remove it when you finish the task 😉

[obsidian carbon]

    let level_image = images.get_mut(current_level.0.id()).unwrap();
}```

[obsidian carbon]

Why is this a boolean @quiet cosmos?

Can I get a Color value instead?

let red   = level_image.data[i]; // Figures out the rgb value. Also something that I don't understand.
let green = level_image.data[i + 1];
let blue  = level_image.data[i + 2];
let rgb = Color::rgb_u8(red, green, blue);```

Does this work?

If I got this first try then I will be very happy.

[quiet cosmos]

i am comparing if the color is equal to 255, aka a full channel, since rgb colors go from 0 to 255, it was just an example

[quiet cosmos]

the image.data[index] is just one channel of a specific pixel

the most common case is, that colors are stored in RGBA, so 4 components (red, green, blue, alpha)

[obsidian carbon]

I don't care about alpha.

[quiet cosmos]

thats why the index is * 4 in the end, if you want to for example go to the 10th pixel, you need to go to index 10*4=40

[quiet cosmos]

just make sure to skip it if it is in your image, it is also possible that the image is just RGB

what i do is check it on a tiny image by just pretty printing it in rust println!("{:?}", image.data);

this way you can see the format, or you can probably find it in the .texture_descriptor field

[obsidian carbon]

The level is flipped for me.

@quiet cosmos

    mut images: ResMut<Assets<Image>>,
    total_game_ticks: Res<TotalGameTicks>,
    current_level: Res<CurrentLevel>,
    mut query: Query<(&TilePos, &mut TileVisible)> 
) {
    if total_game_ticks.0 == 1 {
        let level_image = images.get_mut(current_level.0.id()).unwrap();

        for (tile_pos, mut tile_visibility) in query.iter_mut() {
            let i = ((tile_pos.x + tile_pos.y * level_image.width()) * 4) as usize;
            let red   = level_image.data[i]; // Figures out the rgb value.
            let green = level_image.data[i + 1];
            let blue  = level_image.data[i + 2];
            let rgb = Color::rgb_u8(red, green, blue);

            if rgb != WALL_COLOR {
                tile_visibility.0 = false;
            }

            println!("{:?}", rgb);
        }
    }
}```

[obsidian carbon]

Makes:

It is flipped.

[quiet cosmos]

just changing the sign somewhere should help (for example y -> -y)

[obsidian carbon]

let i = ((tile_pos.x + -tile_pos.y * level_image.width()) * 4) as usize;?

Like that?

That gives me an error.

cannot apply unary operator '-'

[quiet cosmos]

not in the indexing

keep indexing how it is

i mean change the y where you spawn the tiles (the Transform.translation.y)