Improve performance in a conversion BGRA -> RGB operation | Rust Programming Language Community | Page 1

young kayak Jun 21, 2023, 7:28 AM

#

I'm capturing the screen using Windows DXGI Desktop Duplication API through a library and the pic_data contains raw pixel data of a frame represented in a vector of u8 BGRA format. I realized that my screen capture was slow due to the time it takes to convert pixel data to the correct color type. I was wondering how I could improve the following code to reduce the time it takes to complete the conversion operation.

Time taken for RGB conversion: 337.2225ms

let rgb_data: Vec<u8> = pic_data
                .as_bgra()
                .to_owned()
                .iter()
                .copied()
                .flat_map(|pixel| [pixel.r, pixel.g, pixel.b])
                .collect();

Time taken for RGB conversion: 113.4304ms

for rgba in pic_data.chunks(4) {
    rgb_data.push(rgba[2]); // Red
    rgb_data.push(rgba[1]); // Green
    rgb_data.push(rgba[0]); // Blue
}

young kayak Jun 21, 2023, 8:19 AM

#

I'm not sure if this post is applicable:
https://users.rust-lang.org/t/converting-a-bgra-u8-to-rgb-u8-n-for-images/67938/13

Apparently using the swizzling method was the fastest performing - bgra×4_to_rgb×4 (by ref)/swizzle (-39.6%)

The Rust Programming Language Forum

Converting a BGRA &[u8] to RGB [u8;N] (for images)?

Sure, why not: Without target-cpu=native: With target-cpu=native: (This is the size magick gave for the SVG->webp conversions. Order on graph = chunks, loop, overwrite, raw, scatter, shift, swizzle) With, target-cpu=native, performance no change bgra×4_to_rgb×4 (by move)/loop (+2.53%, p = 0.06) bgra×4_to_rgb×4 (by move)/sc...

manic quail Jun 21, 2023, 9:32 AM

#

does it matter what the alpha value is?

#

chopping off the A could result in a decent perf decrease

#

since you either need to realloc or remove a bunch of stuff from the vec of data

#

going from bgra to rgba is just one swap

#

it seems to take about 7 times longer on my machine to allocate rgb data to a new vec, compared to just swapping each R and B value

#

oh wow don't even ask about removing individual elements

modest ivy Jun 21, 2023, 10:35 AM

#

young kayak I'm capturing the screen using Windows DXGI Desktop Duplication API through a li...

ur using the pixel crate right

young kayak Jun 21, 2023, 11:06 AM

#

manic quail does it matter what the alpha value is?

I'll check the code again when I get back home. If i recall correctly, x264 wanted RGB and not RGBA. I'm feeding rgb_data to x264 that encodes it.

young kayak Jun 21, 2023, 11:06 AM

#

modest ivy ur using the pixel crate right

I have to check, but I don't think I am.

#

I would like to avoid complexity and the use of additional library as much as possible.

modest ivy Jun 21, 2023, 11:08 AM

#

why do you have to_owned, then iter then copied

#

let rgb_data: Vec<u8> = pic_data
                .as_bgra()
                .iter()
                .copied()
                .flat_map(|pixel| [pixel.r, pixel.g, pixel.b])
                .collect();

manic quail Jun 21, 2023, 11:23 AM

#

yeah that’s probably close to the fastest you could get(?)

#

i misread and thought pic data was already a collection of u8

#

if pic_data internally holds just a Vec of u8, you ideally interact with that since you want a u8 out of it

stable veldt Jun 21, 2023, 12:02 PM

#

where is as_bgra from?

#

like, what is the type of pic_data

modest ivy Jun 21, 2023, 12:04 PM

#

i'm pretty sure that's from rgb crate, because i suggested that crate to them before

#

https://docs.rs/rgb/0.8.36/rgb/trait.FromSlice.html#tymethod.as_bgra

#

of which i have already suggested a solution here #rust-discussions-1 message

#

hmm why did i do to_owned().iter().copied lol

#

i just copy pasted the code lol

#

hmm why did you do flat_map instead of map(RGBA::from) as i suggested?

young kayak Jun 21, 2023, 12:30 PM

#

modest ivy hmm why did you do `flat_map` instead of `map(RGBA::from)` as i suggested?

It turns out I needed RGB not RGBA

#

But, I'll take a look at the code again when I get back home.

stable veldt Jun 21, 2023, 12:32 PM

#

use core::ptr;

fn shuffle_bgra4_to_rgb4_a4(v: [u8; 16]) -> [u8; 16] {
    [
        v[2], v[1], v[0],
        v[6], v[5], v[4],
        v[10], v[9], v[8],
        v[14], v[13], v[12],
        v[3], v[7], v[11], v[15]
    ]
}

pub fn bgra_to_rgb(bgra: &[u8]) -> Vec<u8> {
    let mut out: Vec<u8> = Vec::with_capacity(bgra.len() * 3 / 4 + 16);

    unsafe {
        let mut i = 0;
        let mut out_len = 0;
        let out_ptr = out.as_mut_ptr();
        while i + 16 <= bgra.len() {
            let bgra4 = *<&[u8; 16]>::try_from(&bgra[i..i+16]).unwrap();
            let rgb4_a4 = shuffle_bgra4_to_rgb4_a4(bgra4);
            ptr::copy_nonoverlapping(rgb4_a4.as_ptr(), out_ptr.add(out_len) as *mut u8, 16);
            out_len += 12;
            i += 16;
        }

        let mut bgra4 = [0u8; 16];
        ptr::copy_nonoverlapping(bgra.as_ptr().add(i), bgra4.as_mut_ptr(), bgra.len() - i);
        let rgb4_a4 = shuffle_bgra4_to_rgb4_a4(bgra4);
        ptr::copy_nonoverlapping(rgb4_a4.as_ptr(), out_ptr.add(out_len) as *mut u8, 16);
        out_len += (bgra.len() - i) / 4;

        out.set_len(out_len);
    }
    
    out
}

#

this should be fast on x86-64

modest ivy Jun 21, 2023, 12:33 PM

#

okay damn

stable veldt Jun 21, 2023, 12:34 PM

#

although it needs SSE3

#

https://rust.godbolt.org/z/1W4EKbT5h

Compiler Explorer - Rust (rustc nightly)

use core::ptr;

fn shuffle_bgra4_to_rgb4_a4(v: [u8; 16]) -> [u8; 16] {
[
v[2], v[1], v[0],
v[6], v[5], v[4],
v[10], v[9], v[8],
v[14], v[13], v[12],
v[3], v[7], v[11], v[15]
]
}

pub fn bgra_to_rgb(bgra: &[u8]) -> Vec<u8> {
let mut out: Vec<u8> = Vec::with_capacity(bgra.len() * 3 / 4 + 16);

...

young kayak Jun 21, 2023, 12:43 PM

#

stable veldt although it needs SSE3

Interesting, is that instruction set supported by modern CPUs?

stable veldt Jun 21, 2023, 12:44 PM

#

yes

young kayak Jun 21, 2023, 12:45 PM

#

Nice, I'll have to have a look and benchmark it.

young kayak Jun 21, 2023, 1:50 PM

#

manic quail if pic_data internally holds just a Vec of u8, you ideally interact with that si...

// Create a vector to store pixel data from the captured frame.
let mut pic_data: Vec<u8> = vec![0; (width * height * 4) as usize]; // for BGRA

// extract pixel data from the GPU texture and store it in the pic_data vector for further processing.
texture_reader.get_data(&mut pic_data, &tex).unwrap();

young kayak Jun 21, 2023, 2:09 PM

#

stable veldt ```rust use core::ptr; fn shuffle_bgra4_to_rgb4_a4(v: [u8; 16]) -> [u8; 16] { ...

wow this improved a lot.

stable veldt Jun 21, 2023, 2:13 PM

#

young kayak wow this improved a lot.

got numbers?

#

it could be a lot better but I ran into issues with the Rust compiler unrolling the loop

young kayak Jun 21, 2023, 2:15 PM

#

stable veldt got numbers?

Time taken for RGB conversion: 25.6561ms

#

down from 113.4304ms

stable veldt Jun 21, 2023, 2:17 PM

#

@young kayak could you try this version?

#

https://rust.godbolt.org/z/K5zjTrona

Compiler Explorer - Rust (rustc nightly)

use core::ptr;

fn shuffle_bgra4_to_rgb4_a4(v: [u8; 16]) -> [u8; 16] {
[
v[2], v[1], v[0],
v[6], v[5], v[4],
v[10], v[9], v[8],
v[14], v[13], v[12],
v[3], v[7], v[11], v[15]
]
}

unsafe fn copy_bgra4_to_rgb4_a4(src: *const u8, dst: *mut u8, srclen: usize) {
let mut bgra4 = [0u8; 16];
ptr::...

#

I can't get it to compile properly without the black_box which hurts performance sadly

#

oh I have an idea perhaps

#

yay it worked

#

try this @young kayak

#

https://rust.godbolt.org/z/TjcWd53G3

#

nvm that's invalid let me fix bug

young kayak Jun 21, 2023, 2:20 PM

#

I think copy_bgra4_to_rgb4_a4() is not called

stable veldt Jun 21, 2023, 2:21 PM

#

it's not it's old

#

https://rust.godbolt.org/z/c9x98Y334

#

this should be right

#

@young kayak please ignore the earlier two versions, try the latest one

young kayak Jun 21, 2023, 2:22 PM

#

Time taken for RGB conversion: 18.0167ms

#

it's improved

stable veldt Jun 21, 2023, 2:22 PM

#

that's the latest one?

young kayak Jun 21, 2023, 2:22 PM

#

yep

stable veldt Jun 21, 2023, 2:23 PM

#

what compiler flags do you use to compile?

young kayak Jun 21, 2023, 2:23 PM

#

npm run tauri dev

stable veldt Jun 21, 2023, 2:23 PM

#

no idea what that is

#

...are you targetting wasm?

young kayak Jun 21, 2023, 2:24 PM

#

I'm using Tauri framework

young kayak Jun 21, 2023, 2:25 PM

#

stable veldt ...are you targetting wasm?

I'm not sure... I know one of the library I use requires NASM

stable veldt Jun 21, 2023, 2:26 PM

#

@young kayak could you do tauri dev --verbose and see what commands it runs

young kayak Jun 21, 2023, 2:28 PM

#

npm verb cli C:\Program Files\nodejs\node.exe C:\Users\user\AppData\Roaming\npm\node_modules\npm\bin\npm-cli.js
npm info using npm@9.6.7
npm info using node@v18.16.0
npm verb title npm run tauri dev
npm verb argv "run" "tauri" "dev" "--loglevel" "verbose"
npm verb logfile logs-max:10 dir:C:\Users\user\AppData\Local\npm-cache\_logs\2023-06-21T14_27_15_522Z-
npm verb logfile C:\Users\user\AppData\Local\npm-cache\_logs\2023-06-21T14_27_15_522Z-debug-0.log

stable veldt Jun 21, 2023, 2:29 PM

#

it should run cargo

#

@young kayak ah you're on windows

#

could you try

#

set RUSTFLAGS=-C target-cpu=native

#

then make sure tauri re-builds

#

make you need a clean command or something

#

@young kayak wait have you not been using tauri dev --release so far?

young kayak Jun 21, 2023, 2:31 PM

#

no i have not

stable veldt Jun 21, 2023, 2:32 PM

#

well that's going to make a huge difference then

#

try first tauri dev --release without the RUSTFLAGS

#

and then again with

#

(to reset the rustflags just do set RUSTFLAGS=)

young kayak Jun 21, 2023, 2:33 PM

#

tauri build ?

stable veldt Jun 21, 2023, 2:33 PM

#

idk how you can get tauri to clean in between, perhaps you have to just delete the target folder

#

npm run tauri dev --release

#

looking at performance not in release mode is always folly

young kayak Jun 21, 2023, 2:34 PM

#

ok I deleted the target folder

#

ok building is going to take a while

#

I've reseted flags, deleted target folder and now running npm run tauri dev --release which is rebuilding.

#

Time taken for RGB conversion: 18.1744ms it's about the same.

#

I'll try with npm run tauri build

#

Time taken for RGB conversion: 6.002ms
with npm run tauri build

smoky shadow Jun 21, 2023, 4:45 PM

#

stable veldt ``` set RUSTFLAGS=-C target-cpu=native ```

try to add also -C target-feature=+sse3,+avx if your target and cpu allows it, this might give you a perf boost as it will try to use SIMD ops

stable veldt Jun 21, 2023, 4:46 PM

#

smoky shadow try to add also `-C target-feature=+sse3,+avx` if your target and cpu allows it,...

no, that's what target-cpu=native does

#

target-cpu=native sets all the flags the cpu on the host supports

smoky shadow Jun 21, 2023, 4:46 PM

#

stable veldt `target-cpu=native` sets all the flags the cpu on the host supports

oh I always thought that you need to add the extra features on top, good to know thanks.

young kayak Jun 22, 2023, 1:03 AM

#

Time taken for RGB conversion: 3.8135ms using ```set RUSTFLAGS=-C target-cpu=native

young kayak Jun 22, 2023, 1:52 AM

#

fn encode_frames(frames: Vec<Vec<u8>>, width: u32, height: u32) -> Result<()> {
    let mut file = File::create("../vid/captured_video.h264").unwrap();
    let mut encoder = Encoder::builder()
        .fps(60, 1)
        .build(Colorspace::RGB, width as _, height as _)
        .unwrap();
    {
        let headers = encoder.headers().unwrap();
        file.write_all(headers.entirety()).unwrap();
    }
    for (index, mut pic_data) in frames.into_iter().enumerate() {
        let image = Image::rgb(width as _, height as _, &pic_data);
        let (data, _) = encoder.encode((60 * index) as _, image).unwrap();
        file.write_all(data.entirety()).unwrap();
    }
    {
        let mut flush = encoder.flush();
        while let Some(result) = flush.next() {
            let (data, _) = result.unwrap();
            file.write_all(data.entirety()).unwrap();
        }
    }
    Ok(())
}

async fn capture_frames(dupl: &mut DesktopDuplicationApi, width: u32, height: u32) -> Result<Vec<Vec<u8>>> {
    let (device, ctx) = dupl.get_device_and_ctx();
    let mut texture_reader = TextureReader::new(device, ctx);
    let mut frames: Vec<Vec<u8>> = vec![];
    let start_time = Instant::now();

    while start_time.elapsed() < Duration::from_secs(10) {
        let tex = dupl.acquire_next_vsync_frame().await;
        if let Ok(tex) = tex {
            let mut pic_data: Vec<u8> = vec![0; (width * height * 4) as usize];
            texture_reader.get_data(&mut pic_data, &tex).unwrap();
            // use bgra_to_rgb() to efficiently convert BGRA to RGB
            frames.push(bgra_to_rgb(&pic_data));
        }
    }
    Ok(frames)
}

#

 match capture_frames(&mut dupl, width, height).await {
    Ok(frames) => {
        match encode_frames(frames, width, height) {
            Ok(()) => println!("Video capture and encoding successful."),
            Err(e) => println!("Error during encoding: {:?}", e),
        }
    },
    Err(e) => println!("Error during frame capture: {:?}", e),
}

Is it necessary to return frames, is there a way to return the output of the conversion bgra_to_rgb(&pic_data) without having to create a new vector ?

manic quail Jun 22, 2023, 2:19 AM

#

why is it a vec of vec?

#

@young kayak

#

ohh wait nvm i see

#

creating a new vec is going to be the fastest

young kayak Jun 22, 2023, 2:20 AM

#

the encode part is from the x264 rust library

manic quail Jun 22, 2023, 2:23 AM

#

i mean again ideally you don't have to get rid of the A value

#

since that's just a mem::swap() for every four elements

#

will easily be the fastest

#

but no, in this case, it's fastest just to create a new Vec

young kayak Jun 22, 2023, 2:24 AM

#

are you referring to frames 2d vec ?

manic quail Jun 22, 2023, 2:25 AM

#

hm?

young kayak Jun 22, 2023, 2:26 AM

#

mem::swap() for every four elements what do you mean by this ?

manic quail Jun 22, 2023, 2:26 AM

#

im just talking about the bgra to rgba conversion

#

you wouldn't need to create a new vec if you didn't have to get rid of the A value

young kayak Jun 22, 2023, 2:27 AM

#

ah, right. The instruction provided by @stable veldt was the fastest.

manic quail Jun 22, 2023, 2:27 AM

#

yes

#

for bgra to rgb that's basically the fastest you can get

young kayak Jun 22, 2023, 2:28 AM

#

yep, I wonder if there's a way to capture the correct format rather than BGRA...

manic quail Jun 22, 2023, 2:28 AM

#

yeahhh

young kayak Jun 22, 2023, 2:29 AM

#

https://github.com/rhinostream/win_desktop_duplication

#

I'm using this library

#

that captures as BGRA

#

https://github.com/rhinostream/win_desktop_duplication/blob/master/src/texture.rs#L66-L121

#

wouldn't the encoder want YUV instead of RGB ? I see there is YUV420 there.

young kayak Jun 22, 2023, 2:54 AM

#

https://github.com/rhinostream/win_desktop_duplication/blob/master/src/duplication.rs#L299-L302

#

let supported_formats = [DXGI_FORMAT_B8G8R8A8_UNORM, DXGI_FORMAT_R10G10B10A2_UNORM, DXGI_FORMAT_R16G16B16A16_FLOAT];
let device: IDXGIDevice4 = dev.cast().unwrap();
let dupl: WinResult<IDXGIOutputDuplication> = unsafe { output.as_raw_ref().DuplicateOutput1(&device, 0, &supported_formats) };

I think this tells it to select BGRA.

#

would it be faster to convert from RGBA -> RGB ?

#

https://learn.microsoft.com/en-us/windows/win32/api/dxgiformat/ne-dxgiformat-dxgi_format

#

or capture in RGBA format and have the encoder to ignore the last "alpha" value, that way we won't need to convert at all ?

young kayak Jun 22, 2023, 4:20 AM

#

I wouldn't mind waiting a bit longer in the encoding process as long as the screen capturing isn't dropping frames.

manic quail Jun 22, 2023, 4:28 AM

#

young kayak would it be faster to convert from RGBA -> RGB ?

probably slightly

#

but i don't think that's the main bottleneck

young kayak Jun 22, 2023, 4:30 AM

#

Hmm, I'm pretty sure i measured the conversion time and in release build it's like 3ms for 1 frame.

#

I'm not sure if that's fine or bad

manic quail Jun 22, 2023, 4:57 AM

#

that's less than a 1/5th of 1 frame

#

at 60fps

#

basically no noticeable stutter

young kayak Jun 22, 2023, 5:00 AM

#

What about higher framerate? Like 100 or 120?

young kayak Jun 22, 2023, 9:55 PM

#

I've got great news, I'm able to avoid color type conversion entirely

#

I can feed the encoder (x264) BGRA color format without converting.

young kayak Jun 22, 2023, 10:05 PM

#

young kayak ```rust fn encode_frames(frames: Vec<Vec<u8>>, width: u32, height: u32) -> Resul...

I changed RGB in the encode_frames() function to BGRA...

#

Now despite having while start_time.elapsed() < Duration::from_secs(10) my recording is capturing more time than 10 seconds now.

young kayak Jun 22, 2023, 11:46 PM

#

Thanks everyone for your help in this matter.
I've now run into another problem:
#1121586405068525630

young kayak Jun 24, 2023, 12:51 AM

#

Instantaneous frame rate: 166.76952 FPS
Instantaneous frame rate: 162.62807 FPS
Instantaneous frame rate: 170.23015 FPS
Instantaneous frame rate: 161.93546 FPS
Instantaneous frame rate: 170.75044 FPS
Instantaneous frame rate: 165.93379 FPS
Instantaneous frame rate: 348.09247 FPS
Instantaneous frame rate: 196.537 FPS
Instantaneous frame rate: 168.1916 FPS
Instantaneous frame rate: 165.23463 FPS

let start_time = Instant::now();
    while start_time.elapsed() < Duration::from_secs(10) {
        let frame_start_time = Instant::now();
        
        // DXGI AcquireNextFrame()
        let tex = dupl.acquire_next_vsync_frame().await;
        
        if let Ok(tex) = tex {
            let mut pic_data: Vec<u8> = vec![0; (width * height * 4) as usize];
            texture_reader.get_data(&mut pic_data, &tex).unwrap();
            frames.push(pic_data);
        }
        
        // Calculate frame rate for this frame and write to file
        let elapsed_secs = frame_start_time.elapsed().as_secs_f32();
        let frame_rate = 1.0 / elapsed_secs;
        writeln!(file, "Instantaneous frame rate: {} FPS", frame_rate);
    }

I don't know why it's capturing more frames than display's refresh rate which is 165Hz.

#Improve performance in a conversion BGRA -> RGB operation