Direct Video Android - Smooth 360, 4k and 8k Video Playback on Android

[burnt sand]

A pet peeve of mine in relation to Unreal Engine is that if you are on Android , video playback performance is abysmal. It just isn't possible to play video of any modern quality level using Unreal Android without your game frame-rate dropping. This is because since Unreal updated to use Vulkan graphics API, the Unreal media player classes use Java and for each video frame it is decoded to CPU then sent back to the GPU again.

This (paid) plugin plays video in Unreal using hardware acceleration from end-to-end, which means buttery smooth video without frame stutters. My first customer for this project uses it to play 8k 360 video on a Meta Quest 3 and the headset maintains a stable 72 fps game frame rate.

https://joemarshall.github.io/directvideo/
https://www.unrealengine.com/marketplace/en-US/product/direct-video-android-beta

[carmine kraken]

Hello, @burnt sand
How do you think, will it package well for Vive Focus 3 headset?
Maybe someone alredy test it?
Cause I have 7 fps with 8k360 video using UE media player 🥲

[burnt sand]

I'm not sure - I can probably build a test apk assuming it's just a matter of downloading and enabling the vive openxr plugin. In theory it should work because it's the same CPU as quest / Pico, but not tested yet.

[carmine kraken]

Sure, it's the same, but lenses resolution is different
It would be great if you can provide test apk for Vive

[burnt sand]

https://github.com/joemarshall/directvideo/releases/tag/1.0

I built a demo for vive focus - I have no idea if the project is setup right as I don't have one to test against, but it is worth a try, apk on the link above.

@carmine kraken

[carmine kraken]

Thank you for help
I tried to install sample on headset, but it throws an error
Anyway, yesterday we bought plugin and I test it now

@burnt sand So i added plugin to the project and built it. It works!
It improve fps from 7 to 45-55 for 8k video on Vive Focus 3
However, it's not target 90, and even 72 like on Meta Quest 3(we tested sample on Meta Quest 3 as well)
As I said, lenses have 5k resolution: https://business.vive.com/eu/product/vive-focus3/specs
But CPU the same

[carmine kraken]

@burnt sand I tuned up your project for Meta to make it run on Vive Focus 3
I set pixel density to 0.7, limit frame rate to 75 and I managed to achieve target ~75 fps in app
I used shipping build as well
However I still have screen shift in the middle(but it's not on Meta quest 3) when I rotate head 🥲
It also triggers Frame Time spike
It's similar to vsync=off in shooter games, when you move camera fast
It's not visible on screen record, so I made a pic for you:

Maybe you have some ideas how to fix it?
Im pretty sure its smth with vive+plugin connection

[void thicket]

Can you not turn vsync on? That's literally the fix for this. It's what it was designed to do.

[carmine kraken]

Thank you for help
I tried this, but nothing changes:

[void thicket]

😦

[carmine kraken]

@void thicket Maybe Im wrong and there is another way to turn on vsync for mobile app

[void thicket]

I don't know, sorry! I just know that vsync is designed to stop screen tearing like that.

[burnt sand]

That's half good news. The good news is that it the video decoding side of things is running fast enough. The bad news is that something is making the draw calls too slow.

You can tell this because decoding happens in it's own thread, so it doesn't slow the game thread or the render thread. Draw calls get put into the render command buffer, and if they take too long the whole frame stalls. If a frame stalls on vr, you get this weird jumping effect you're seeing because the head tracking isn't being run at consistent frame rate.

Given it looks like that side of things is holding stuff up, I'll take a look at the drawing code with Snapdragon profiler - it may be that there's something I can tweak to improve performance in the vulkan code.

[carmine kraken]

Thank you!
Yesterday I add fade-to-black effect and notice some white stripes on the upper part, during the fade, like on pic:

Maybe this will help you

[burnt sand]

I've just tested and I can play 8k 60fps fine on pico neo 3, but that's lower resolution. I've done a few tidying up changes to vulkan things which I don't think will make a big difference to performance, but might slightly - if you could email me on [e-mail] I can send you an updated plugin to test.

[carmine kraken]

I sent an email
What about video... I tried to play 8k 8192x4096 30fps with bitrate 35Mbps

I mostly relied on Meta video guide, because I didn't find it from Vive
https://creator.oculus.com/media-studio/documentation/video-spec/

[carmine kraken]

And I also reduce PixelDensity to compensate higher resolution of ViveFocus3: https://forum.htc.com/topic/12383-vive-focus-3-low-performance-in-ue4-build/

I tried vr.PixelDensity 0.8
And 0.7, and even 0.5
It helps with overall fps, but not with "Frame Rate spike" 🥲

[carmine kraken]

@burnt sand I tested new plugin, in short words its huge improvement, but not perfect. I send you more details in email

[carmine kraken]

@burnt sand Hi!
Did you recieve my trace file? I just want to know that I send everything correctly and this is what you wanted

[burnt sand]

@carmine kraken I released an updated version which now offers the option to render directly to a mesh, which should in theory be much faster (it was already happy at 120fps on quest, so I can't verify this, but it reports about 50% or fewer fragment writes, so it should be.) There's a focus build of the mesh direct rendering on the https://github.com/joemarshall/directvideoexample releases.

For anyone reading this, this is now on fab store by the way - https://www.fab.com/listings/3259a389-1214-4312-a6aa-14fc8012ce7b