Phobos - Vulkan abstraction library in Rust

[night charm]

At the start of the frame you can make one

Then if you want to make a buffer for that frame and use it across passes, you can allocate it immediately

When recoding you give the pool to the graph, which will pass it to each pass callback

You can then also use the same pool to allocate resources local to a pass, for example a uniform buffer

[hybrid pilot]

No worries and thanks

That makes perfect sense

[night charm]

[languid fern]

I have a slight feeling... that the doctests are failing/outdated a bit

[night charm]

very much

cargo test --tests

to skip them

i think

actually its not so much them being out of date but mostly them not having enough context to really compile

[languid fern]

Ahhh I see okok

Yea I just noticed cause I tested everything in my workspace lol

Yea since now I have tight integration with phobos in my engine I'll be submitting prs and issues much more frequently (and possibly fixing them myself which is even better)

[night charm]

Very nice

[languid fern]

for #70 something like this should work right?

pub unsafe fn reset(&mut self) -> Result<()> {
    // Compressed size after reset when we have multiple buffers
    if self.buffers.len() > 1 {
        // Find the largest block (could optimize this by storing index instead)
        let index = self.buffers.iter().position(|buffer| buffer.size() == self.largest_block_size).unwrap();
        let buffer = self.buffers.remove(index);
        self.buffers.clear();
        self.buffers.push(buffer);
    }

    self.current_buffer = 0;
    self.local_offset = 0;

    return Ok(());
}

This definitely saves you from having to "compress" allocations but the heuristic that depicts what block size / block that gets chosen might need some tuning

[night charm]

yep that looks good

[languid fern]

let swapchain = phobos::image!("swapchain");

let clear_pass = PassBuilder::<domain::All>::render("clear")
    .clear_color_attachment(&swapchain, ClearColor::Float([0.0; 4])).unwrap()
    .build(); 

let present_pass = PassBuilder::present("present", clear_pass.output(&swapchain).unwrap());

can someone explain to me why you'd need the clear_pass.output(&swapchain) to make this work?

Both passes should be reading and writing to the same virtual resource image so why is that extra bit needed?

Also I'll submit the pr for phys device selection and optimized scratch allocator rq

[night charm]

To make the order of the reads/writes unambiguous

Declaration order is ignored when building passes because that gets confusing and isn’t very flexible

[languid fern]

Oh okay I see

So I'll assume you'd also need that for buffer read/writes as well?

[night charm]

Yes, any resource with writes will need it

If you only read its fine

Ill get back to those prs later. Not really in a good place rn.

[languid fern]

Alright no worries no worries

[iron shadow]

Uhh did we ever replace it?

[night charm]

no

[iron shadow]

literally unplayable

(Take your time)

[iron shadow]

uh penguin

i remember you said there was a gui integration

but like it was broken yeah?

yeah egui

could i theoretically do the integration myself? or is this something were i need to get into phobos' inner workings to get running

[night charm]

no it should work fine

I think I updated it

But yeah you could do the integration yourself it doesnt touch any internals

https://crates.io/crates/egui-winit-phobos

[hybrid pilot]

Hello, long time no see.
So

    pub fn write_storage_image(mut self, resource: &VirtualResource, stage: PipelineStage) -> Self {
        self.inner.inputs.push(PassResource {
            usage: ResourceUsage::ShaderWrite,
            resource: resource.clone(),
            stage,
            layout: vk::ImageLayout::GENERAL,
            clear_value: None,
            load_op: None,
        });
        self.inner.outputs.push(PassResource {
            usage: ResourceUsage::ShaderWrite,
            resource: resource.upgrade(),
            stage,
            layout: vk::ImageLayout::GENERAL,
            clear_value: None,
            load_op: None,
        });
        self
    }

Adds the resource both to input and output, which makes sense

    pub fn write_transfer_resource(mut self, resource: &VirtualResource, stage: PipelineStage) -> Self {
        self.inner.inputs.push(PassResource {
            usage: ResourceUsage::TransferRead,
            resource: resource.clone(),
            stage,
            layout: vk::ImageLayout::GENERAL,
            clear_value: None,
            load_op: None,
        });
        self
    }

Only adds it to the outputs which feels strange. Is there something I'm missing?

[night charm]

hmmm

let me think

I think this is a copy paste error of me

When dealing with images you needed input and output at some point to transition the layout I think

Actually I'm still not sure

Maybe the point is writing to a storage image in a computer shader implies reading from it

But writing to a buffer with a transfer command doesn't

Man it's been so long I really need to get back to this project

[hybrid pilot]

sure but how does it only imply reading it

[night charm]

Oh wait that's a read

[hybrid pilot]

if it was only on the output sure

[hybrid pilot]

No worries worst case I'll bang my head on it once it gives me actual problems

[night charm]

Starts to smell like a bug or some arcane magic I wrote while high on sleep deprivation

[hybrid pilot]

hahaha

Anyway after a loong time I was getting back into rewriting my renderer with phobos

I was a bit worried about the faith of the project seeing how inactive it has been

[night charm]

It's not entirely dead, I haven't been working on anything at all

When I do get back to working on things this is the first thing I revive lol

[hybrid pilot]

great to hear!

[night charm]

Either way you're always free to ping me here

[hybrid pilot]

Thank you!

[languid fern]

When you do get back working on it please lmk as I also need a side project to keep working on

[hybrid pilot]

Work on it while they can't 🐸

[languid fern]

Honestly yea that doesn't sound too bad

[night charm]

[hybrid pilot]

Penguin, why are you reordering my updatebuffer cmds before the things that actually use them? 😦

ah actually it looks like I've added update commands so I might have fucked up myself

[night charm]

my brain is itching to work on this

fail

maybe soon

i kinda want to get the bindless stuff going

[languid fern]

Me but with any rust project atm (I yearn for rust maxxing)

[night charm]

honestly true

ive been writing too much java

[languid fern]

If you are gonna pick up working on phobos eventually though I'd like to improve error messages

I was tryna set up a simple compute renderer the other day and the errors were not helping ngl

Gave up and went with c# n opengl

[night charm]

That's a pretty good call

If you have some concrete scenarios of where you need more precise reporting I'll write it down somewhere

[hybrid pilot]

I have a branch with a bunch of improvements a d a working version of the bindless stuff

[night charm]

Yeah I was going to work off of that

[hybrid pilot]

Let me know when and I can update the MR

[night charm]

Will do

[hybrid pilot]

I found somewhat of a deep bug.
Suppose you have three passes, A B and C,
A produces a resource R1 used by B and C, and a resource R2 used by B
B produces R3 used by C.

Now R2 and R3 are handled correctly but let's see what happens to R1.

A barrier is created between B1 A and B, another barrier B2 is created between A and C.
This is not correct, B1 might transition the image to a certain layout so B2 can't be eliminated, B2 must happen after B and use the new_layout from B1 as old_layout

In a sense it is as if a pass should always produce an output, even if the resource is only read, because transitioning it effectively mutates it

[night charm]

Hmmm yeah that seems correct

Sorry for the late reply I completely missed this

I guess it should be added to the outputs if the layout is transitioned

[hybrid pilot]

We had a long discussion about this in #vulkan
The gist of it is that barriers should probably be handled differently

[night charm]

Seems reasonable

The graph system is probably due for a rewrite some day anyway, I'm not very happy with the code

[hybrid pilot]

I see

I was thinking of reworking the barrier thing myself so I can get going with my engine

For now I have an ugly hack to make it work

[night charm]

I see yeah

[iron shadow]

how did you remain sane while porting over every god damn ash struct to your stuff

[night charm]

you dont

[iron shadow]

Hey sorry for asking uh how did you do a scratch buffer?

Is it just a cputogpu memory location?

[languid fern]

Iirc yea something like that

I messed around with that in my pr I forgot how it went though lel

But yea just a small temp buffer that gets pushed to gpu eventually. Idk if that included push constants with it or not though

[night charm]

yis

sorry for le delay

[hybrid pilot]

btw I kinda ended up forking https://github.com/pac85/phobos-rs/tree/pfork

though like, if there is interest we can try merging most things (after cleaning up lol)

first would be bindless prob

[night charm]

oh i totally missed this, ive been stupidly busy with thesis and other deadlines but picking this up is something i wouldnt mind doing after i graduate

feel free to just work on your fork for a while but getting things merged in would be neat

i can add you as a collaborator on the main repo too if you think thats easier

[hybrid pilot]

Oh no worries I just wanted to make you aware

[hybrid pilot]

Currently my branch is not very clean (and in general I'm iterating on things) so I think letting it mature might be a good idea

[night charm]

I mean you can put it on a branch on the main repo is what I mean (if you'd like, completely up to you)

Doesn't need to be clean in a dev branch

[hybrid pilot]

Ah I see. Yeah I can do that

[night charm]

I've sent you an invite to be added to the repo, feel free to make a branch there (or repurpose develop, whatever works best)

I'm going to set up a branch protection rule against pushing straight to master for now then just because those are nice to have when working with multiple people

[hybrid pilot]

sg! I guess I'll just have a develop_pac branch perhaps

or develop_gamekernel since it is the branch used in my project?

[night charm]

Sounds good

[hybrid pilot]

great

[night charm]

you can also make branches per feature but I assume that's going to be a lot more involved for making frequent changes

[hybrid pilot]

yeah so

[night charm]

But as long as the commits are somewhat isolated we can cherrypick them to master easily enough

[hybrid pilot]

I think having a branch where I just linearly add commit, then whenever we decide we want to merge something I move them to a branch, squash and rebase the dev branch once the pr is merged

[night charm]

Yeah that sounds good

I had a quick look over the diff of your branch so far, most things look good and fairly straightforward

There was some stuff in the render graph that looks a bit messy but honestly that whole system is somewhat messy and the backend with lifetime nonsense could use some refactors some day (one day I'll get to it)

(surely)

[hybrid pilot]

So my plan there is to change it quire profoundly. Currently I'm doing the barrier handling during recording which is not ideal because recording should be lightweight. What I want is to introduce an intermediate object that contains a linear list of passes and barriers and then that can be recorded cheaply. Also my barrier code is trivial and I had plans for a better algorithm that should generate something more optimal

[hybrid pilot]

Yes there's lots of misc fixes and additions

thx for looking at it!

[night charm]

Sounds good to me

[hybrid pilot]

Cool

[iron shadow]

@night charm sorry for the ping, but could you possibly shift the git commits from my old email address of [email protected] to [email protected]?

[night charm]

Yeah, I’m out all day today but remind me tonight or tomorrow if I forget

[iron shadow]

Alright thank you so much!

[iron shadow]

Yo penguin

I was going through your fence code

uh what is the purpose of: wait_thread_spawned?

I don't see it's value being changed whatsoever in your code so I'm just lost

[iron shadow]

impl<T> std::future::Future for Fence<T> {
    type Output = Option<T>;

    fn poll(mut self: Pin<&mut Self>, ctx: &mut Context<'_>) -> Poll<Self::Output> {
        let status = unsafe { self.device.get_fence_status(self.handle).unwrap() };

        if status {
            self.call_cleanup_chain();
            self.wait_thread_spawned = false;
            return Poll::Ready(self.as_mut().value());
        } else if !self.wait_thread_spawned {
            let waker = ctx.waker().clone();
            let device = self.device.clone();
            let fence_handle = self.handle;
            std::thread::spawn(move || {
                unsafe {
                    device
                        .wait_for_fences(slice::from_ref(&fence_handle), true, u64::MAX)
                        .unwrap();
                }
                waker.wake();
            });
        }
        Poll::Pending
    }
}

Like 😭 i'm pretty sure this is spawning a whole entire thread everytime we poll it

[night charm]

Wowzer I forgot to check this

uhhhh

[night charm]

You didn’t remind me

let me look at it when I’m home

If I don’t forgor again

(I likely will, long day and I got other things to do when I get home as well)

I think @hybrid pilot wrote some of this but I’m not sure

He definitely contributed to the future impl

[hybrid pilot]

I changed it from polling to waiting iirc

[night charm]

Right yeah

[night charm]

This is true but it’s also because of how rust futures work

Once the future is polled it won’t be polled again until the waker wakes it up

And this waker only wakes up when the waiting thread calls it

So polling a future here just spawns a thread that waits on it and then signals the future when it’s done

It only happens once per future

[hybrid pilot]

So a further improvement that I never got around to work on would be to keep dome thread around per queue, then send the fence+waker through a channel to it so we don't pay the price of spawning a thread every time.

Not sure if it worth it though

[night charm]

Yeah that would make sense

Alternatively allow people to supply some kind of thread pool or detect if Tokio is present

[iron shadow]

I tried implmenting my own solution, but ended up going with the thread per waiter. It was a headache trying to figure out synchronization amongst multiple waiters. Only difference was I used a dedicated struct with a lifetime ref to the original fence

[iron shadow]

@night charm reminder bonk

[iron shadow]

also penguin

do you just use one queue in your render graph?

[night charm]

yea

[night charm]

ok let me figure out hwo i do this

ok i found a script but its bash so ill run it later on my laptop

rn i have to prepare for chess match later

[iron shadow]

good luck with the match

next magnus ong

[night charm]

not really

playing another FM today though

won against one earlier this weak so maybe i can go for a repeat

[iron shadow]

FM shares one letter with a GM

you're like 50% the way there

[night charm]

im not one im just playing vs one

[iron shadow]

okok but like you're fighting someone who is 50% a GM

also penguin should i handle my PSO cache using CreateInfo hashes like you do or just an opaque handle using some sort of free list mechanism

I kinda just don't wanna have to make my CreateInfo structs just to support hashing

[night charm]

but why not