bevy_trenchbroom

[halcyon frigate]

Possibly a change in how Avian handles transforms?

(or Rapier, if you're using that)

Like, maybe if the physics lib wasn't handling the transform heirarchy correctly before, bevy_trenchbroom could've compensated, and if it's fixed now then it'll be adding the transform twice

[cinder parcel]

i have no clue. I think something got mixed up and trenchbroom started generating compound colliders and passing the wrong translation into the isometry for the collider

it used to be offset_of_the_collider_worldspace - transform.translation but now it's offset_of_the_collider_worldspace? I think?

[dim warren]

FYI @digital heath ^

In case this is also Avian transform related

[cinder parcel]

im mildly certain this is on trenchbroom's side. But not completely certain

i think the parameters being passed to Collider::compound are just wrong

specifically the isometry is given the wrong translation

or if not wrong, it at least changed between the version i was using and 0.10

[halcyon frigate]

It sounds like it’s straightforward to fix at least 😅

[urban zealot]

Whoops, I forgot about your problem, sorry! Make sure to ping me since Discord's UX team has been taking a break for the past 6 years so I can't unmute specific channels

Are you using scene hooks for creating the colliders, and are you using .maps, .bsps, or both?
The isometry passed to each collider is actually always zero, instead it uses the brush's local vertices, which unless you have an origin brush will always have its origin at 0,0,0. Are you doing some non-standard transform somewhere I haven't accounted for maybe?

[cinder parcel]

wrt to first line: 😭

.maps, the convex collider scene hooks that bevy_trenchbroom comes with and go in the hooks part of the macro

and no the isometry passed to the collider is not zero. Setting it to zero produces very strange results

subtracting the transform from the isometry produces perfect results

leaving it as is and setting the transform of the collider to zero and then copying the former transform to its child meshes also produces perfect (ish) results but is hard to work with since now the pivot point for rotations is always the world origin

[urban zealot]

It isn't zero for cuboids, but for convex hulls this is the line of code

colliders.push((B::ZERO, transform.rotation.inverse(), collider));

If they aren't zero that would be very weird

I should probably go to sleep, as it is midnight for me, I really shouldn't be troubleshooting right now

[urban zealot]

Are you using origin brushes then?

[cinder parcel]

me too lmao. I will send proof that it is not zero and then do the same

i did not make the map myself so I'm unsure

maybe some of the details i sent earlier could answer that? There's the origin bits on the target entities that if commented out, fix the behavior

[urban zealot]

Oh interesting. I'm pretty sure I read and set the origin property for brush origins somewhere. Could you send me the map file? I'll look more into it tomorrow

[cinder parcel]

a compound collider isometry picked from those I have to fix up, from before any updates are made to it:

(
    Isometry {
        rotation: [
            0.0,
            0.0,
            0.0,
            1.0,
        ],
        translation: [
            -3.2512,
            0.4064,
            -29.252863,
        ],
    },
    SharedShape ( Arc<Cuboid(Cuboid { half_extents: [[3.2512, 6.5024, 0.0079375]] })> ),
)

[urban zealot]

That's a cuboid so that's expected

[cinder parcel]

and is it expected to be in world coordinates?

[urban zealot]

I don't think so

[cinder parcel]

i am fairly certain it is meant to be local to the entity's origin. But instead it is local to world origin and that causes odd behaviors

[urban zealot]

I'm guessing half the process doesn't think to account for the entity's origin, but the other half does because no origin brush but yes origin property, imma go to sleep now though

good night

[cinder parcel]

good night! I also think so

[urban zealot]

@cinder parcel Try rolling main, I think I fixed it

[cinder parcel]

I'll give it a look when i get the chance

[dim warren]

@digital heath @urban zealot I've been contemplating whether the current design of rolling up all colliders of a solid class (most notably worldspawn) into one big compound collider is efficient. Does that not grind against the BVH?

If so, would it make more sense to spawn separate children with a collider for each brush?

Or is there some important reason that BTB works like it does now?

Or does it not matter because the BVH internally splits the compounds?

[urban zealot]

I don't believe there is, having it as a compound just feels more organized to me

If it would be good for performance, I'd be fine with changing it though

[dim warren]

Am I remembering right that each solid class is its own rigid body?

[urban zealot]

Correct

[dim warren]

Neat, then changing designs should be easy enough

Curious what @digital heath has to say

[halcyon frigate]

The workspawn collider could be split by leaf at least

[digital heath]

Compound colliders internally use a BVH as well, so performance-wise I don't think there should be a huge difference, at least right now

With the future BVH broad phase (I swear I'll finish it someday ) we might do some parallelization stuff that the compound shape BVHs might not do though

[halcyon frigate]

Do you manually reorder things for better locality? That might be a reason to still manually split by leaf since those will likely be tested at the same time

[digital heath]

Not sure what you mean by reordering things 🤔

Every shape is its own leaf node in the BVH

[halcyon frigate]

I meant in memory, for cache locality, but now thinking about it the whole usp of avian is that it uses the ecs for storage so you don’t have control over that

So yeah, sounds like the current way bevy_trenchbroom does it is just fine

[dim warren]

Thx 👍

[dim warren]

Is it possible that using Layers is confusing btb @urban zealot ?

(this is a .map)

[urban zealot]

What do you mean by Layers?

[dim warren]

I just started using this feature the other day haha

[urban zealot]

I'm not sure, I don't quite remember how layers are represented in the map file

It is possible

I won't be able to look into it today though

[dim warren]

yeah no worries, it works perfectly fine

really just printing a silly warning is all 🙂

[urban zealot]

Oh, everything still appears to have collision?
That's good at least, perhaps the layering system creates brush entities without any brushes

[dim warren]

Yep 😄

any bugs you see are my own doing

[urban zealot]

Oooh, are you making a proper Quake-style character controller?

[dim warren]

exactly ⭐

[urban zealot]

Sick! That is sorely needed IMO, tnua is cool but hasn't fit my needs when I've tried it
Let me know if/when you finish it or if something interesting happens!

[dim warren]

If all goes well this will lead to upstreaming collide_and_slide to avian 🙂

[eternal pecan]

Very sweet, I know many people (myself included) have been waiting for something along the lines of that.

[dim warren]

Check #showcase 🙂

[distant pollen]

how do you guys learn how to use the plugin itself?

[pallid fox]

refdocs and picking apart examples mainly

[dim warren]

There’s a manual

[distant pollen]

could you link it?

nvm i found it

yeah its jut a little shorter than i thought it would be but fair enough

[dim warren]

@urban zealot I'm having a weird issue where something in my map causes the spawn_player solid class to not be spawned

if I remove everything from the scene except the spawn_player, they spawn

weirdly, it gets included in the Scene

just never spawned in the real World

It's on this branch here: https://github.com/janhohenheim/bevy_ahoy/tree/better-surf

in the surf example

[dim warren]

update: the fault is in the things belonging to func_group

update: if I remove all func_groups except one, it also works

so the issue is with one or more brushes

but uuuh

hard to know which, given that I have 1082 of them

kinda stumped on how to debug this one :/

[dim warren]

Yeah looks like this causes BTB to just not spawn anything

[urban zealot]

I am currently on vacation, and cannot help, sorry!
I should be home in a couple of days

[granite canopy]

im stoked on that, there are tons of premade character controllers and even when ive tried to just extract their collide and alide algorithms they make too many assumptions about how my character should behave.

a barebones fully robust move_and_slide akin to godot's would be killer

[dim warren]

Done 😄

[dim warren]

#showcase message

[dim warren]

Ah, enjoy

[dim warren]

Update: the issue is not that it doesn't spawn, it's that it takes long, very very long for it to spawn

it's on the main branch now

[dim warren]

Update: looks like func_group is for some reason really really expensive

even though it doesn't really do anything

having thousands of brushes works easily, having thousands of brushes in individual groups makes it really slow

[urban zealot]

Each brush is in its own group? That makes sense that it's slower, but I shouldn't be that much slower.
How are you grouping them individually in TB? I might try recreating it and profiling for the bottleneck

[dim warren]

it's converted from source BSP

sec, I'll shoot you the slow commit

[urban zealot]

Oh interesting

[dim warren]

https://github.com/janhohenheim/bevy_ahoy/blob/158b2e596753d74d70e4714ac6171f22c4c27f59/assets/maps/utopia.map

this map at this commit

removing the groups makes the load almost instant

also, replacing all func_group by worldspawn still lags the same

[urban zealot]

I put this in the maps folder, ran the map_loading example on it, and it loaded in under a second (I confirmed that the func_group entities were there)

It might be something that uses the meshes though, like avian or steam audio, I should try that

[urban zealot]

Loading with Avian colliders doesn't see a noticeable increase in loading times

I guess the only thing I can suggest is to either somehow profile or start removing things until hopefully you find the cause

[dim warren]

Could you check out that commit and run cargo run —example surf?

[urban zealot]

I'm getting a load time of about 8 seconds

Movement feels great though

[dim warren]

Hehe that’s good to hear

[dim warren]

Yeah I'm at quite the loss as to what the reason is

Either BTB doing something weird or worse, the parser having trouble

But given that it doesn’t take long on your test, it’s probably not the parser

[dim warren]

@urban zealot any idea how to debug this? Maybe slap a bunch of traces on it and ask tracy?

Do you have an idea which methods could be involved?

[urban zealot]

I've used tracy with Bevy only once a while back just to try it out, so I am unfortunately a noob on this subject
Personally, I would probably start by getting a local copy of BTB, and putting a little Instant::now(), at the start of the map loader, and print out the elapsed time at the end
Once you've made sure that it was indeed the loader causing the lag, narrow it down by moving the start and end to try to find the part causing the lag

[covert heron]

would it be useful for the parser to parse the comment blocks before each entity? I'm personally of the opinion that hiding semantically significant info in comments is a Bad Idea™ but since TrenchBroom does it anyways I might as well give in

largely, though, the map parser itself is intended to be agnostic of what editor is used to build the map. It seems not to be the case with BTB by its very name

[dim warren]

Related to that, the Utopia port (#showcase message) came from a decompiled Source BSP, but opening it once in TrenchBroom, saving it and closing it again filled in most comments. Though not all, it for example didn't fill in the regular func_group comments

So it would be good if missing comments didn't result in failure

but yeah I'd be also in favor of using those comments when available

Though I'm not entirely sure what we'd do with that information exactly

possibly provide default Names?

[covert heron]

the comments before worldspawn has some information, like the game profile used

[dim warren]

that could be used for some runtime validation

[covert heron]

I guess Game, Format, and entity # are the only fields stored there. Maybe these aren't terribly useful

the parser kinda just figures out the format based on the grammar

[dim warren]

@urban zealot it just occured to me that our current strategy of "1 material = 1 mesh" is horrible for culling

but amazing for targets without bindless (most notably: WebGPU)

I know we could have something really neat in this regard if we used the visibility data available in BSPs through ericw tools

But for me as a .map user this wouldn't help

would it be okay for you if I PRd an alternative meshing strategy?

I was thinking of splitting a brush by its face materials and creating a mesh for each

e.g. two brushes with the same texture would be two meshes

and two brushes that each have two textures would be four meshes

[halcyon frigate]

Visdata calculations are implemented in qbsp now, I put a PR up a couple months ago. Should be relatively easy to make that work in bevy_trenchbroom

I was going to put up a PR to implement one mesh per leaf (or viscluster in Q2) + visdata handling in bevy_trenchbroom but I’m taking a break from my quake work atm unfortunately

If you don’t care about multi-camera support then it should be really easy. If you care about multi-camera it’ll be more of a pain to do efficiently but it’s totally doable

[dim warren]

I care a lot about multicam unfortunately

[halcyon frigate]

I wonder whether combining textures of the same size into a 3D texture would help with batching, I did that in my old handrolled renderer for seismon, although seismon is intended to render unmodified Q1 maps so it’s easier to rely on texture sizes being power-of-2

[dim warren]

Textures are also POT in Bevy

Otherwise the renderer will crash haha

[halcyon frigate]

Oh, really? Then that should be really easy to implement

I had no idea

[dim warren]

It works with non POT… until it doesn’t haha

Like, the diffuse can be non POT but the normal cannot, stuff like that

[halcyon frigate]

It’s not too bad, you just use render layers. You need to store the old visleaf/viscluster that the camera was in in a component so you only do recalculation when that changes. Remove the camera’s render layer from the old PVS and add it to the new PVS. Getting the PVS for a leaf or cluster can be done with APIs in qbsp

Only Q2 maps use visclusters, everything else uses visleafs

(Well, everything else that qbsp supports)

[dim warren]

What’s the difference between visclusters and visleaves? 🙂

[halcyon frigate]

Visleaves are directly tied to the bsp tree, visclusters contain multiple leaves that would otherwise have identical visdata

It’s just a way to optimise visibility in more complex maps, so you can reduce recalculations and reduce the space taken up by visdata

For the purposes of bevy_trenchbroom you'd have either one entity per visleaf or one entity per viscluster (containing a scene with all its geometry) so you don’t really need to care about the difference in the code that sets the render layers, it’s only important when building the scene

[dim warren]

Hmm is that even applicable when I don’t use a .bsp then?

[halcyon frigate]

.map files don’t have visdata at all so no 😅

In that case you’re just relying on bevy's regular culling, so the best you can do is split up the meshes either per-brush or based on some joining heuristic like distance

Maybe there’s some 3D k-mean clustering algorithm you can do to join the brushes into larger pieces, I dunno. If I was in your position I’d just combine the textures into layers of a 3D tex and have one mesh per brush

[dim warren]

That kind of stuff would require writing my own PBR shader, since the default does not support that

Which I will have to do at some point anyways

But not today haha

[dim warren]

Yeah I was thinking one mesh per unique texture per brush

[halcyon frigate]

Hm, I think if it's split per-texture and per-brush the number of unique meshes might get out of hand, especially since bevy can't do any instancing

I think you could just have one mesh for planar 3-vertex and 4-vertex faces since you could remap them with a transform matrix, which would reduce the number of unique meshes somewhat I guess. Not sure exactly how bevy handles instancing though, like I'm not sure whether you can have an arbitrary mat4 to transform an instance or if you need to have it be expressible as translation+scale+rotation. I'm also not sure how you'd calculate that matrix off the top of my head but it's def possible

[halcyon frigate]

Like, it can't do any instancing if every mesh is unique, I mean

[halcyon frigate]

I'm kinda just spitballing here, there's definitely a better way to reduce the number of unique meshes

Is there a particular reason that you need to use a .map rather than a .bsp?

[dim warren]

Oh heck good point

[halcyon frigate]

I kinda feel like without visdata the sacrifices you'd need to make to allow frustrum culling wouldn't be worth it

[dim warren]

Got fed up with the limitations on BSP :/

specifically, the texture path length stuff and the supported texture formats

I had a script that took my .map compliant stuff and converted it into BSP compliant things, and there was always something new to add to it

[halcyon frigate]

What limitations specifically? I think BSPX stores most of the .map data (like brush lists) in case you need that for collision etc

[dim warren]

not really format limitations, but tooling limitations

[halcyon frigate]

Oh interesting! Could you elaborate on that? Do you have your own toolchain?

[dim warren]

my "toolchain" is a giant asset baking Python script lol

but yeah, what I had was

• take the .map and look at which textures it references
• find the textures on disk
• convert the textures into a format that fulfills all of the following
  • has compression
  • is supported by bevy
  • is supported by ericwtools
  • has mipmaps
• look at the .toml file for the material
• search the referenced textures, e.g. normals and roughness
• convert all of those too
• rename them and pull them into the root texture dir so that they fulfill the 15 character maximum
• rename their references in the .toml material file
• rename their references in the .map
• run ericwtools
• some select textures are referenced in the rust code, so rename those too
• compile the rust code with a feature flag that uses the .bsp code of BTB instead of the .map stuff

since I switched to .map, I was able to remove almost everything from this list

Only thing still left is generating mips and compression

though I still need to fiddle with the .toml files unfortunately, since I need to at least change the filetype ending

e.g. from .png to .ktx

of course, none of this is an issue if you have a workflow where you already obey the 15 character limit in your file structure and serve the actual unprocessed .pngs 🙂

[halcyon frigate]

Ahaha maybe check out the Q2 bsp format? It has expanded texture length limits and doesn’t embed any textures into the bsp. You can also put a "load free texture" hook in the bevy_trenchbroom config to handle the ktx stuff (plus the normal map etc too I think)

[dim warren]

Oh it doesn't embed? That would indeed be very helpful. I think the length is still something silly like 30 characters though, right?

[halcyon frigate]

31 iirc

[dim warren]

Also, what's that hook you mean?

post_scene_spawn_hook?

[halcyon frigate]

load_loose_texture field on the config struct

[dim warren]

thanks, I'll dig around later in the code to figure out what it does haha

[dim warren]

seriously, the part about not embedding anything is huge

[halcyon frigate]

The Q2 bsp format is honestly really nice, that’s why I really wanted to get it into qbsp 😅

Q3 is even nicer but it’s also basically a ground-up rewrite of the format so way harder to implement

[dim warren]

now if only it didn't have that character limit, grumble grumble

[halcyon frigate]

Plus the tooling is worse

[dim warren]

oh really?

[halcyon frigate]

Yeah, trenchbroom doesn’t have proper support for it and the various radiant derivatives are IMO a janky pain to use, trenchbroom is much nicer

[dim warren]

Oh I didn't know TB didn't do Q3

Agreed, Radiant is not very fun to work with haha

[halcyon frigate]

It does but it doesn’t support patches which are really important for Q3 mapping

[dim warren]

what are patches?

like displacement in Source?

https://ws.q3df.org/level_design/thicken_patches/

oh or like a curve?

[halcyon frigate]

Yeah they’re like 3D beziers

They’re p similar to displacements but way more powerful

No that’s unfair, they’re more powerful in some ways and less powerful in others

Anyway, they’re cool and trenchbroom doesn’t support them (or at least doesn’t support them properly)

[dim warren]

that would not block BTB support for the format, right?

[halcyon frigate]

Nope, I even already have a parser for the format that I made like 6 years ago or something but it doesn’t tesselate patches

[dim warren]

as in, Q1 and Q2 don't have patches either, so if I'm happy with mapping those I should be also happy with mapping Q3 sans patches

[halcyon frigate]

Oh right

Yeah if you integrate my q3bsp library that should work great

[dim warren]

👀👀👀👀

[halcyon frigate]

I’ll track it down, give me a sec

https://github.com/eira-fransham/bsp

[dim warren]

looks like Q3 has 64 characters, now that's something I can actually work with

[dim warren]

you said it's very different from Q1 and Q2, right?

would the BTB side of things also look very different?

[halcyon frigate]

Yeah it couldn’t easily be integrated into qbsp but it would be fairly easy to integrate into btb bc it still fundamentally works the same, the fields are just all in different places

[dim warren]

ah, that's neat!

and for compiling, does it use something else than ericw?

[halcyon frigate]

Oh yeah I forgot about that, the tools are bundled as part of netradiant or netradiant-custom (I don’t know why the latter project stuck with that terrible name but it’s a very good fork), you can find them in the respective repos

If you install one of the two using your distro's package manager then you’ll get the binaries

I don’t have much experience with regular netradiant but I can vouch for the nr-c tools

[dim warren]

cool, thanks

ngl this may very well pull me back to using BSPs

[halcyon frigate]

That’s another reason that the tooling is less good, ericw tools are fantastic and the nr-c tooling is good but a lot easier to make crash 😅

[dim warren]

is there anything I should know before using the parser?

like, some stuff that isn't supported or known bugs

[halcyon frigate]

Ja gimme a sec

Here’s a usage example https://github.com/eira-fransham/amethyst-bsp/blob/master/src/lib.rs

[eternal pecan]

JACK also supports Q3 patch editing as well (id love to get patch support in BTB)

[halcyon frigate]

Ooh I didn’t know that!

I only used jack for goldsrc ages ago

[eternal pecan]

99.9% of my mapping is done in hammer, so jack just feels right compared to TB

Yes, im crazy

[dim warren]

I should really learn Hammer at some point

[halcyon frigate]

I don’t know of any bugs and afaik everything can be loaded, I can’t remember if irradiance vols are supported and I don’t support for tesselating patches but you can at least read the control points and do tesselation yourself if you want to at some point down the line

[eternal pecan]

Look into Hammer++ if you do

[dim warren]

are you crazy enough to implement a Source BSP parser? 😛

[halcyon frigate]

It exists!

One sec

[dim warren]

[halcyon frigate]

https://codeberg.org/icewind/vbsp

[eternal pecan]

Interestingggg

[halcyon frigate]

I actually reached out to the dev asking if they want to collaborate on some qbsp/vbsp shared stuff since there will be some overlap but they’re not using bevy so I haven’t found any places to work on some shared libraries between the two yet

[dim warren]

so you're telling I could in principle create a map in Hammer++ and then patch BTB to load that? 👀 👀

[halcyon frigate]

I think that "in principle" needs to be like 400% more italic 😅 but yes

[dim warren]

Is the format that different?

[halcyon frigate]

That’s quite the damn sidequest though

It’s a whole damn dlc

[dim warren]

hahaha

[dim warren]

just looking at the structs, it looks similar?

(ignoring extra features like displacements)

[halcyon frigate]

I mean, if you only care about implementing the subset of features that are analogous to features in Q1/2 then yeah it should be ok

But I would still try Q3 first honestly

[dim warren]

for sure 🙂

[eternal pecan]

If you did get it working weew, there are some things which i have been missing in TB/Jack. Displacements, func_instance and hotspot texturing would be so so nice

[dim warren]

what are the last two?

[dim warren]

oh, about that: do you know if the class definitions that BTB generates are accepted by TB when in Q3 mode?

[eternal pecan]

func_instance you can build say, a building on grid, and then instance it in another map and rotate it. If oyu edit the instance it will update all instances, for hotspot texturing easier just to read this: https://developer.valvesoftware.com/wiki/Hotspot_texturing

[halcyon frigate]

Absolutely no idea, my use of BTB is for loading stock maps so I don’t use the generated defs. I don’t see why not though

[dim warren]

also, what's up with Quake3 (Valve) as a format?

i.e. I see this

and it makes me think "Ah, Quake3 (Valve) is a subset of Q3"

but if that's true, how come it's in BTB's format enum? https://docs.rs/bevy_trenchbroom/latest/bevy_trenchbroom/config/enum.MapFileFormat.html

given that I thought it didn't support Q3

or is this only for .map and not .bsp?

[halcyon frigate]

If I had to guess, I’d say the "(Valve)" ones have vmf support

vmt?

The valve pre-pbr material format

Anyway you probably don’t want that bc valve's material format is gonna be much worse than anything you can do with pbr

Unless you’re going for that retro phong look

[dim warren]

definitely not haha

[eternal pecan]

OH, also vertex painting, no idea how I forgot that.

[dim warren]

I'm just wondering why it's in the BTB file format enum

[dim warren]

Aaah so like Collection Instances in Blender, got it

[dim warren]

I would love vertex painting

[halcyon frigate]

That’ll require a custom shader afaik

and I think it’s only on displacements

[dim warren]

yeah definitely

[halcyon frigate]

Don't quote me on that tho

[dim warren]

I'm pretty sure StandardMaterial does not do vertex painting

(but again, I'll have to write my own one at some point anyways for performance reasons )

[eternal pecan]

Yep, only displacements, but thats mostly when you need to paint anyway

[dim warren]

I don't really get the hotspot texturing help

Like, you pre-define rectangles of well-aligned textures

and then what exactly happens next?

[halcyon frigate]

I think that Q3 is the best middleground of power vs pain-in-the-ass but hammer++ is very appealing so that would be amazing if you manage to get it working

[dim warren]

Do you texture a model and then it tries to split it into those well-behaved rects?

[dim warren]

from what you told me it seems like Q3 (sans patches) would be easy to integrate as a first step 🙂

I'll add it to my future to do list when I am deeper in level design

[eternal pecan]

So say you have this texture, you can define quadrants for all the different patches of textures. When texturing you can just hit the hotspot key and it will autoassign the best fitting quadrant to the brush you are editing. This is a bad texture example but it speeds some stuff up a lot. Also Makkon textures are SO good.

[dim warren]

I'm currently doing the last few necessary things on the KCC, then polishing it a bit, and then creating some simple maps for various gameplay features

[eternal pecan]

I agree, even Q3 support would be huge.

[dim warren]

I'll probably try to get Q3 support in during that, and try my luck at Hammer++ integration at least for a little bit when it's time to work on the real demo level

[dim warren]

Oh so it looks at the shape of the face and then selects the rect that best fits that shape?

[eternal pecan]

Exactly

[dim warren]

That's brilliant!

[eternal pecan]

For some stuff it can really speed texturing up

[dim warren]

I can see how that would easily make textures look extremely better

Update: there is also a parser for the completely new format Source 2 uses: https://github.com/NameIsJakob/datamodel-rs

Though supporting Source 2 Hammer would prooobably be an even bigger quest haha

Fuuuuuck Source 2 Hammer looks good though

It looks really good

@eternal pecan did you try it yet?

[eternal pecan]

I tried it (many) years ago in the dota2 branch, but I haven't recently. All the stuff I run/mod is on standard source unfortunately.

I do know there are a lot of amazing changes though, I should really try it again.

[urban zealot]

I would be fine with it as an option, but probably won't work on it personally

[dim warren]

yeah I meant that as something for me to PR 🙂

[urban zealot]

This is really cool and kinda tempting but to be honest, I have been working in Hammer for years, and after using TrenchBroom quite a bit doing brushwork in Hammer is painful in comparison
Also getting it to work well on Linux is a task and a half

[urban zealot]

This is really cool to see for me!
Before I choose TrenchBroom for the module that would become BTB, I really wanted to use Source 2's tooling.
The biggest reason I ended up not going this route is that I didn't want to try to reverse engineer the new map format.

It's been a little while, I've largely forgot the nitty gritty of Source 2's tooling, but I'm pretty sure we'd need to use all of it to get this to work well, and I doubt much from our Quake tooling will transfer over, we'd be starting from scratch
At that point, in my opinion it would be better to try to recreate the parts of Hammer 2 we like in Bevy's editor. Hammer is listed as one of the projects to think about when making it in its design doc

Also, even though Valve switched their tools to use Qt, it's still not cross-platform for some reason, and so jankyness on Linux ensues

[dim warren]

I'm right now in Hammer 2 👀

the map itself looks very parseable

the challenge rn is getting my own entity definitions in

[urban zealot]

Oh, I wasn't even thinking of that lol
Source 2's baked lighting is top-notch, I was just assuming we'd always want that!

[dim warren]

probably haha

baby steps first

[dim warren]

I know for a fact that Hammer 2 can load FDGs

it's just not clear where and how

[urban zealot]

I'm pretty sure there is some convoluted way the game tells the engine what FGDs to load, but I have forgotten

[dim warren]

🙂

this is what happens when you have an invalid fdg entry lol

who is this

okay, I know now how to inject our own entities

and how to read the uncompiled file, more or less

it's completely viable using Source 2 Hammer as a level editor for Bevy @eternal pecan

@tidal galleon do you have some Hammer 2 experience?

now let me see if I can make sense of the compiled map

[urban zealot]

Just remembered that the compiled maps can also bake Steam Audio information, that would be nice :)

What game are you using the tooling from?

[dim warren]

yesss there's buttons right there in the editor 😄

[dim warren]

CS2

which means it's actually running CS2 minimized while the editor is open

now trying s&box

[eternal pecan]

My god

Id love if bevy eventually had some built in tooling like what S2 has, but if we can just use the S2 tooling, it would be HUGE

[dim warren]

do you have words on the s&box tooling?

(I'm setting it up rn)

[eternal pecan]

I know they wrap around S2 (you can launch the S2 tooling from within s&box)

Ive also been meaning to play around with it since its open source now, ive just been busy with other stuff.

Let me know how it goes though, very interested.

[halcyon frigate]

The same happens in Source 1 ahaha 😅 I don’t think it existed in goldsrc though

It’s a pic of the fonz with a (former?) valve employee's face awkwardly photoshopped on top

https://tf2maps.net/threads/the-mystery-of-the-obsolete-guy-is-solved.6158/

Valve has a history of using silly images for tiny particles, they used a png of a coconut for a coffee bean particle in tf2 too

[halcyon frigate]

Yep, not to be a broken record but I still maintain that the Q3 toolchain is the best compromise for most bevy projects right now

Like, it’s not usable right this second but I would say we’re very close to it being usable and it’d get us to a pretty good point, source and source 2 would need way more work

[dim warren]

how the HECK do I start hammer in s&box

[dim warren]

I think you are right, but I cannot help myself looking into Source 2

[halcyon frigate]

I don’t think it would be out of the question to update Q3's toolchain (i.e the editor and map tools) to support PBR textures, that’d need a fair bit more work but I reckon it’s achievable

[halcyon frigate]

If you’re looking into alternate bsp formats maybe check out titanfall's format, there’s a fair bit of work been done to reverse engineer it and there’s a radiant fork that supports it

and it supports pbr

[dim warren]

thanks, but I really don't like working in Radiant 😓

[urban zealot]

I haven't used the tooling since before they changed to ECS, so I don't know these days. Just make sure you're running sbox-dev.exe
The fact that they use ECS now might either be really good or bad for us, there's a good chance they don't use FGDs at all

[halcyon frigate]

This would need some research but a possible issue with source 2 is that afaik it’s not a bsp format, so a bunch more work would have to be done on culling in bevy. All the bsp derivatives have visibility baked into the format and the visdata all works basically the same

[urban zealot]

For interop we'd probably have to export C#

[dim warren]

there is visibility data in .vvis_c

but I don't know the format 😄

[urban zealot]

I've heard mentions of a vis3 technology that they combine with occlusion culling

[dim warren]

update: their scene editor seems very... basic?

I think Hammer is the way to go for now

their Discord also says that people do their mapping in Hammer and use the scene editor to fiddle with scripts and such

[urban zealot]

Yeah they are currently in this half-Unity half-Source-2 workflow

[halcyon frigate]

If you can get source bsp working (esp if you can make it work with pbr) that would be so damn good

[dim warren]

lol yep

very noticeable

hrmmm

I can find some videos of people launching hammer, but it's from an old UI that doesn't exist now

OMG YESS

I found the holy button

(also btw, you cannot rebind Hammer keybindings in their keybindings menu. You have to change the .txt)

okay okay let me get my FDG into this

[halcyon frigate]

It would be so damn good if we can use the full S2 toolchain, although it’s been a lot of work to get the formats we already have working and they’re all thoroughly documented. I’m a bit worried about leaning too hard on black boxes. Hopefully s&box releases some format docs

[urban zealot]

Yeah, this is cool to look into but I still think the time it would take to do that would be much better spent on the Bevy editor

[halcyon frigate]

Yeah agree

[dim warren]

From my personal selfish view, I'd much rather have S2 integration than working on the level editor I'm afraid

[halcyon frigate]

I do still think Q3 is worth looking into though. Look up some modern Q3A maps if you’re interested, you can make some truly beautiful stuff with it

and it supports irradiance volumes

[dim warren]

given that the Bevy editor currently requires a whole plugin infrastructure to be designed, a pseudo-reactivity concept for UI, regular breaking updates to keep up with the BSN branch, etc.

[urban zealot]

Q3 BSPs fully support irradiance volumes? Not like directionless light grids?

[dim warren]

all of that work before you can even draw a single brush

then once that comes, I'm not sure other devs would be as enthusiastic as I am about first having a brush and prefab based workflow

[halcyon frigate]

Oh yeah you’re right I’m using that word wrong 😅

[dim warren]

something something opinionated

[halcyon frigate]

I’m fake news sorry

[urban zealot]

That's very true. Personally it has felt very hard to help out with it, all the work I can see is low-level and would take a while to get fully up to speed

S2 integration probably would be faster

[urban zealot]

It's all good

[dim warren]

So in my own opinion, connecting S2 would be much more time efficient for someone like me, who does not want to revolutionize shit and just wants a level editor

[halcyon frigate]

If we have format docs then I agree

[dim warren]

there may be parsing code one can yoink

[halcyon frigate]

That’s the ideal situation imo

[dim warren]

at least the uncompiled map should be fairly easy to get up and running

[urban zealot]

We have this

[dim warren]

from what I've seen

[dim warren]

that's for the uncompiled map

[halcyon frigate]

There could potentially be licensing issues but that’s also true with quake series stuff since it’s all gpl

[dim warren]

I have not looked at all into the compiled one

[urban zealot]

Oh

[dim warren]

maybe it's the same, maybe not

[halcyon frigate]

The source formats were all reverse-engineered long before tf2's source was officially released so we’re potentially in the clear there

[dim warren]

hrmmmmm CS2 Hammer has a "reload FDG button right here"

but s&box does not

[halcyon frigate]

Maybe s&box has its own system since it has a different approach to separating game code

[dim warren]

I don't think so, they have FDGs

[halcyon frigate]

Hm

[dim warren]

just need to inject mine

[urban zealot]

Last time I used it, I could define S&Box entities (and I assume now components) entirely through C#

[halcyon frigate]

If you can define ents in c# it should be relatively easy to write a layer that ties it together with bevy, c# has incredibly powerful ffi and reflection

[dim warren]

looks like indeed it can create FGDs from C#

heh, classic

[halcyon frigate]

At least they got x and y in the right order lmao

[dim warren]

okay okay, update

I really cannot figure out where it's expecting me to put FGDs

I know it's somewhere

I just don't know where

in my confusion about why CS2 has a button more, I decided to check out hammer.dll

and turns out CS2's hammer.dll is twice as big as s&box's 👀

so maybe they have a hammer-lite?

so I tried just replacing the hammer.dll, but that leads to sbox not starting up haha

now I'm compiling it locally

with the reasoning that this code mayyyyyyy show me how to just open hammer.dll directly 👀

is this good?

don't tell me this must lead to me downloading Visual Studio (non-code)

[urban zealot]

Why are you sure?

[urban zealot]

What are you compiling locally?

[dim warren]

s&box

how about no? wtf

[urban zealot]

Oh right for some reason I thought you were compiling Source 2

[dim warren]

dw I just leaked their source code no biggie

[urban zealot]

Oof, why can't you just use vscode?

Its C# support seems good enough if I recall correctly

[dim warren]

because

• HL:A and CS2 both use FGDs
• Hammer implicitly loads those FGDs on startup for CS2's Hammer
• sbox ships their own FGDs
• sbox allows me to open their Hammer

so unless they compiled a version of Hammer that has FGD functionality ripped out, it should work

[urban zealot]

This is a good point, there's a chance we could make Hammer our own and not rely on running through another game

That would be sick

[dim warren]

probably, it's just that they set up their project to be opened with C#. It's been a long time since I was a C# dev, but I remember that the VS Code back in the day used to sometimes not like the advanced shit VS configured

[urban zealot]

I wouldn't put it past them, I'm pretty sure they are modifying Source 2 for their needs quite a bit

[dim warren]

fair

all the more reason to find a way to directly start CS2's Hammer

[urban zealot]

Should at least try, Rider is also an option if you have access

[dim warren]

too late

haha

[urban zealot]

Do you know if S&Box's hammer uses ECS for entities? If so we should prefer it quite a bit

[urban zealot]

NOOOOO

[dim warren]

I know absolutely nothing about that

wish I knew someone to ping who coded a bit in it

[urban zealot]

It shouldn't be that hard to tell, I could have a quick look

[dim warren]

I have it compiled now

[dim warren]

yes plz!

heh

I love reading these haha

[urban zealot]

Same, I somehow always forget that facepunch is made up of a bunch of brits

[urban zealot]

How did you get into the editor? The game isn't even starting for me

[dim warren]

need to start the second thingy

[urban zealot]

Oh, I didn't know they separated them, thanks

I got the message box when running through steam, the console window (stack overflow) running through my system wine I installed the .net runtime into

The joys of Linux

And the process just freezes through Lutris, not crashing, but not producing any window or anything

Perhaps I won't check after all

Dear god Lutris can't even kill the process

[dim warren]

pfff amazing

[dim warren]

Friends, I have some excellent news

Valve's Source 2 Viewer contains functionality to decompile compiled stuff

which means I now know how to get the compiled visibility data, compiled geometry, etc. into a format we can parse 👀

I still don't know how to start a standalone Hammer, but I'm pretty sure it should be possible with a little bit of digging around ugly DLLs

but I have successfully decompiled the files exported by my CS2 map, where I know how to inject my own FGDs

here's the baked light irradiance map

(it's a map made out of like 2 brushes, that's why it looks like this)

here's an associated file called lightmap_query_data.kv3

does that mean anything to any of you?

update: the decompiler can run over the CLI and is open source, heck yeah https://github.com/ValveResourceFormat/ValveResourceFormat

[urban zealot]

Dang, this is way more promising than I expected

[dim warren]

okay, little recap at this point of my findings:

• CS2's Hammer can be used for Bevy purposes by injecting your own fgd at Steam/steamapps/common/Counter-Strike Global Offensive/game/core
• The uncompiled map is a VMAP
  • That format is a binary encoded DMX file, which can be parsed by https://github.com/NameIsJakob/datamodel-rs
• The compiled map is a VPK archive
• Such VPKs can be opened with S2V
• Inside the VPK, you will find a bunch of files with the _c suffix. That stands for compiled. I don't know what format they are in exactly, but S2V can decompile them all into a human-readable format (!)
• The produced decompiled files are a mix of file types
  • Some get decompiled into DMX (see message before)
  • Some get decompiled into KV3, which can be parsed by https://github.com/dxshie/kv3
  • light data gets converted to .exr, .png, and KV3
• All of these files can be parsed by Rust libraries

@halcyon frigate @eternal pecan @urban zealot

sorry for mass ping, but I thought y'all mind find this interesting 🙂

To append:

• sbox also ships Hammer, but their hammer.dll is about half the size and I can't figure out whether it can read custom FGDs
• replacing the sbox hammer.dll with the CS2 hammer.dll results in a crash
• The code for talking to hammer.dll is open-sourced as part of sbox, so one may be able to directly open CS2 Hammer that way. Or not, it's very possible that FacePunch have a very custom hammer.dll and that none of the methods overlap with the vanilla hammer.dll
• Using CS2 Hammer for level editing seems not that bad, with the notable quirk that it must run CS2 minimized in the background and that it contains a bunch of entity definitions that we don't care about, but maybe one could yeet them (if you're fine with not being able to create real CS2 maps until you fix your state)
• This works as a DIY solution, but it would be much user-friendlier if we didn't need to hijack CS2

[dim warren]

BREAKING NEWS: I went to the sbox discord to ask around. Indeed, they have their custom Hammer with no FGD support as Nox suspected. Further, they cannot wait to finally yeet hammer entirely, so we should not depend on sbox Hammer

[halcyon frigate]

I’ll say that it looks very little like a Quake/goldsrc/src visdata lump

So I’d guess that we won’t be able to transfer much expertise

[halcyon frigate]

Omg!

[halcyon frigate]

It’s even MIT licensed!!

[halcyon frigate]

I do ahaha

[halcyon frigate]

Oohhhhh v interesting

If I had to guess, I’d say that CS2 hammer would be our best bet for now, since presumably at some point valve will release some kind of standalone editor and it’ll be possible to adapt our work to that. It’ll still be mildly hacky but I reckon that’s better than chasing s&box who I’d wager are more likely to end up going more in the direction of in-game editing

The only real issue I had with that approach was the big unpredictable amount of work it’d take to reverse-engineer the formats. If there's MIT-licensed OSS parsing code out there already then I think that’s all we need to get started

If you’re already in touch with the s&box folks, maybe ask if their custom editor will be standalone and potentially retargetable?

[halcyon frigate]

👀👀👀

I love theft

Do you know what license s&box uses for their OSS work?

[dim warren]

MIT

update on the "custom" hammer from s&box:

Yeah its the old alyx version with some backported stuff from cs2 etc

[halcyon frigate]

Damn this is seeming really promising

I love GPL code but less-restrictive licenses mean that potentially some smaller studios could support development which seems better for the long-term ecosystem

[dim warren]

update again: no one knows for sure if the Hammer actually supports FGD

there's conflicting evidence

@halcyon frigate mind sending me the tiniest PointClass that should be valid?

need to try something

Just a name and a prop basically

[halcyon frigate]

I don’t know, I’m just a contributor 😅 Ask nox

[dim warren]

I mean you can just open any FGD from quake

that would already help

I just need something

I would generate one myself, but I'm on my Windows, where I don't have shit set up haha

[halcyon frigate]

I really don’t know much about fgds 🙈 but I’ll see if I can find something

[dim warren]

should be in ~/.Trenchbroom or in one of your funny Quake dirs 😉

oh hey I found this https://developer.valvesoftware.com/wiki/Quake.fgd

[halcyon frigate]

@PointClass size(-16 -16 -24, 16 16 32) base(Appearflags) = 
    item_flag_team1 : "CTF: Red Team Flag" []

[halcyon frigate]

Yeah that’s where I got it too ahaha

[dim warren]

thanks!

hehehe

success!

this is sbox Hammer running custom FGDs

which we have the source code for

[halcyon frigate]

Wow!!

[dim warren]

which means that in theory we could fork this, and easily turn it into a standalone Hammer

[halcyon frigate]

Dang

[dim warren]

even better: we have the interop code

[halcyon frigate]

Good work

[dim warren]

so we could even call it from Rust

[halcyon frigate]

Scratch that, fantastic work

Super exciting

[dim warren]

let me link something, sec

oh this is generated code, it's not on github

doing a gist instead

https://gist.github.com/janhohenheim/5ce1885eb809080ffc41f189cb813b9f

there you go, 4k LOC of stuff callable on the hammer.dll

Not sure what we'd need it for, but it's there

this would allow us to remotely communicate with the Hammer window

Now, Hammer itself cannot "just" run, it needs to be attached to something made with S2, in this case sbox

furthermore, the hammer.dll must come from somewhere

we already know that sbox will at some point stop shipping it, and we know that this code is only valid for exactly that version (to my knowledge. Maybe it can be slightly tweaked, idk)

Now I don't know how legal this is, BUT

we could in theory write a tiny C# program that just starts the source engine and hammer and package it with the files that sbox ships today

the thing is that distribution rights exist

and I don't know if we're allowed to distribute hammer.dll

we may be

we may be not

and if we did that, we would never ever ever get updates for it

second possibility: we simply choose one steam product that we can hijack for hammer

CS2 works and is the newest Hammer, but it comes at the cost of having to download a shit game having a bunch of CS2 predefined stuff inside the editor

But I got a hot tip: I should look into SteamVR's Hammer

which I shall do now

You may ask: can I use SteamVR without VR?

and the answer is https://www.youtube.com/watch?v=WgGrtjvEWeU

[halcyon frigate]

It doesn’t matter I don’t think, so long as it’s downloadable somewhere

It’s only needed by devs not by end users

[dim warren]

the issue is the somewhere, since sbox won't distribute it for long

[halcyon frigate]

Ah

[dim warren]

and the sbox code requires that precise version (again, you can probably tweak it to work with the newest hammer)

[halcyon frigate]

Well that’s a question for their team I guess, would be great if we could be pick up that torch

[dim warren]

they want to get rid of hammer

[halcyon frigate]

Right but maybe we could continue to maintain their version

[dim warren]

huh, I see

[halcyon frigate]

If that’s possible

[dim warren]

From looking a bit more into it, I really think that someone who knew their shit would be able to easily make it support the newest Hammer

since we can rip out every single function call except "start the hammer window plz"

Unfortunately, I do not know my shit

[halcyon frigate]

I don’t think I do either but if I get a spare moment I’ll do a bit of digging

I very rarely have spare moments though

I’m supposed to be taking it easy bc I’m sick 😅 but I still can’t help fixate on tech stuff

[dim warren]

oh no, please take care of yourself

and there we go, SteamVR works perfectly

just requires a funny command line option to run without a VR headset

but we could automate that

in which case the user experience would be "Download SteamVR, then run our executable"

if steam can start downloads from the CLI, we could even trim it to "run our executable, which will download SteamVR if needed"

for anyone trying this at home: the .fgds are usually in steam/steamapps/common/<your_game>/<some path>/game/core

depends a bit on the product

easiest is to just search for base.fgd, that will lead you to the right place

now all of that comes with the caveat of me having tested on Windows only

will now reboot into my trusty Linux and see what does and doesn't work there 🙂

for me later: this is my point class:

@PointClass = bevy : "Heck Yeah" [
    coolness(integer) : "How cool is that" : 200
]

[dim warren]

update on Linux:

• I cannot get SteamVR running such that it allows me to use the workshop.
• I can get sbox running if I change the compatibility to proton and use protontricks to install dotnet, but I think it got into a dispute with my desktop manager and disappeared I blame niri for that, will try again tomorrow
• It looks like I can get into the workshop tools of CS2 by switching the compatibility to proton

[halcyon frigate]

Interesting, I would’ve figured steamvr would be likely to work on linux considering their headset runs linux

[dim warren]

Two issues:

• it is known to not work on GNOME. I don’t use GNOME, so I was able to at least start SteamVR, but:
• I think that the workshop tools (which are the gateway to hammer) just in general not available natively. It looks like you have to start your stuff with proton for them to be available. Dunno if that’s a technical limitation or if they just didn’t bother. In any case, Steam VR does not like it when I start it with proton.

I think there may be a workaround to running the steamvr workshop tools directly in proton, bypassing steamvr itself.

[halcyon frigate]

Hm

Well we’re still missing a lot of pieces but you’ve discovered so much good stuff here

[dim warren]

I'll try the CS2 version on Linux in a few minutes

I'm very confident in that working

Also, S2V has literally everything we need for parsing it seems

[halcyon frigate]

Even if all that comes out of this is reusing the file formats that’s already great

[dim warren]

I would not be surprised if it contained information on the visibility and light query formats

Also, turns out that tool was written through reverse engineering

Even though it’s published by Valve

Lol

Hat off to them

Splendid job

[halcyon frigate]

Is it published by valve? It’s not on their github

[dim warren]

Oh hey you’re right

My bad

[halcyon frigate]

Anyway, reverse-engineered file formats are fine, license-wise

[dim warren]

Haha

[halcyon frigate]

Just so long as they weren’t made with reference to code under an incompatible license

[dim warren]

Damn I wish

I learned so much from the Source 1 leaks I read

[halcyon frigate]

Reverse-engineered file formats are kinda a grey area anyway

[dim warren]

I hope someone reverse engineers the entrypoints to the hammer dll I mentioned

Didn’t find any efforts online, but maybe some people are toiling at it in some Discord

I mean, we can’t be the only people who want a standalone Hammer 2 haha

[halcyon frigate]

Might be worth making a doc to collect info like that gist you made

[halcyon frigate]

Yeah no way 😅

[dim warren]

Yeah I have one locally

I'll yeet that into a gist later

I'll also check what the material and model definitions look like

Since we also need to get our mats and models into Hammer, and then read it in Bevy

I think is should be alright

If that works, I think I can throw something small together later today / this weekend for reading uncompiled maps in Bevy

As a POC

Honestly, it looks to me like the basic functionality of "read brushes and entities" should be fairly simple, all the tools seem to be in place

The difficult parts are supporting various miscellaneous things that S2 compiles, like the visibility data

But we can add that on a case-by-case basis

@urban zealot may I continue misusing this channel for these experiments?

update on sbox: works fine on Linux (when using protontricks to install dotnet), but the "Tools" toolbar I have on Windows is missing for some reason? So I don't see how I would start Hammer from here. There's a bunch of path errors, that's probably why. Some tinkering would probably fix this with a bit of ellbow grease, as it's open source.

Got CS2 Hammer working on Linux 🙂

[eternal pecan]

Bypassing the game to launch hammer would be big. Its weird you have to launch CS2 or whatever else just to access the tooling.

I also wonder if the steamVR hammer is a different version compared to CS2 (and all other hammer 5.x), CS2 will be the most updated.

[dim warren]

it takes fucking 4 GiB of RAM just to have CS2 open

[dim warren]

it's a bit older

I've managed to get SteamVR Hammer running on Linux btw

it opens a blank screen that thinks it's the "Steam Tours Environment Explorer" or something like that

haha

[eternal pecan]

Yeah I was playing aroud with it a few moments ago. I tried going all the way back to destinations (precursor to steamVR home) in hopes to see if there was a standalone exe, but no shot.

[dim warren]

nope

just hammer.dll

[eternal pecan]

If i have some time around lunch, ill load up Ida and poke around and see what its doing to load hammer. May be able to just bytepatch and bypass launching the game entirely

[dim warren]

it may be easier to launch the tool shelf

i.e. the thing that opens when you start a Source 2 game with -tools

[dim warren]

That would be great! I've also poked @gloomy fable for reverse engineering it a bit

In the meantime, I have understood a bit more about how Hammer 2 loads configs

I think I may be able to completely remove all the customization they did for CS2

Even better: I believe we could do some own customization, like making important components be their own dedicated button

but that's for the future

I'm a bit hindered at quickly iterating my stuff because if I crash something, it means that I need to restart CS2, which takes a bit of time on each attempt

[eternal pecan]

Very sweet (other than the crashing)

[dim warren]

Oh, another annoying thing about CS2: starting it checks if steam is running. But since I'm running it over wine, I need to be running a Windows Steam installation for that

That's something that SteamVR does not need

it literally has Steam in its name

but it does not require steam

In case y'all wanna try, here's how to run the toolshelf. If you're on Windows, just remove the wine call and change the ~/.local/share to C:/Program Files (x86):

• SteamVR: wine ~/.local/share/Steam/steamapps/common/SteamVR/tools/steamvr_environments/game/bin/win64/steamtours.exe -tools
• CS2: wine "~/.local/share/Steam/steamapps/common/Counter-Strike Global Offensive/game/bin/win64/cs2.exe" -tools -steam

Oh btw @eternal pecan there's source code for how sbox opens Hammer

Unfortunately I can't get sbox to run when replacing the hammer.dll

so idk what that's about

but it may be useful

[eternal pecan]

Thats unsurprising, they are probably completely different hammer versions

[dim warren]

here's their generated interop: https://gist.github.com/janhohenheim/5ce1885eb809080ffc41f189cb813b9f

[eternal pecan]

Oh sweet, ill take a look

[dim warren]

yes, they use an old version with some CS2 backports they did manually

[dim warren]

And this here is the code: https://github.com/search?q=repo%3AFacepunch%2Fsbox-public+hammer&type=code

[eternal pecan]

You need to patch the steamAPI calls. Use Goldbergemu https://mr_goldberg.gitlab.io/goldberg_emulator/

[gloomy fable]

ok I'll take a look and see where it checks that the cs2 process is running

oh!

[dim warren]

WOW thanks! Let me try

[eternal pecan]

Oh

Actually be very careful

That may get you banned, move the CS folder out of steam

Launch with -insecure as well to be sure

[dim warren]

lol thx

[eternal pecan]

Will (should) work for SteamVR to remove the check as well so you can run it from any directory

[dim warren]

has no check 🙂

[eternal pecan]

Ah ok

[dim warren]

(the way I posted above at least doesn't)

[eternal pecan]

To be safe may be better to use an alt steam account with that emu, I have no idea what kinda checks CS2 has now

In TF2 I can just run with -insecure and it just doesnt load VAC so I can play around with memory

[dim warren]

doesn't look like it likes that

[dim warren]

TBH I don't mind getting banned in CS2 haha

[eternal pecan]

Did you rename the original steamapi(64).dll to steamapi(64)_o.dll

[dim warren]

I simply replaced them

whoops

[eternal pecan]

Yeah that wont work lol

[dim warren]

maybe I should actually read the readme

hmm?

doesn't look like I need to rename?

[eternal pecan]

Welp, guess its changed since the last time i used it

Lol

It could also be that CS2 needs more than a patches steamapi now

Ah, the emu hasnt bee updated in a while, there is a fork thats updated. May work: https://github.com/Detanup01/gbe_fork

[dim warren]

aight lemme try 🙂

[eternal pecan]

Go into the tools folder and drag the CS2 steamapi.dll onto it to generate the interface file as well

[dim warren]

IT WORKS!

[eternal pecan]

Sweet

[dim warren]

my call is now just wine ~/.local/opt/Hammer/game/bin/win64/cs2.exe -insecure -tools

[eternal pecan]

Much easier

[dim warren]

note that I was able to move it out of steam and change the name

[eternal pecan]

Yep, no reliance on steam now

[dim warren]

fuck yeah

[eternal pecan]

Now we just need to bypass the game or make a loader and we golden

[dim warren]

Oh update: if I have -insecure the toolshelf instantly closes

so no -insecure for me

[eternal pecan]

Should be fine, if it doesnt say you are playing the game in steam, then VAC isnt loaded.

[dim warren]

Considering that my steam is not running, that should be good 🙂

Update: cannot compile on Linux, heck

there's probably some workaround or another

But I'll reboot to Windows to continue experimenting

Hell, even if it only works on Windows I'm still happy

sure thing, Windows

[dim warren]

you marvelous bastard

is that what I think it is?

[eternal pecan]

This is with steamVR since its so light compared to CS2. When you patch the steamapi dll, it just never ever loads the VR interface at all. Then run with -tools -nowindow

[dim warren]

hehe, neat

wait

-nowindow 👀

ah heck doesn't do anything for cs2

btw using the steamapi.dll from the experimental dir works perfectly when Steam is open too:

[eternal pecan]

Yeah, steam interface watches for calls to the steamAPI, so if there are no calls, you can run as many "cracked" versions as you want.

-nowindow is still a hack, it would be better to be able to just start the asset browser tooling by itself, but its cool it works so fast this way

[dim warren]

wait I'm also getting this on windows now

[eternal pecan]

Is that when compiling, or just loading?

[dim warren]

compiling

worked yesterday

[eternal pecan]

-lightmapcpu

[dim warren]

👀

[dim warren]

on startup?

[eternal pecan]

In compile options

I know earlier versions of hammer 5 required a raytracing compatible GPU

[dim warren]

weirdly enough I have a raytracing GPU

and again, it worked before

[eternal pecan]

Oh well thats evern weirder then

[dim warren]

oh hey

the compilation window shows its command

neat

[dim warren]

sorry to say that for me, Hammer is not usable on Linux :/ Running into the same viewport bugs as this person and some others: https://github.com/ValveSoftware/Proton/issues/2704#issuecomment-2185924544

same error here: https://github.com/doitsujin/dxvk/issues/2937

Maybe one of you Linux wizards out there knows what's up

back to Windows for the nth time today haha

[halcyon frigate]

Oof I’m pretty dang experienced with linux but this is some deep voodoo, I’m not sure I’d know where to start

[dim warren]

Update: successfully purged all CS2 stuff out of hammer

now testing how much I can trim the boilerplate files before it crashes

[halcyon frigate]

Yo what

Well done

[eternal pecan]

Did you strip out all the vpk loading and unneeded folders, or do you still need the full base install?

[dim warren]

full base unfortunately

I'm right now reverse-engineering how gameinfo.gi works

ideally we should be able to write our own

the one used by SteamVR is very tiny, and there's some info here: https://developer.valvesoftware.com/wiki/Gameinfo.gi

@eternal pecan maybe you know about this: it seems like Hammer is very confused about which directory to use

try creating and saving a map and you'll see what I mean

it's okay, you can save it wherever, but the F9 build expects it to be in a certain place

but that place is read-only

In regular usage, you would create a new "addon" and then save your stuff under the directory that creates

[eternal pecan]

I see what you mean

[dim warren]

yep, using the UI to create an addon and then selecting it with -addon foo when starting the tools works

(in SteamVR)

i'll find out what that does exactly

[eternal pecan]

Make a new folder in SteamVR called content/steamtours_addons, make a folder in that folder called whatever you want (match it to -addon name) and save the map into that folder and you can build

[dim warren]

I was just about to post 😄

I did it by creating a dir SteamVR\tools\steamvr_environments\content\steamtours_addons\foo and a dir C:\Users\hhh\SteamVR\tools\steamvr_environments\game\steamtours_addons\foo

[dim warren]

what do you mean by "match it to -addon name"?

aah nvm

got you

[dim warren]

so that would mean I don't need the second dir

okay, the "optional" directory gets rid of this warning

and makes "CTRL-S" go to the correct dir in content

let's keep that in mind

now I'll check what CS2 has to say about that

indeed, same concept 🙂

[dim warren]

@eternal pecan I think I figured out how to configure Hammer to not require any CS stuff on disk

BUT

The CS2 window itself still needs it 😅

[eternal pecan]

The hard part lol

[dim warren]

@eternal pecan @halcyon frigate do either of you know your way around .mdl files?

internet says I should export a model with the Blender Source addon to SMD and then use a thing called Crowbar to convert it to MDL

does that sound right?

[eternal pecan]

I haven't compiled a model in years, but back then you would export from blender and compile the model with studiomdl. Crowbar is a wrapper (and will decompile models as well).

[dim warren]

thx

trying to get custom models and materials in now 🙂

[eternal pecan]

Sweet

[dim warren]

apparently I need a QC file

that's just a tiny markup for how to collect models and materials, right?

[eternal pecan]

Yep

[dim warren]

I present: our first pointclass that has a model

@eternal pecan all of the material and model tools you need are part of the Source 2 tools actually 🙂

No need to fiddle around with Blender, Crowbar, anything

It's a bit tricky to figure out, but basically you're good as soon as you have an FBX and loose textures

I did this with the GUI, idk if this works over a CLI, but that's fine by me tbh

Okay!!! I got a compiled map with my stuff in it

Now next step is to read this stuff in Bevy

and do a writeup about how all of this works lol

[dim warren]

here's the decompiled output

(mainly for myself when I boot Linux)

@ebon burrow these experiments have one point of relevance for you btw

Hammer 2 has builtin Steam Audio stuff 👀

[eternal pecan]

All the compile tools were also included source 1 as well. But are you saying you can just load up an fbx (or whatever) directly, or is there still a conversion step into mdl?

[dim warren]

IF this works, we may be able to e.g. preview audio navigation point samples in-editor and use those exact same baked things in bevy

[dim warren]

there's still a conversion step

but it's like 3 buttons

[eternal pecan]

From the little bit of reading it looks like justa bout all the tools other than vrad are now combined into resourcecompiler

Thats cool

[dim warren]

and that can be run as a CLI? 👀

[eternal pecan]

Yep

[dim warren]

heck yeah

that would mean we could automatically create all these things from plain old glTFs

[ebon burrow]

what do they got in there? like baking tools, or does it go further than that?

[dim warren]

I currently just know about the baking

there's definitely a ton more, but I don't know if those things are reusable for us

what's really cool about that is that this audio baking is just part of the regular map compilation

[eternal pecan]

That is super cool

Liek for HRTF stuff?

[dim warren]

nope, this here is for the 3D navmesh that steam audio uses to know how a sound travels around

[ebon burrow]

in other words, more fancier

[dim warren]

it gives you navmesh generation, lightmap baking, audio baking, mesh optimization, visibility data, idk what else, all at the press of a button

dunno how much we can reverse engineer and bend to fit bevy

but the possibilities, man

(@ebon burrow for context, I already know how to read all of this into Bevy, at least in principle)

[ebon burrow]

hm, so would the idea be to use just hammer for everything, or only for like audio baking or other one-off steps?

[dim warren]

just hammer for basically everything except code

[eternal pecan]

The way it should be

[dim warren]

well, still gonna author my glTFs in Blender obviously

[ebon burrow]

in other words, hammer over trenchbroom?

[dim warren]

exactly

[eternal pecan]

Yep

[ebon burrow]

how could you say this in #1359582967613100282

[eternal pecan]

Shhh

[dim warren]

We are all big Hammer 2 fans in this channel 😛

[ebon burrow]

hm, well it seems like that would make baking forbevy_steam_audio waay easier to use

with any luck the rest of its steam audio features might be easy enough too

[dim warren]

it only comes at the low cost of having to run CS2 with 4 GiB RAM usage in the background while using the level editor

we'll figure out how to avoid that haha

[eternal pecan]

Or 1.8GiB with steamVR

[dim warren]

absolutely

been focusing on CS2 to make sure what I'm doing works with the newest Hammer

but so far I don't think there's much difference

[eternal pecan]

Yep that makes total sense, its much much newer

[dim warren]

I didn't have much luck previewing light in SteamVR Hammer

[eternal pecan]

I tried Deadlock this morning, but there is no tooling for it yet

[dim warren]

idk if that's a skill issue, something not implemented yet, or something one can just copy-paste into the FGD

[dim warren]

does one still need an invite?

[eternal pecan]

One does, I think I have a few if you want one

[dim warren]

yes plz 🙂

would primarily love seeing how their KCC works

DMd my steam

[eternal pecan]

Just need to reinstall (again)

[ebon burrow]

okay this is actually pretty funny xD

you might be the first person to try it for that reason

[dim warren]

hehhehe

[dim warren]

you mean rereinstall

[eternal pecan]

Exactly

[dim warren]

not to be confused with rerecast

@ebon burrow

[eternal pecan]

Says you already have access to it, search your steam, can probably just install it.

[dim warren]

indeed I do

since when lol

[eternal pecan]

"yes"

[dim warren]

haha

[dim warren]

@ebon burrow looks like you can also author reverb zones in Hammer

movable reverb zones 👀

[ebon burrow]

oh i wonder how that's implemented

[small dove]

@dim warren https://github.com/kristoff3r/bevy_vpk

[dim warren]

Awesome, thank you so so much!

I looked through the code and this would have taken me ages to write due to all the async stuff I'm not familiar with

[dim warren]

I assume you call AssetServer::load on the _dir.vpk?

[small dove]

You need to do something like this

const VPK_PATHS: &[&str] = &[
    "/home/kris/.steam/steam/steamapps/common/Counter-Strike Source/cstrike/cstrike_pak_dir.vpk",
    "/home/kris/.steam/steam/steamapps/common/Counter-Strike Source/hl2/hl2_textures_dir.vpk",
    "/home/kris/.steam/steam/steamapps/common/Counter-Strike Source/hl2/hl2_misc_dir.vpk",
];

commands.trigger(LoadVpks {
  paths: VPK_PATHS.iter().map(|p| p.into()).collect(),
});

Then after LoadVPKDone you can do

  asset_server.load("vpk://<path_inside_vpk>")

[dim warren]

Ooh okay I see

Thx!

I'll need to adjust it slightly to deal with S2's new formats inside the VPK, but that should be easy

It’s all KV3 and DMX

[small dove]

Yeah currently the rust eco system for valve packages is focused on TF2 content, and I have slightly modified it to work with CS:S, and my plan was to maybe go up to CS:GO at some point. But the new Source2 stuff is gonna take some work

Also when I said the loader is kinda crappy, what I meant is that it opens a new file for each inner asset, including those that are referenced from other assets, so you really want to ensure that you do caching so you don't more than you need

[dim warren]

@small dove does this support hot reloading?

[small dove]

Not currently, but coarse grained hoy reloading might be possible to add, i.e. reload everything if any of the VPKs are modified

Doing fine grained sounds really hard

[small dove]

I added extremely speculative file watcher support, but I have no way of testing if it works. If you get it hooked up then let me know if it does anything 😄

Ah, it probably won't actually, I don't even think there's a suitable AssetSourceEvent that would work

[halcyon frigate]

Only quake and hl1

[halcyon frigate]

Yeah that was mostly similar to my process like 15 years ago 😅 Haven’t got a custom model into source since then

[dim warren]

Ah heck. Well, considering @feral prairie will be on it soon, we can probably use this suboptimal strategy for now and then switch it out once we have something better upstream 🙂

Andriy, mind pinging me of you find a good design?

[feral prairie]

https://github.com/bevyengine/bevy/pull/22104

🙂

[dim warren]

That was mich quicker than I thought 👀

[dim warren]

What if I were to… you know… add it to the milestone?

[feral prairie]

It was already mostly cooked, I just needed to put some finishing touches on it haha

[feral prairie]

That would be spicy 🔥

[dim warren]

@small dove do you think you could review this? I don’t feel qualified

[feral prairie]

@dim warren I think you'll also enjoy https://github.com/bevyengine/bevy/pull/21890 🙂

[dim warren]

Ha, neat!

[dim warren]

Would this allow hot reloading stuff inside the archive?

Not sure how that would look like

But kinda important for level iteration

@feral prairie if not specific files inside the archive, could this at least hot reload the entire archive?

So that all assets that come from that source are reloaded

[feral prairie]

There's no reason you can't do that today, it's just a pain. It also depends on your file format having a "cheap" way to detect changes

[feral prairie]

That should be doable too

The biggest issue I haven't really figured out is how to "nest" asset sources - it would be cool if your vpk asset source could be based on another asset source, and then hot reload events of the other asset source, could let you reindex the vpk file, and produce your own hot reload events

Today you're limited to directly messing with the file, which kinda sucks and won't work when you recompile to web

The main problem here is that today asset source events are a channel, but that means reading from the channel will make the asset system miss events

[dim warren]

Oh I haven’t considered web

Would this just download the entire archive each time you want a specific file?

[feral prairie]

I think so. Tbh I'm not entirely sure how the Wasm stuff works

I wonder if http requests can ask for parts of files 🤔

I suspect no, and you'll have to download the entire file and cache it somehow

Neat looks like http does support it in theory! https://stackoverflow.com/questions/17643851/downloading-a-portion-of-a-file-using-http-requests

But we'd need to actually take advantage of that lol

[dim warren]

here's the compiled one too

[dim warren]

@eternal pecan I spent some hours reverse engineering. I believe the spicy function is Source2Main in engine2.dll

I figured out some of its parameters

But I still can’t run it manually

[eternal pecan]

Damn perfect timing ping, I just finished reading all the latest messages

[dim warren]

Hehe

[dim warren]

I'll send you my best guess in a sec

[eternal pecan]

Looks like we need an input dir, ill play around

Was that entrypoint in the exe or was another dll loaded first?

[dim warren]

The exe calls it directly from the DLL, if that’s what you mean

[dim warren]

@eternal pecan very basic guesses:

• param1: HMODULE, called with (HMODULE)&IMAGE_DOS_HEADER_140000000, which is "MZ"
• param2: called with 0, idk
• param3: command line arguments (I think passing "-tools" here made it load an extra dll, so I'm fairly sure about this one), called with (undefined1 (*) [16])_get_narrow_winmain_command_line();
• param4: some window mode flag, called with __scrt_get_show_window_mode();
• param5: something like a path? It does string comparisons with "\"
• param6: hardcoded to "csgo"

well, maybe not entirely basic guesses

but please take this with a grain of salt, this is not my area of expertise

or in rust terms:

type Source2MainFn = unsafe extern "C" fn(
    image_base: *const (), // Base address of exe (HMODULE)
    reserved: *const (),   // Always null
    cmdline: *const i8,    // Command line string
    window_mode: u32,      // Window show mode (SW_HIDE = 0, SW_SHOWNORMAL = 1)
    dir: *const i8,        // Maybe directory containing the exe?
    app_name: *const i8,   // App identifier ("csgo")
) -> u64;

[small dove]

What are you trying to figure out by reversing? It's gonna be a long road if you start at main, they have a lot of dynamic loading of subsystems that you can't easily figure out statically

[dim warren]

I was kinda hoping to just call Source2Main and let it start up the tools for me

that may be foolish haha

[small dove]

Hmm, how are they normally launched? Are they like subwindows spawned from hammer?

[dim warren]

the other way around

[eternal pecan]

Game exe > subwindow

[small dove]

So this is just about launching hammer without going through some other program?

[dim warren]

I believe it's Game exe -> "tool shelf" (my name for it) -> hammer (and co)

[dim warren]

yeah, or more precisely launching the tool shelf

or workshop tools

or Asset Browser

same same

[small dove]

It's been so long since I opened hammer that I don't even know how you do anymore 😄

[dim warren]

it's weird in Source 2

do you want to see how?

on CS2 you enable the workshop tools

then launch them here

which runs cs2.exe -tools

Then this thingy here called the "Asset Browser" opens up

that is the window I keep calling "Tool Shelf"

from there, you click the hamer icon

and there it is

[dim warren]

What I would love would be a way to start this exact window without going through CS2

because as part of this process, CS2 is also open in the background, eating 4.5 GiB RAM

also running a livestream for some reason???

@eternal pecan I decided to look into Deadlock modding, and I MAY have made a big discovery 👀

https://deadlockmodding.pages.dev/modding-tools/csdk-12

[small dove]

It's the CS2 major, pretty exciting games yesterday

[dim warren]

aah that explains it haha

[eternal pecan]

Looks liek those tools are just pretty mcuh the same as CS2, work would still need to be done to load stuff

[dim warren]

looks like it

but 5 GiB is already much better than 50 haha

[eternal pecan]

True, true

[small dove]

Ok, that makes sense and might be possible. But since Valve has made you launch it in this obnoxious way there might be a stupid technical reason why it's not possible

[dim warren]

but SteamVR does it 👀

when the emulator is on*

well, there is still a steamtours.exe task running

but it at least has no window and no significant RAM usage

[eternal pecan]

Steamtours (or cs2) spawns a QT window

[dim warren]

oh hey someone in the Deadlock modding community posted their fork of Blender Source Tools updated for Source 2, neat!

[dim warren]

I see

so we may need to also spawn a minimal QT window, and launch the tool shelf as a child window of that?

[dim warren]

@eternal pecan the deadlock tools look like they already put some effort into minimizing the cost of running the tools

this is the "game" window

this here is the resource usage

[eternal pecan]

I get more memory usage, over 3 gigs

[dim warren]

@eternal pecan good news: I have an executable that calls Source2Main. It works.

bad news: it still starts the cs2 game lol

this here was a good reference: https://github.com/mikkokko/cs2_patch

[dim warren]

update: i can load arbitrary dirs in game/

now I just need to figure out what it's loading from there

[eternal pecan]

Well thats cool, ive just been slowly stepping through memory

Looks like toolframework2 is whats loading all the tooling (weird, right?), or its just calling funcs from it.

[dim warren]

oh yeah I saw that one too

[eternal pecan]

Theres no main entrypoint for it, well to just "load" everything

[dim warren]

I was wondering if that was just their tooling API, i.e. "implement this interface in your DLL and you can be called"

or if it's the actual window

the tool window MAY be in assetbrowser.dll

where there's CreateInterface

[eternal pecan]

Yeah I saw that as well

[dim warren]

but idk what to do with that interface once you have it

there's some code using that API in sbox, but I don't understand it

it's some heavily indirected autogenerated FFI there

[eternal pecan]

Oh fun

[dim warren]

(i.e. the sbox code is indirect)

here's my code

compile this, and place it next to cs2.exe

[eternal pecan]

Either way it looks like some parts of the engine will have to be loaded, as the tooling uses then for model previews and stuff like that

[dim warren]

then call it with foo.exe -tools -game csgo

the csgo part will specifically point into the game/csgo dir

e.g. if you copy-paste it and rename it bevy, you can run foo.exe -tools -game bevy

[eternal pecan]

Oh thats awesome

[dim warren]

inside of the csgo dir it will still open up cs2, I suspect game/csgo/bin/win64/client.dll is to blame