Whadduya know, I just hit this again but

[astral light]

seems like this could be something in userspace, maybe? https://github.com/etcd-io/bbolt/issues/72

wouldn't rule it out at least

@lusty otter idk if you have time or anything, but if you are looking for something, could you look into this? bbolt itself is in maintenance mode so if it's a problem in it there's very little we can do other than start a project to get off this (which is hard because it implies customizing or patching containerd potentially).

If it's something in userspace it's probably containerd's fault? Though hard to say for sure, it's the most direct user of this.

This seems like the type of thing that's worth searching around for (and having an LLM do research into too)

[lusty otter]

Yeah with pleasure 🙏

[astral light]

Took a look while codex is spinning on writing tests, found this: https://github.com/containerd/containerd/issues/5175 and https://github.com/containerd/containerd/pull/6761

Which lead me to https://github.com/containerd/containerd/blob/81ac9a8b94cbfcaca01667241f975147e4edef05/plugins/metadata/plugin.go#L137-L157

I wonder if we should just set NoFreelistSync and NoGrowSync everywhere in each boltdb, in addition to where we are already setting NoSync? It's a couple places around the codebase

[lusty otter]

just saw that it's impacitng Kyle atm, prioritizing now for tonight

[astral light]

idk if it is, possibly

it just explains random weird stuff he's hit

[astral light]

it's not an emergency, I think it's just a matter of updating a couple of structs in a few places and then re-running CI a bunch to see if it's gone

[astral light]

Want to start trying it out in CI cause I keep seeing users report strange crashes and other weird things, sent it out here: https://github.com/dagger/dagger/pull/11850