db corruption debugging | Foundry VTT | Page 1

amber flame Dec 9, 2021, 3:21 PM

#

I've got a report of db corruption on what I thought was a relatively simple module.

I remember the ,,,,,, fiasco and that's not what I'm seeing reported here. Instead there's just... lots of duplicate actors in the compacted database.

sullen harbor Dec 9, 2021, 3:30 PM

#

If there are duplicates it probably isn't finished compacting?

amber flame Dec 9, 2021, 3:49 PM

#

Ok, on a hunch I restarted the server and gave it a while to chug, stopped it, and then took a look at the db files.

Now they show far fewer duplicates. So I guess this is 'expected' behavior false alarm?

sullen harbor Dec 9, 2021, 4:32 PM

#

I would expect so, yes.

shut igloo Dec 9, 2021, 4:39 PM

#

Duplicates are not a problem necessarily

#

NEDB writes in append-only mode (because that's very fast for IO)

#

the DB files are periodically compacted, which retains only the last record for each _id

#

Compaction is debounced - with a cap on the number of writes that can occur before it is forced. So if an actor is updated many times very quickly it will produce lots of records which are later cleaned up

amber flame Dec 9, 2021, 4:47 PM

#

Thanks, that makes me feel better. I dug into the diffs between these individual records for my own peace of mind and it doesn't look like it's things that my module would affect anyway, so I'm comfortable saying the issue I got is a false alarm.

sullen harbor Dec 9, 2021, 4:49 PM

#

This does lead into an interesting question though, which is one I'd appreciate your thoughts on, @shut igloo . At the moment the way I make a compendium for distribution is to create a module on my own install, put stuff in it, and then upload the databases for the compendiums as part of the module install. But A) I can't guarantee capturing the database at a moment of optimum compression/cleanliness and B) I keep getting changes firing in my github manager because it sees changes in the databases in my local copy of the module (presumably due to background housekeeping etc. even if I don't change anything).

Is there a better practice I should be using, or should I just not worry overmuch about these things?

shut igloo Dec 9, 2021, 4:51 PM

#

sullen harbor This does lead into an interesting question though, which is one I'd appreciate ...

Shouldn't need to worry much about it - but one thing you can do is whenever a World is shut down (i.e. you "Return to Setup") every open database file is compacted at that time

#

so if you want to be sure, you can just shut down your active world before committing

sullen harbor Dec 9, 2021, 4:52 PM

#

Ah, yeah, I had thought that was the case... just shutting down Foundry itself doesn't guarantee compacting?

shut igloo Dec 9, 2021, 4:55 PM

#

yes, that would as well, unless you close it in a "kill the process" sort of way

#db corruption debugging