V11 Prototype 1 - Help Us, Help You!

[hazy harbor]

Hello <@&596076164104323104> !

In anticipation of the forthcoming Version 11 Prototype 1 build, we want to take a moment and share with you some of the awesome features this will contain. Most of all, we want to share a request for early testing. We anticipate releasing V11p1 within the next ten days, possibly as early as tomorrow barring any surprises found in testing.

@tame barn is committed to doing a development livestream (announcement forthcoming) in the next few days to show off some of these features and take questions.

New Features
There's a lot contained in this prototype, minor tweaks here and there and in many places small changes that could have profound benefits. We'll hold off on sharing the full update notes yet as they're still being drafted, but here's a sneak preview of our patch notes with some highlights:

https://foundryvtt.com/releases/11.292

note that these are preliminary and the patch notes will be updated as we work through writing them

An Emphasis on Stability
One of our specific goals with V11 is to keep the impact for module and system devs to a minimum. We should all be excited about new features, not fearful about the amount of work updating our packages might take. Ideally, we would like to release a V11 build where community developers can simply update their listed compatibility and have their package work the same as in V10. We want to make V11 a smooth transition for everyone, but we can't do it alone.

That's where you come in.

There is often a give-and-take during development cycles: we change our API based on feedback, causing breaking changes, and then alter our approach later in the cycle only to realise that we are causing further breaking changes for those who industriously updated their modules and systems during the first few builds. We think you'll agree this has been a long-term point of frustration for community developers, and we want to try something different:

When V11p1 releases I'd like to ask you to test your packages---you don't have to update them, you don't have to make changes---just test them and report on what errors you experience. By processing these reports, we will be able to establish trends and judge the impact of changes we have made in a more data-focused way. With that information in hand, we will be able to adapt quickly, reducing the severity or reach of breaking changes by correcting for them in our API. The goal here is to cut down on cases where community developers feel obligated to update-and-reupdate their packages in the same cycle of development, and reduce concern about the impact of breaking changes overall.

To that end I have created a #1065354386508873798, which you can use for reporting on focused testing. You have permission to create posts in there, to talk about your specific packages and how they are impacted. Any specific error messages or identification of exactly which change (where possible) caused a negative impact for your module will be most beneficial.

[plush gulch]

One of our specific goals with V11 is to keep the impact for module and system devs to a minimum.

Thank you. I'm still reeling from V10 😉

[thick seal]

Not just you.

[mild fiber]

link clicked, I am safe now 😄

[thick seal]

But then I've spent more time looking at the typedefs project and that's it's own brand of insanity.

[mild fiber]

NeDB is Dead, Long Live LevelDB
< O v O >

[plush gulch]

But hey, it did kick me into gear to rewrite shit that was bad. So there's that.

The classic "while I was in there"

[crimson void]

We haven't even worked out sane typeascript types for v10 yet! So I'm glad if v11 is trying to be low-impact.

[plush gulch]

But this sounds solid. Will report when I know more. ❤️

[mild fiber]

Windows
No, not the operating system. Version 11 [...]
🤔 windows, not the OS 11
that is a fun coincidence there 😂

[hazy harbor]

Made me laugh 🙂

[mild fiber]

cool feature though, maybe, will have to see it live 😄

[hazy harbor]

I can't emphasise the "early feedback" part of the equation---The sooner we can get details about the impact of our changes, the sooner we can take steps to reduce the impact without devs having to adjust their packages.

We can't promise that the version update will require NO code changes for community developers, but we can try to make it require as few as possible, for as few people as possible.

[tawdry karma]

What’s the correct place to ask question regarding v11 changes, that are not directly related to testing a package? Just #dev-support, or #1065354386508873798 , or something else?

[tame cedar]

A new type of actor with its own data? Modules can now define subtypes of documents.
I really really hope that there is some proper seperation between system and module actor/item classes, because I can see that wreak a lot of havoc. Not saying it's a bad feature, quite the opposite, but I feel it's something that needs a LOT of care

[plush gulch]

I'm still halfway through updating to V10 so... we'll see how soon I can give you any actually useful V11 feedback. But will try.

[quartz totem]

o/

[frank orbit]

"Framework laid for the future of a manual fog-of-war implementation"

Later in v11 scale future, or future major version scale future? (both exciting, just wondering)

[hazy harbor]

TBD.

[twilit elbow]

Gotta love the death of NeDB. Tho I wonder how hard it is to rewrite tooling for building compendiums from JSONs with the new stuff.

[tame cedar]

it happens too often that users can't tell where a window or button comes from (is it a system thing, is it a module thing?) so having document types is gonna be even more fun :D

(but I will look forward to making spaceship actors)

[inner hollow]

Probably depends on how you were doing it in the first place

[brazen socket]

Out of interest do you have any benchmarks for NeDB and LevelDB that you used?

[twilit elbow]

Using NeDB nodejs module. 🤷

[hazy harbor]

This is me speaking, so take the complete lack of coding knowledge for what it is-- but because we are auto-migrating DB files from NeDB format to LevelDB, the tooling would probably not have to change so long as you can just execute the migration.

[hazy harbor]

No benchmarks to share at this time, but atropos has performed some benchmarks.

[tame cedar]

Some people were using other means to generate pack files from source data for modules and systems. I wonder how those will be impacted

[twilit elbow]

Problem with that is that would always bundle the old format compendiums with the systems/modules and require migration on user side.

[tame cedar]

speaking of, is it okay if I share this with my folks at sigil?

[tawdry karma]

That would have been my question. There are a bunch of systems that maintain compendium packs as json, and convert it to NeDB files during the build process (dnd5e, pf2e, my own system, …).

I would assume that this has been considered, since it affects several of the biggest systems, but it would be nice to have some discussion about this.

[hazy harbor]

Some should be in here to see it anyway, but if not, yes, please do

[mild fiber]

given nondestructive migration I guess not too hard
but the tooling will need to be redone in due time of course.

[vestal moat]

Subtypical does this include defining completely new template types?

Err

MeasuredTemplates, specifically (yes, I saw afterwards how profoundly that could have been confused)

[tame cedar]

swade too, I keep every item as a seperate yaml file that get concatinated xD

[hazy harbor]

and @tawdry karma --

Unfortunately I don't think we can account for "I developed a completely separate tool to build compendium databases". There's going to be too many of those and too widely varied for us to anticipate how this change will affect them, and we can't very well guess what might have been done for separate tooling outside our API.

[tame cedar]

that's fair enough

[sterile plume]

I hope LevelDB will fix the perf issues at least

working on a campaign with 40 scenes isn't fun for walling

[tame cedar]

that seems to be the main advantage from what I've read

[mild fiber]

can you share this with us Nath:
Did the Benchmarks say "It got better"? 😄

[sterile plume]

I still think TypeScript should get first party support, it would lower the impact of API changes greatly

[mild fiber]

like not any numbers or stuff, just that performance is going in that direction^^

[hazy harbor]

I am not certain if it will extend to MeasuredTemplates. Perhaps @crude thicket may be able to answer this specifically.

[twilit elbow]

Problem with the foundry provided API is that it can't be used with build systems since it requires running things inside Foundry, and Foundry doesn't provide API for doing things outside of Foundry instance.

[hazy harbor]

I'm not sure i'd agree that's a problem.

[tame cedar]

speaking of LevelDB, is it this LevelDB? https://github.com/google/leveldb

[sage lily]

little steps my boy

[twilit elbow]

It works for small projects, but for anything else it becomes harder to manage and update.

[tawdry karma]

Sure, but all of the existing tools work very similar, at a basic level. Have the data in some format, and convert it to newline separated JSON, which is how NeDB files work.

If a workflow like that is not supported anymore, well, that’s your choice, but that has significant impact on several (big) systems

[sage lily]

yes but Nedb will be replaced with ClassicLevel a abstract layer of that project

[hazy harbor]

"not supported anymore" suggests it was ever supported to begin with.

[twilit elbow]

I feel like Foundry needs to provide external API/tools for dealing with this at this rate.

[tawdry karma]

Well, it was. Maybe not officially, but considering that even dnd5e does it, I think it should be considered „supported“.

[tame cedar]

so far the goto answer was "just use nedb" and I guess that will remain the official stance

well, except "just use leveldb"

[twilit elbow]

Looked into that, not as easy.

[tawdry karma]

Maybe there is a solution where we directly use leveldb in our own tooling. I’m also not trying to argue here. Just saying that I want to have a discussion about this.

[crude thicket]

There are plans to provide a CLI for common JSON pack/unpack operations

[sage silo]

omg windows ❤️

[crude thicket]

Not MeasuredTemplates

[tawdry karma]

Thanks, that’s exactly what I was hoping for ❤️

[inner hollow]

Not just windows, the implications of walls that can change vision on the other side depending on a token's proximity to the wall

[analog wagon]

~ foggy ~ windows

[mild fiber]

I have no idea how that would look like

[tame barn]

there is information available here https://github.com/foundryvtt/foundryvtt/issues/5065. It's not comprehensive, but it's a good start.

[mild fiber]

I demand screenshots!

[inner frost]

I immediately thought of ICE actors for my SWADE game

[tame barn]

not for measured templates at first, but it's something that could be expanded to support in the future

[lavish hornet]

Hoping the window thing changes the FoV as you approach (as opposed to just radial range).

[vestal moat]

Yeah I'd love some more support on custom MeasuredTemplate types. Right now the easiest thing to do is just hijack an existing type and override the shape methods based on some extra flags that a mod can provide. But I'd love a better way that's actually supported

[safe pelican]

I imagine you could use the Classic Level NPM package directly to create them

[gritty wolf]

Exciting stuff! Time to dig back into to the 5e developer tooling

[tame cedar]

I could do the same with NeDB but opted not to because I didn't need to and would rather not add development packages unless I need to. Well I guess now I need to, unless we actually get those tools

[tame barn]

We will have some screenshots/videos to show of windows soon

[safe pelican]

I rely on having the ability create/modify db files a lot at World Smiths and with my work at The Forge, so I think we definitely need to have a way to do that without going through the FVTT client.

[proud turtle]

Curious why LevelDB though. Yes it is fast, but it is, similarly to NeDB, abandoned. But by Google.

[tame cedar]

Google likes to abandon stuff but that is quite an interesting point

[grave jasper]

ClassicLevel is reasonably maintained, with commits only 3 months ago

[tame cedar]

I suspect it was the best compromise of perfromance and work

[grave jasper]

it has a wide community, good performance, and a clean API that allows us to do some potentially really cool stuff

[proud turtle]

Ok so you are going with the abstract layer

[grave jasper]

https://github.com/Level/classic-level for reference

[modern pendant]

For the JSON/YAML to NeDB conversion process that systems/modules do, that's on the systems to maintain compatibility with the format that Foundry expects IMO. It's well outside the API, and when I set that up for 13th Age, I did it with the assumption that it was a convenience tool for me and I would likely have to revisit it in the future.

With that said, I'm interested to see what comes of the CLI. I don't think I could use it in my process, but I'd like to see what I could pull from it regardless.

[inner hollow]

That's true. But people pointing out that it seems awfully supported when dnd5e is doing it have a point

[grave jasper]

I won't promise anything, but I have hopes to provide core options for human & commit-friendly serialization

[tame cedar]

I see it a lot in premium content. It's just so much easier to keep large amounts of data seperated and you can also actually automate the process. Change a single file, push a release.

[modern pendant]

Oh yeah, having human-readable versions is absolutely critical for long-term maintenance IMO. When we switched the 13th Age compendiums from pack files to YAML, it made a massive improvement in both editing them and reviewing changes.

[twilit elbow]

Last non-automated commit was half a year ago.

Oh, they have more active branch.

[grave jasper]

for a mature database, that's not unsurprising for their main branch. They are keeping up on security updates, and there is a wide community of people building things for Leveldb that we can lean on as well

I fully understand the fear of google products, but I don't hold the same fears for google technologies

[twilit elbow]

As long as someone has picked them up and continues to work on them, they're okay. But if they're abandoned by everyone else, too, then there's some concern.

[thick seal]

Yea, the base LevelDB is not accepting new features, just bug, security and other up keep fixes.

It's really not going anywhere, it's used in Minecraft Bedrock for the world/save data.

*Ask me how I know.*

[proud turtle]

What DBs were considered in the process? PouchDB? Redis? RxDb? Perhaps even IndexedDb?

[thick seal]

I can tell you Redis was not. Would not have made the list at all.

[proud turtle]

How so

[thick seal]

And I have zero inside knowledge on that.

[hazy harbor]

I tried to suggest using NathDB which just replaces every key index ID with a different profanity but nobody took the bite.

[thick seal]

No external DB server was most probably on the top of the requirements.

Redis fails that and, if I ignored that, would be a total bear to have some arbitrary number of databases that are independent for compendiums.

[severe jacinth]

You would run into characterset problems when you started to need profanity from languages that don't use the Latin alphabet.

But can I interest you in EmojiDB 💾 ?

[serene osprey]

Oh yea, Redis' persistence model is designed for recoverability, not portability. Foundry's Database solution needs to mainly fill three requirements imo:

1. It needs to work, reliably and securely enough
2. It needs to be so easy to set up that the foundry server can do it on its own without messing with anything else on the system
3. Exportable and Portable without being too much of a burden on the developer. Ideally single file.

[inner hollow]

You can get creative and chain profanities, dramatically increasing your dictionary size

[serene osprey]

if LevelDB is still like, relevant enough to its users that it has no glaring security issues, I don't see why a Feature EOL should matter if it covers all of Foundry's usecases

[obtuse basin]

Sounds great 🙂

[tame barn]

We evaluated around 7-8 different potential technologies to use. A non-negotiable requirement is that the db technology we use needs to be an embedded local database that can be included inside an Electron app. MongoDB, Postgres, Redis, along with most other "serious" databases, are disqualified on these grounds.

[grave jasper]

TeDb, LowDb, GUN, and sqlite

[serene osprey]

Is the migration straight forward or do you have to use both databases in tandem for it? If the later, will there be a date at which you permanently discontinue NeDB?

[safe pelican]

I assume you also looked at this NeDB fork? https://github.com/seald/nedb

Not that I don't think ClassicLevel is a good choice

[grave jasper]

We evaluated keeping on nedb, but level had better performance and features

[sage lily]

~~For anyone interested here some comments from the man on the sky about it the choice https://github.com/foundryvtt/foundryvtt/issues/5065#event-8246536195 ~~ never mind it was already linked...

[autumn forge]

Data is compressed and stored in binary format. This means that database files are no longer human readable.
Dang, no more using sed to search and replace on my .db files ☹️

[small compass]

If any thing it's better, because it'll deter people from fiddling with those tempting human-readable .db files

[autumn forge]

Out of interest, did you look at SQLite?

[inner hollow]

#1065372745581469727 message

[inner hollow]

I haven't looked at the new database format, but I'll bet they wanted to keep it NoSQL-like, rather than having a relational database like SQLite

[autumn forge]

Yeah, as a professional developer, binary db files are much better, though I will miss the hackiness of sed-ing the files (though I've only done it about twice)

[inner hollow]

It's one of those things where some of us miss the power. But also, I won't be sad to see people who have no business poking the files not doing so anymore. That has lead to data loss in the past

[autumn forge]

I'll have to look into LevelDB... I presume there will be some scriptable way to alter the files? The few times I've had to do it were to bulk change some URLs

[twilit elbow]

Shouldn't the release article say Foggier instead of Fogger?

[hazy harbor]

ruins the flow of my pun, mana

[autumn forge]

It was a daft pun

[gaunt solstice]

To help ease the tooling transition to the new DB format, any plans on an example for tooling? Kind of a leading question b/c I'm wondering if that was one of the ideas for this package: https://foundryvtt.com/packages/dnd5e-srd

[tame barn]

Yeah, we want to provide some official tooling that can be used to pack/unpack the new database files into a maintainable JSON format.

That tooling doesn't exist yet in V11 Prototype 1, but it's on our agenda for the remainder of the V11 development cycle.

[spice pewter]

I'll join my voice to what everyone else said above and confirm "I agree" with everything, lol.
I think the change to leveldb will be good on the long term but will likely be painful right now, especially with all the tool changes. I would have to request :

• limiting fields or documents to 16KB max size for example.
  The most used tool that I have which edits db files is one that does a massive search/replace of repeated strings that broken modules insert into an entity becomes a few hundred MBs in size. If we can't fix broken DBs using a tool like that, then I hope core Foundry will now have its own sanity limits on entity sizes (mongodb has a max 16MB size per document, though in this case, I think it's be good to also have a max 16KB per field for example, so you don't have overly huge flags, or a 100MB base64 embeded image inside your actor's bio)

[tame barn]

bear in mind that base64 is now extracted to static files rather than stored in the database, but I take your meaning

[spice pewter]

I did test it when you implemented that, but you only extract base64 from img field, not from <img src="base64..."> inside the bio of an actor, or inside a journal entry, which is what I referred to above

(Ideally, if you could also do that, it'd be great 🙂 )

[tame barn]

if the biography is edited in prosemirror, pasted base64 image data is also extracted as a static file

but that doesn't migrate existing data

[spice pewter]

I think the base64 stuff was in v9 and prosemirror in v10, so that's probably why it hadn't worked when I tested (I only tested that use case when the base64 removal support was added, so in v9)

[inner hollow]

No, base64 stuff was V10, I'm pretty sure

[hazy harbor]

base64 changes were early v10

[spice pewter]

Humm... not sure then, cause I distinctly remember testing that use case and finding that it did work in the img field but not in the html

[inner hollow]

So long ago it feels like V9 😆

[spice pewter]

anyways, that's off topic right now 🙂

[lavish hornet]

Feel like the persistent module storage will be manageable with respect to bazaar delivery etc?

[spice pewter]

Another topic I'd like to discuss, is the persistent storage for modules. That's a great idea! However, I do feel like storing the data under the module's folder may be a bit risky because an accidental uninstall would delete it, which is something that can happen quite often when users are trying to debug issues. I'd recommend storing the data under a new Storage folder, sibling to Data, this way it's safer and independent on the module being installed or not, but that would require new APIs to upload files to it (I do see the new uploadPersistent)

[lavish hornet]

Haha

[spice pewter]

I think the persistent storage would be a non-issue on the Forge since any upload already goes to the user's assets library which is separate, so all uploads made to the module's folder are already 'persistent' 🙂 My proposed change above would actually complicate things for us, but would be better for self-hosted users 🙂

[tame barn]

One thing to be clear about is that it is 100% intentional that when a module is uninstalled its persistent data is also removed. This data is only intended to survive package updates not uninstalls.

Wherever we end up placing that data, it's gone when the user uninstalls the package, even if it was an "accidental" uninstall

We need to make sure that is abundantly clear to any developer who intends to make use of this feature as well as to users who may be uninstalling modules which have persisted data.

[lavish hornet]

We do use “do a quick uninstall/reinstall” all the time as a troubleshooting step for modules that are borked (usually stuck on an old version due to manifest mishaps). Will need to have a means of retaining in those cases.

[tame barn]

My naive response here would be for us to identify and cure the root cause rather than fixing the symptom of that.

let's get rid of the "manifest mishaps"

[lavish hornet]

Suuuuuure

[spice pewter]

https://tenor.com/view/it-crowd-chris-o-dowd-roy-have-you-tried-turning-it-off-and-on-again-off-gif-4711577

[thick seal]

Humor me, but can we add blocking all uploads to modules/systems that don't have an opt out flag on them? Far too many people keep uploading more token art into, say, dnd5e, only to get nuked on update.

[lavish hornet]

😅

[thick seal]

We've suggested that for issues outside Foundry at least. FS corruption or sync programs messing with Foundry data.

[hazy harbor]

I am not 100% certain, but i recall reading somewhere that 'uninstall the package' for packages that have persistent data DOES raise a warning that it will wipe persistent data.

[tame barn]

I recall that being a feature that I discussed with Cody when we were implementing it

[inner hollow]

I also suspect that the number of modules using this for intentional cross-update data storage will probably not be huge overall

[tame barn]

We're going to be reworking some aspects of the setup view in Prototype 2 and beyond (I don't want to derail this topic to discuss that though). We may be able to add a "repair" or "reinstall" option for modules

[grave jasper]

solid suggestion, Github it

[thick seal]

[lavish hornet]

https://github.com/foundryvtt/foundryvtt/issues/8467

[thick seal]

Ah beaten to the punch.

[thick seal]

I added my thoughts. https://github.com/foundryvtt/foundryvtt/issues/8467#issuecomment-1396383586

[river root]

Not sure this is the right place for feedback?

• v11 seems to go in a direction I love: more performance, less breakage. Yay!
• Love the QoL stuff like scene resizing (and small ones like AE descriptions!)
• Typo in release notes: "will closes"
• The per-module persistent storage folder seems like a trap for newer devs, if it gets deleted on un/reinstalls. Really only useful as a cache then? Needs to have big scary warnings attached.

[twilit elbow]

One potential option for the removal of the uninstall/reinstall, is that there's some kind of "force update" option, that does not check the currently installed manifest for updates, but whatever is in the package registry. I've seen plenty of modules that get the manifest field "corrupt" and then never update, but if you re-install them, you get newer version. I honestly thought the update process already checked Foundry's site for updates, but these cases clearly indicate it is not the case.

Also, installing over with manifest URL probably does the forced update for unregistered modules. I hope. So that should preserve the storage in those cases.

[void marlin]

We should all be excited about new features, not fearful about the amount of work updating our packages might take. Ideally, we would like to release a V11 build where community developers can simply update their listed compatibility and have their package work the same as in V10. We want to make V11 a smooth transition for everyone, but we can't do it alone.

Due to a stressful time I'm not very active in the community and developing my modules but please let me say that for this statement alone you have my utmost and sincere gratitude. ❤️

[azure cave]

RE: Persistent Storage

Does non GM users have permissions to write to this storage, or is this something the module has to manage itself?

[twilit elbow]

Probably controlled by the normal upload permission.

[proud turtle]

Agree, I have struggled a bit with this concept. Package registry at foundryvtt.com is only checked if the local manifest url is valid. Should at least fall back to package registry or better yet check the registry first

[tame barn]

A guiding principle has always been for Foundry to be decentralized so that it’s not reliant on a central web server. Centralization brings some obvious benefits, but trusting a locally installed module to designate its own remote manifest URL allows for the foundry VTT website to be optional in the whole equation

[proud turtle]

Right, but that doesn't have to come at the cost of user experience. You have a package manager, why not use it? Using the package registry as a first point of contact doesn't preclude you from falling back to the local manifest. Not only that, but it doesn't prevent non-listed packages working either.

You already have this confusing manifest upddate flowchart that does utilize the package registry. It could be improved a lot

[tame barn]

We do use the package registry though. That’s what the “sidegrade” workflow does to replace local data with different/better data from the website

[proud turtle]

Exactly

It only works if local manifest points to a valid remote one.

[tame barn]

That is not correct

[proud turtle]

Packages change hands, hosting urls change, etc. etc. It always breaks foundry's package update model

I still get support questions about this: #free-league message simply because I stopped using /releases/latest to host manifests on Github.

There is no path for the user but reinstalling the package

[tame barn]

If you can write up a clear summary of the situation it makes sense for @grave jasper to review and either verify that there is an issue or communicate the misunderstanding

This is not a V11P1 change though so suggest using #956306859491471420

[thick seal]

Going to make a guess you removed the exact package version they have from the system listing, thus there is nothing for Foundry to sidegrade.

[proud turtle]

I trim versions on the listings page as they become redundant (to keep the list short an relevant for a first time user).

If this plays into the update process I am very much inclined to put all of them back

[thick seal]

Without doing digging, it might. Just call it a gut feeling.

The check for migration might be looking up package ID and version looking for changed URLs.

[proud turtle]

I was under the impression foundry used the version ranges listed to look up.

[thick seal]

Yes, for installs, might not for side migrations.

[proud turtle]

I'll post in correct channel

[grave jasper]

https://github.com/foundryvtt/foundryvtt/issues/8115

I believe you're looking for this enhancement

[proud turtle]

Ah, yes. Very much this

[quiet holly]

First round of hammering on V11 in my system and the only thing I can find is some deprecation warnings regarding Item and Actor localization keys. Looks great so far!

[hollow stirrup]

Thus far all my modules load up and seem to be functioning without issue. A few minor updates to do on some (like SweetNothings), but... other than my regularly scheduled updates and maintenance, things look great!

[twilit elbow]

Attempting to update an old version of PF1 with the v11 prototype caused the updater to crash. The last bits in the logs are errors claiming the packs lack type (they don't).

[twilit elbow]

FoundryVTT | 2023-01-20 03:30:19 | [warn] System validation errors:
  packs:
    21:
      type: may not be undefined
    20:
      type: may not be undefined

etc.

After getting around that by just hooking up dev version of the system, launching a world gives fun errors:

FoundryVTT | 2023-01-20 03:36:30 | [warn] Actor [o8n6yv1cqRgirAm8] validation errors:
  type: Cannot read properties of undefined (reading 'Actor')
  items:
    T0bSbexlzIX3KmUJ:
      type: Cannot read properties of undefined (reading 'Item')
    JcZhYq6beN6VIlRC:
      type: Cannot read properties of undefined (reading 'Item')

And so on, the same error keeps going with different IDs.

[tame barn]

Could you share:

a = game.actors.getInvalid("o8n6yv1cqRgirAm8");
i = a.items.get("T0bSbexlzIX3KmUJ");
console.log(i.toObject()); // what does this look like?

actually... that might correct the data, you might need to look at:

a = game.data.actors.find(a => a._id === "o8n6yv1cqRgirAm8");
i = a.items.find(i => i._id === "T0bSbexlzIX3KmUJ");
console.log(i);

[twilit elbow]

Well, the errors aren't showing up anymore now that I launched v11 again to get that data....

Oh, I idea, I guess I should delete the v11 databases to redo the initial conversion.

Yeah, I don't understand, the errors aren't happening anymore.

[lavish hornet]

Looks like something changed in the Configure Game Settings CSS...

[fleet ledge]

Should I use the v11 reporting to report core bugs that don't affect my modules or leave it to another place?

[lavish hornet]

#1065764070680186930 yep

[fleet ledge]

Thanks, I wasn't following the whole category, so those new channels weren't showing for me

[autumn forge]

Do I need to give any feedback about modules which seem to work absolutely fine in v11 without any changes?

[rapid fjord]

Regarding the LevelDB, I see a folder for each database, and inside is a file that ends with** ".log"** which is the only sizable file within the folder. Is all the data really being stored inside a file with the ".log" extension on it?
Certain people might have scripts which just automatically remove log files from their systems, so having log files containing critical data probably isn't a good idea.

[small compass]

I noticed this too, the extensionless files look very small, only the log really had contents

[autumn forge]

yeah that log file confused me. I thought it must be some log of the upgrade from NeDB to LevelsDB at first, but it does look to be the contents

[rapid fjord]

Looking at LevelsDB on the web, it really does store data in .log files!

[tame barn]

I don't think that is correct. The binary data is stored in a file with the .ldb extension

[rapid fjord]

In a closed down world, the journal.db was 384 kb in size. The contents of the journal folder are:

[severe jacinth]

I recall reading something in the spec about one of the important files being called the "log" - I think it's the most recently active layer of the data.

[rapid fjord]

I quickly found the following using google: https://blog.senx.io/demystifying-leveldb/#:~:text=It stores new key%2Fvalue,memory structure called the memtable.

[rapid fjord]

I don't have any .ldb files

[severe jacinth]

Iirc, as the DB gets bigger, it ends up with more files. But log is the most recent stuff.

[severe jacinth]

Probably don't have enough data in that DB.

It hasn't done its magic to it yet.

[rapid fjord]

No. so all the important data is in a file with the "log" extension.

[severe jacinth]

I believe that is the case. Any tooling that indiscriminately deletes files with that extension will likely need tamed.

[rapid fjord]

But users will need to be warned that these are important files in Foundry 11

[autumn forge]

I didn't have and .ldb files, but now I do. I did just restart my server, maybe that caused them to get written?

[twilit elbow]

I'm seeing all the data being in .log also, with no .ldb file.

If it matters, my test only included conversion from old nedb packs.

I was also seeing some weirdness with the sizes, like the original nedb was 250kB, but the .log was 500kB, then in followup attempts the .log was like 50kb smaller than the source nedb, which felt questionable.

[autumn forge]

Well... all my modules seem to work absolutely fine in v11 (Cautions Gamemasters Pack, Damage Log, Connection Monitor, Multi Token Status)

[severe jacinth]

A log file (*.log) stores a sequence of recent updates. Each update is appended to the current log file. When the log file reaches a pre-determined size (approximately 4MB by default), it is converted to a sorted table (see below) and a new log file is created for future updates.

A copy of the current log file is kept in an in-memory structure (the memtable). This copy is consulted on every read so that read operations reflect all logged updates.

https://github.com/google/leveldb/blob/main/doc/impl.md

[grave jasper]

I thought we were compacting at world shutdown, but perhaps something isn't going 100% right

[twilit elbow]

PF1 is making explosions with v11, but I'll leave it to @marsh belfry & co to figure out.

[twilit elbow]

NeDB was definitely compacted on shutdown, I have no clue of the new.

[severe jacinth]

Tl;Dr: The Database is organized into a series of files that make up multiple "levels" of the data. The first level, level 0, is the .log That was incorrect, the .log file is above level zero if I'm reading this correctly. file. As the files grow, and pass a certain size, the DB re-organizes itself into more levels.

If no compactions have happened yet, you won't have any .ldb files.

(Each level has some number of files allowed in it, and each file some allowed size)

[twilit elbow]

That would indicate it's not compacted on shutdown..

[severe jacinth]

Not to be confused with the LOG file which is actually a log.

[grave jasper]

we'll definitely investigate this 👍

[twilit elbow]

Here for example, source nedb file and the produced leveldb files (after several world restarts and shutdown of the entire server)

[autumn forge]

well I eventually ended up with .ldb files, but not sure how or why

[severe jacinth]

Most likely a compaction was triggered.

[severe jacinth]

Maybe telling it to compact is just a "suggestion" and it looks at the files and says "this is good" then does nothing?

[grave jasper]

maybe!

[twilit elbow]

There must be harder compaction command in that case. There's no way they've omitted something that is guaranteed to reduce data size.

[rapid fjord]

The ldb got created when I started the world, not when stopping the world (and the log file was set to 0 size).
This was many minutes after using another world.

[thick seal]

Non zero chance the LevelDB people consider 4 MB trivial.

[crude thicket]

4MB is typically the amount that will fit in an 'extended page'

Which makes sense if it holds the entire log in memory

[rapid fjord]

Regarding creation of LevelDB files, what happens in the following scenario:

• I have module X which contains either macros or compendiums, loaded in Foundry 10
• I upgrade to Foundry 11 and launch a world that uses module X (this creates LevelDB for each macro/compendium in the module)
• Later, I update module X to a later version which contains macro/compendium, but the module is still maintained in Foundry 10 (so have .db files)
• I relaunch the world - will the .db files be migrated again into LevelDB format, or will the module still try to use the previously generated LevelDB files?

[thick seal]

Foundry deletes the module, so it would reconvert if the module shipped NeDB compendiums in the update.

[crude thicket]

Updating the module will wipe the module's installation directory. When the user loads into their world again, the .db files will be re-migrated.

[rapid fjord]

Ah, of course.

[inner hollow]

It sounds like the answer in this case is "don't commit your LevelDB files to your repo"

[thick seal]

I don't even commit the .db files. Too much churn in the diffs.

[twilit elbow]

That actually causes one complication. If you're using nedb bundling outside of foundry, you'll need to add leveldb pruning to the build process so the changes reflect.

Hopefully the promised tools will make this a non-issue.

[thick seal]

I built tools to do this under NeDB over two years ago, I'll just do it again.

[twilit elbow]

I'm curious how the packs field will be changed with v11? Clearly it no longer will refer to .db files.

[thick seal]

I've not had the time to boot my PC and dev env, but I expect it's either a flag or looking at the path or target file.

[rapid fjord]

The ldb files are created immediately on starting the world, regardless of how long it was since you stopped the world.

[twilit elbow]

Not what I experienced, I launched a world several times with no change.

[twilit elbow]

My homebrew system with no public releases seems to be starting up and running with zero issues. Some deprecation warnings, of course, but those are not important.

[rapid fjord]

My main issue during testing was with systems or modules which had set "compatibility.maximum" to "10" in their module.json/system.json.
Some better guidance on the use of when to use maximum for developers would be useful, particularly in the lead up to the release of 11.

[twilit elbow]

People have set maximum because they didn't want to deal with bug reports for a foundry version they haven't tested. At least that's what I assume.

[small compass]

Is that not the point of the maximum field? So you don't open a world on a core version the system isn't compatible with and brick your world until it is. That was an all too common occurrence in V10

[twilit elbow]

That's what backups are for.

[small compass]

I mean, yes, but you can still prevent the situation from ever happening

[crude thicket]

I'm not sure what better messaging we could use. What else does 'maximum' suggest in this context?

[rapid fjord]

I thought the point of the "verified" field was to tell users what version it has been tested against. The "maximum" absolutely prevents it from being used with later versions.

[twilit elbow]

Rename maximum with broken?

[crude thicket]

In the same way that `minimum' absolutely prevents it from being used with earlier versions

[small compass]

But verified doesn't block you from opening worlds. The reality is users often don't care about "warnings" and will blindly open their worlds on "unverified" versions

[twilit elbow]

The problem mostly with maximum is that people use it as precaution rather than fact.

Maybe add second one-time confirmation dialog when opening such worlds?

[small compass]

Like the "opening this world will migrate your stuff" one no one reads?

[twilit elbow]

It'll be clicked through like verything else, but you should not handhold people who refuse to read instructions.

[crude thicket]

I guess my point is: Is it the case that developers do not understand that setting 'maximum' will prevent their package from being used in a later version? If that's not the case, then I don't know what other guidance we can offer. It's their decision to do so, and a deliberate one.

[twilit elbow]

Migration warning doesn't happen because the verified version is "incorrect"

[severe jacinth]

I think maximum is working as intended.

[grave jasper]

the block of world launch if minimum or maximum is exceeded is intentional, and if developers are using it to hedge against the average user launching into an as-of-yet untested new major version, I'm comfortable with that usage.

As a user looking to test V11, you can locally edit and remove that maximum - essentially taking the safeties off

[severe jacinth]

I think people beta testing a new version with a test server are probably capable of going into their modules folder and editing the manifest if they need to force-enable something.

There is no need for the average user to override this.

[grave jasper]

possibly the CLI could support something like fvtt packages remove-safeties "1001-fish"

but that doesn't seem much easier

[severe jacinth]

It's even trivial to write a Node script to update all your installed modules maximum fields if you wanted to.

It's just JSON after all.

[hazy harbor]

incase no one responded to you --- if your modules work perfectly in v11 that is excellent news

but we probably don't need reports for that

(maybe once we open the compatibility spreadsheet)

[twilit elbow]

My rolltable sheet had broken with v11, but I don't think it's reasonable to expect Foundry to not break that. The super.getData() had become async and the _animateRoll hooked into different UI elements than before.

[hazy harbor]

Reposting this before closing and locking this thread in favour of other feedback channels.