Efficient Storage Mod

[autumn radish]

If anyone knows how to benchmark properly, or preferrably can benchmark it for me, I'd love to talk once I get everything working

I'm hoping, fingers crossed, that this is better than any storage container size 10+. My guess is it's somewhere in the 20-30 range where improvements are seen

All it does is create a couple of special entites like Memory Storage, but it can only store as much as a normal container of that item. So one wooden chest of iron ore can store 800, so an optimized "chest" can only store 800

[dense turtle]

well you could have the exact same setup but with normal chests and see how fast you can run it

or build a crap ton of them and do the same thing, that way the UPS numbers are a little bit more bearable

[teal trench]

cpp-based unsorted container with o(1) access time from factorio devs would be nice

[autumn radish]

Yeah, i can do a poor mans benchmark, but i would love a proper one

[autumn radish]

Thats essentially what it tries to do

Filtered slots, only 2 slots total, minimal memory actions

[untold whale]

How goes the progress on this? I went all in on warehouses for my train stations and am paying the UPS price as a result, this would be a great fix

[autumn radish]

Going through a rewrite to support larger warehouses without a ton of separate entities. May i ask, what is the lowest stack size of items you are transporting?

I assume 50?

@untold whale ^

[untold whale]

No I want to say there's a station for nexelit batteries which is a stack size of 10 would be my lowest

[untold whale]

But it would be pretty easy to handle that differently if 50 being the lowest stack size was some kind of constraint by removing stations below 50, would only affect a couple

I'm curious how the mod works, do you have to add in a special warehouse that's UPS efficient? Or would it work with all existing chests and things

[autumn radish]

I can configure it to not accept items of stack size lower than 50

[autumn radish]

It works the same as memory storage by melon. A small inventory with a few slots, saving the total inventory to global. A combinator is used to read the total item count. Updating the global data can be spread out between ticks, so not everything is updated every tick. Im creating a basic chest with 2 slots for every size of chest in the game (1x1, 2x2, etc) then a combinator with the graphic of each chest in the game (wooden, iron, steel, etc).

[untold whale]

Oh gotcha, so it's essentially a "small chest" with a couple slots that then stores its inventory elsewhere, and the combinator handles moving things between "storage" and the couple of slots, is that correct?

That's very clever. Does it ever have issues with items getting stuck in the chest slots or anything like that? Also sounds like that would necessitate replacing all my warehouses with the new combinator

Which is still worth it to save on UPS, my steam deck hates me rn

[untold whale]

It would be nice if it worked with all stack sizes just for consistency's sake, but it would be a super minor inconvenience

[autumn radish]

Combinator just sends output so the player can use it, but im thinking of using that signal as the inventory count to save operations

[autumn radish]

No, it has no issues because it uses filters on the item slots. You dont need to replace them with combinators, i'm gonna include a selection planner to transfer from vanilla to and from custom warehouses

[untold whale]

I'm very interested in this mod and I very much appreciate your hard work, it sounds like a fantastic idea and sounds like you're making the implementation as painless as possible which I appreciate

[autumn radish]

Yes, but with the delayed update system each chest is updated once per second, and with big warehouses (more insertion opporotunities) and small stack sizes, the internal slots fill up fast. Too fast for it to update. Once i finish the mod im gonna benchmark different inventory sizes to see what works best, then maybe include a dynamic inventory system for smaller stacksize items.

[autumn radish]

Thats the plan when i make qol/efficiency mods. No gameplay differences, minimal work on the player's part

It also doesnt work with items with health bars (tools, science, ammo) but i have a potential fix that i might use.

[untold whale]

Interesting, makes sense why it would be an issue. I wonder then would it just mean you pull small stack size items out of the chest at a slower rate then? Or does it cause weird problems

I'm not sure if the warehouses are the main cause of my UPS issues, but I know I have 1-2k of them with lots of inserters so I wouldn't be surprised if that's a big chunk of it

[untold whale]

You guys who make QoL mods are the unsung heroes of factorio lol

[autumn radish]

Take armor for example, with my current 2 item inventory. One slot is always full (assuming the total item count is greater than one stack) and one is always empty. Once a single item (equal to one stack in this case) is inserted, it has to wait until the chest is updated before another can be inserted

[autumn radish]

Its generally seen when inserter update times are high, because they have to search each slot of the chest

[untold whale]

I'll have to run debug when I get home and check my inserter update times, that's good to know

[autumn radish]

Iirc removal is more expensive than insertion, but i dont know by how much

[untold whale]

Also I'm curious, so you have a 2 item inventory right? What happens when you try to pull 10 different items out of the chest at the same time?

I like to use my warehouses as a jumbled bag I pull out of

I wonder could you set the chest size as a dynamic amount based on demand? I know nothing about factorio modding so that might be incredibly impractical

[autumn radish]

These warehouses will be strictly single-item. But i could probably get a multi-item warehouse going

[autumn radish]

No. Its not as much dynamic as it is replacing the chest with a bigger one

At that point, use a bigger inventory to begin with

[untold whale]

Yeah darn, that would be nice if you could set chest sizes but it doesn't not surprise me that it's not feasible lol

Also I use all my train stations as multi-stop multi-item warehouses, which again is probably a UPS nightmare

I guess multi-item warehouses might be an extra challenge task to solve once you're able to get the single-item version working. I can imagine it might be much harder to implement though

[autumn radish]

Just thought of a multi-item solve. Shouldnt be too hard but it will definitely need a set of new computations.

Definitely an after the fact addition

If i do the multi-item solution, do you think from a gameplay perspective it should work out of the box or a selection (i.e. you select the number of items it might contain)

[untold whale]

Yeah sorry to add onto your task list with my unreasonable requests lol. Even a one- item warehouse would still be a great add

I think for me, selecting the items it could contain wouldn't be a problem at all

[autumn radish]

No, its completely reasonable. I just hadnt thought of it yet

[untold whale]

I set the train requests anyways, so it would just be the same items

I think there's the possibility though of train station contamination, where an item accidently goes where it's not supposed to

[autumn radish]

If i make it work out of the box, how many different items do you think would be enough? Like 8? 10?

[untold whale]

How expensive computationally is it to replace a chest with a bigger inventory? If you had to do it whenever a new item got added to a chest

Also I play py, so recently I just set up a station with like 10 items in it so it's hard to say

There's definitely an upper limit for sure

Like worst case scenario, I think the max you could pull out of a depot would be like 20 or 30 or something crazy. So a lot, but still x10 or x20 less slots than a depots 800 right

Also one other thought I have, a lot of people use cranes in py which moves one stack at a time. So if you have 2 cranes for one warehouse, maybe you want two empty stacks for the cranes to drop into instead of 1

I wonder if you could make some of these parameters mod settings? For "heavy users" like myself being able to set more item slots? Again not sure if these suggestions are reasonable at all lol, just spitballing here

[untold whale]

Sorry to hit you with a million suggestions for your mod in progress lol

[autumn radish]

Unsure. It would be best if it always had a few of each item... that way it would never need to be replaced... ill cross that bridge when i get to it

[autumn radish]

Yeah, thats part of the benchmarking and dynamic inventory stuff. Depending on the mods installed (inserter cranes, loaders, qai, etc) the minimum inventory size will change

[autumn radish]

Probably. Depends on the implementation

[untold whale]

Well i look forward to seeing what you come up with, seems like a great idea. Thanks for taking the time to tell me about your mod

I can imagine designing mods must be difficult trying to cover all the potential use cases people come up with lol

[autumn radish]

Difficult, but not impossible. Asking the end users helps

[untold whale]

I feel like you're right, that the best implementation would be something that doesn't need to self-replace to change stack size. And ideally something that works out of the box without too much fiddling

But that's much easier said than done lol

[autumn radish]

I think ill keep the single and multi item systems separate, so you have to specify if a chest is multi-item

[untold whale]

That sounds pretty reasonable

I think even if you had to specify every item that goes into the chest for multi-item, that wouldn't be a problem for my use cases

[autumn radish]

@untold whale I have a beta version avaliable, would you be interested in testing it for me?

[untold whale]

Oh sure I'd love to! Would be interesting to see if it helps my UPS

[autumn radish]

It's only got single item storage atm

And might be savebreaking

[untold whale]

Hmm I might not be able to load it directly into my current py save then right? If I have lots of multi-item warehouses

[autumn radish]

You can always load it and not use it (although i recommend backups/autosaves)

It comes with an upgrade planner so it's selective on where you use it

[untold whale]

I can make backups of my saves and mod folders and everything to beta test, I'm trying to think of what the easiest way to test then with single item storage

Oh gotcha, if you have to use an upgrade planner to upgrade the warehouse, that's much more workable

Yeah sure I'd be happy to beta test then! I probably won't see huge UPS gains then because most of my warehouses are multi item, but I do have some single item storages so it'll definitely be useful and I can give it a good test run

[autumn radish]

Note I haven't tested it's mod compatability yet, it may not load properly

[untold whale]

Yeah I'll make sure to make backups then, and I can give it a good test lol

[autumn radish]

https://mods.factorio.com/mod/efficient-storage

[untold whale]

I usually am able to get on factorio around 8 30 pm, I'll make backups and everything and try it out tonight!

[autumn radish]

Sweet. Thanks!

[untold whale]

No thank you for your hard work!

[untold whale]

Apologizes for the screenshot of the screen but i got this error

[autumn radish]

Strange. Was that on load with pY?

[untold whale]

Yeah it was, I have a load of other random QoL mods too so it could've been one of those

[autumn radish]

No, something else broke i think

[autumn radish]

@untold whale do you happen to use loaders with your warehouses?

[untold whale]

Nope just cranes and inserters

[autumn radish]

Sweet, cause loaders are currently broken lol

I'll upload a new version in a few

[autumn radish]

@untold whale mod is updated, changelog has new details

[untold whale]

Sweet, I'll give it a try when I get home

[quasi void]

I have a K2 save that looks like this, almost every cityblock (unfortunately most blocks have multi-requester warehouses) has multiple warehouses (merging chests) inserters and loaders. I wanted to test the mod but I couldn't figure out how to use it. Is it enough to install it? It doesn't seem to have changed the UPS much

[autumn radish]

There's a selection tool to resize items.... I'm only just now realizing that it's completely innaccessible lol. Let me fix that and push a new version

@quasi void ^^

[autumn radish]

Please update and try again @quasi void

[quasi void]

hmmm I get this error

[autumn radish]

It can only do single item chests

And i havent tested wide chests

[quasi void]

I can't convert empty or single item wide chests

[autumn radish]

Then I would consider them incompatible for now

[autumn radish]

new thought while watching dosh's recent video (would ping him here but he probably wouldnt like it lol) add chest entity variants of tanks, cars, and possibly storage wagons to allow for easier copy-pasting of them, as well as reducing logic overhead for physics, and entity count for the wagons. i think i'd include it as a base feature and not a setting, what do you guys think?

[quasi void]

I don't understand technical details at all, but I would like to help with testing as I love big warehouse + loader combos

but I really want this mod to support https://mods.factorio.com/mod/WideChests because all my base is built around it

[autumn radish]

the way my mod makes things UPS efficient is by updating the containers less than once per tick, usually once per 60. i haven't done benchmarking yet, so there could be wiggle room, but any mod that adds large (tile wise) containers can run into the issue where it can insert/remove items faster than my mod can update. i don't want to speed up the updates, because that will eat into UPS, so i will do my best to try and find a way around if possible

once i get benchmarking done, then i can start optimizing to allow for stuff like inserter cranes, bobs crazy inserters, etc

[quasi void]

This is the first time I've heard of Tick update, if I understand correctly, the mod also reduces warehouses with hundreds of slots to a few slots, right? Like having 50.000 iron plates in 1 slot instead of having 100 stacks of iron plates in each of 500 slots. Doesn't that in itself give a massive ups increase? People were saying that the reason why loader/inserter-warehouse interaction eats so many ups is because the more slots there are, the more checks happen every tick

[autumn radish]

it's impossible to have more than 100 items in a slot if the stack size is 100. the number of items is stored internally, i.e. in a variable invisible to the player. this mod reduces the number of slots, and stores the total in memory. the remaining slots still hold 100 items per slot (for items with a stack size 100) so if there are 2 slots total, keep it half empty to make sure that stuff can be inserted and removed, then you can only insert/remove 100 items per update before you are working faster than the internal logic. adding more slots allows for higher transfer rates, but too many slots reduces the UPS gain from the mod. you also run into throughput issues with low stack size items, which this mod does not support for that reason

[mortal sandal]

If you can refactor your code to support special containers with more frequent updates, you could probably trigger off the rotate event and change a container's update rate specifically when it needs it
For an example of rotating (or maybe a key I'm misremembering) to configure an entity, look at the shields from SE (I'm pretty sure that's a separate mod, but I remember it specifically in context of the SE overhual)

[autumn radish]

wdym trigger off the rotate event?

[mortal sandal]

https://lua-api.factorio.com/latest/events.html#on_player_rotated_entity

Ah, reading the description they probably did some jank to make it work with the shields

Then you can also look at the pushbuttons mod to see how it lets you 'press' the buttons and switches (constant combinators)

[autumn radish]

oh, you mean like press the button to change the update frequency?

[mortal sandal]

Yeah

[autumn radish]

id rather not, i want to try and keep everything behind the scenes and automatic

[mortal sandal]

I guess you could just see that you have to move a whole stack of items, and implicitly increase update rate so that the updates are only moving max 80% of a stack at a time

How to dynamically ramp down, tbd

[autumn radish]

i would prefer to keep it without dynamic polling so that UPS gain is the same no matter how much you use the mod