#Optimization idea?

1 messages · Page 1 of 1 (latest)

topaz sierra
#

TLDR: Combining logical parts of base into single entities can improve game performance

If this idea has already been posted somewhere, please let me know

~~ ~~

A typical base consists of many logical blocks, each with numerous interactions within itself and much less with other blocks. If you record a timelapse of a specific part of the base, you'll see it's almost periodic. This is assuming that there are no biters, players doesn't change anything inside and both input and output are sustainable

Base can be partitioned into systems, which consist of entities and connections to the outside world. For example, a system could be a set of belts, assemblers, and inserters, connected to the outside through chests

Each entity has its own state at every moment in time, so there is a flow - function that determines the state of the whole system over time

Let's consider following approach. The flow may be (almost) periodic. If this is true for many cycles over a certain period of time, it's likely indeed periodic function. We don't need to update all entities, only abstract system state and state of connections.

~~ ~~

This is not a "complete suggestion". It's a question (title) and suggestion for discussion

Some anticipated remarks

  • it's based - ok, sorry for this, could you give me the link?
  • just do it yourself - imho this should be in core, not in separate mod, isn't? and in this case could you help me a bit with that - what could i learn to do it?
  • there are many difficulties in implementing: this, this and this - yes, there are, If anyone is interested in discussing details, I'd be happy to explain further
kindred kite
#

The game wouldn't function the same since there wouldn't be a check for every interaction. Essentially you could optimize by making a super large building that takes resources and have unique recipe. For example takes in x amount of iron (and other resource) and spits out a engine.

topaz sierra
#

If you record a timelapse of a specific part of the base, you'll see it's almost periodic

I can record example tomorrow, but it seems obvious for me

kindred kite
#

That's only if it is working correctly which requires computation to make sure it works correctly

wooden cairn
#

Isn't that baisically Factoriossmo2?

lapis onyx
#

No

wooden cairn
#

ok

topaz sierra
lapis onyx
#

That simply puts things on different surfaces. It doesn't reduce the simulation load

topaz sierra
kindred kite
lapis onyx
#

Anyway, I honestly doubt something like this is feasible, as there are tons of ways the outside could affect such a module, and any of these would need to have it broken up instantly to accurately simulate everything

kindred kite
#

If make recipe based on factory planner (or helmod), that will work. And it will greatly reduce entity count

waxen mistBOT
#
RumRun's nuclear power plants

UPS friendly nuclear, achieved by converting a set of power plants into a single game entity. Removes all non-scalable calculations encountered by nuclear in a megabase; however, it still requires the ingredients, space, and fuel to use it.

Owner

RumRunner

Downloads

271

lapis onyx
#

This may be interesting for you though

topaz sierra
#

system is similar to finite state machine, if connections are stable.
Of course, if part of base some time works, some not - it couldn't be optimized in this way, but (for me) even in the midgame most parts are stable ones you have appropriate logistic.

I honestly doubt something like this is feasible
It's just the game - set of disturbances is finite. If players know, that stability improves performance - they would have possibility (and people who like spm will use it) to optimize their base to make such optimizations possible.
Example - belts, assemblers, chests, inserters. If we have overproduction (all belts mostly filled), it's enough to check

  1. energy
  2. available resources in input chests
  3. available space in output
  4. there are no building/breaking
lapis onyx
#

The circuit wire messes all of that up

topaz sierra
kindred kite
#

You dont need all of that. We can already do this with scripts. Changing engine code is whole another story that I doubt devs will do

topaz sierra
lapis onyx
#

I do see one rather significant issue: you still need to animate everything, and currently all that depends on the simulation

topaz sierra
kindred kite
#

It's based on same example of the mod mentioned above

lapis onyx
#

That mod doesn't use much scripting though, it adds prototypes for a bunch of premade reactors

kindred kite
#

To consume variable amount of resources against a production planner, most likely you will need a script

topaz sierra
kindred kite
#

In sense you want to take out the interactions computation which makes this games work. But if you want to do that, you can make scripts to simulate insertions of xyz resource and output ABC resource

lapis onyx
#

Anyway, I will note that if you want to reach the developers, you should post on the official forums, not the Discord.

ember cradle
#

The only way I can see an optimisation like this implemented is that if full input leads to full output.
But the biggest issue would be the interaction of inserters with belts. The first cyle it might grab items from pos 1, 2 and 3 but the second cycle is swings from the other way because an item was .01 tiles closer and it picks items on the positions 4.5, 3.5 and 2.5. Yes .5 because the belt also moved.

#

The amount of places an item can be on a belt is;

#

15 p/s at tick speed of 60 so 4 ticks per item.

#

4 ticks are between the items and 8 items fit on a belt. So 32 different sports an item can be on.

#

That is for a single item. For two items on a belt it would be 32*(32-7) = 800 different combinations. And I can already hear you scream, but with a full belt that never happens. Then I raise you the inserter that grabbed an item from that belt to make it a belt that is no longer full.

plush flame
#

There's a mod that lets you build links between different saves. Could enable swarm parallelization if someone set up good orchestration. @radiant isle was playing with it, maybe he can tell you more

hollow gale
radiant isle
hollow gale
#

it's more "let's have a bunch of servers all running at once and sync a bunch of things between them"

#

I was part of the gridlock cluster before that ended and man it's impressive

topaz sierra
#

Thanks for discussion
I'm going to record flow of some system and check, if my assumption correct
Is there existing mod for that? If not - #mod-dev-discussion is place where i can ask for advices on implementing this, right?

gentle stratus
#

And worse performance because I didn't use combinators and did more calculation stuff

topaz sierra
#

Dupebox is close - I'll definitely explore its implementation, but different - it's about parallelization in space, and my suggestion is about parallelization in time
Could you share your attempt? I'm not able to understand it now, but in a month it should be possible to learn enough about modding

plush ridge
plush ridge
#

On page 12, that I linked, there is my suggestion and under it I summarized most of what I found interesting on the previous 11 pages.
Link links to a forum post that other threads were merged into, so there are at least like five suggestion of this type.

Your type of suggestion is very similar to mine.

I don't agree it should be vanilla, because vanila is not made to launch 1 rocket per second and really doesn't need to be. Less then 0.1% of players get to that part. And the solution, as you describe, isn't exact. Entirely new game with this insight could be designed, as one of the last posts on page 12 states.

topaz sierra
#

Thanks! I've already started worring it wasn't suggested before

I've looked through this topic, and came to conclusion, that nobody actually need it, only some people, who find it interesting (not usefull, just interesting!) (including me):

That said, I enjoyed the base idea quite a bit and found it very creative, intriguing and inspiring.
Too bad there doesn't seem to be much interest beyond a very niche crowd 😦

Maybe another dev will correct me if they feel otherwise,
But this is something we are not really interested in doing
accuracy and granularity of the simulation is part of what makes Factorio such a great game to me,
Everything is 'real', nothing is just 'a number decrement here, increment over here'.

Although it seems weird to me because i think the core idea of Factorio is building megabase...

So, the question got its answer

gentle stratus
#

Yep, that is also the conclusion I have come to. There is no "right" way to optimize by skipping calculations like that. Closest I have gotten is clusterio gridworld which allows for basically infinite automatic scaling at the cost of compute and some jank.

rapid karma
rapid karma
#

you assume that there is a (say) magic-flow-function
((in pseudo-typescript-ish for the sake of example))

const observed_inner_states: Array<State> = []
var cycle_index: null | number = null

function flow(entities: SetOf<Entities>, connections: SetOf<Connections>){
  const actual_inner_state = magic({entities, connections})

  observed_inner_states.push(actual_inner_state)
  return actual_inner_state

  // but once we are sure we've found a "cycle", somehow...
  // we are now using a memoized version of the magic stuff directly:
  const memoized_inner_state = observed_inner_states[cycle_index]
  cycle_index = (cycle_index + 1) % full_cycle_length_we_found
  return memoized_inned_state
}```
#

I apologize for the sake of non-typescripters, just in case; it is c-like, so probably bearable for most.

#

but still;
"on paper" we get to replace the caculate-next-state from cpu-job to a memory-lookup-job.

sounds very charming for the sake of a game, that is practically bound by a cpu core performance.

#

but let's get to the real world of not how it looks, but how it works.

aside from static sets of entities and connections (which are not necessary just boxes-with-items; it is not hard to assume any other one edge-something tbh)
there are also:

  • inner state the one we've tried to mimic
  • outter state aka the context of where the system is located.
#

if the system on our hands is actually stateless, this does imply the inner state is irrelevant.

reasoning about stateless systems is easier, right? the very reason functional stuff is so much fun, for example.
but here, we can not make such an assumption.

everything has the state. every entity, basically, you can say.
if the assembler has the "productivity bar", the bonus-item output depends on what it was a tick ago.
thus we get
magic({entities, connections, previous_inner_state})

#

on the outer state, aka context:
you are bold to assume it to be "static" via

assuming that there are no biters, players doesn't change anything inside and both input and output are sustainable
this is cool actually; but that also means the meme about cache-invalidation fires on any second we DO happen with a capped output, for example.

think about it, how would you listen to everything like power-supply-levels, inputs and outputs "quality", just to make sure it is still OK and we can use out memo cache :>
the cache-invalidation problem is a meme due to "hidden" costs of it and due to the cache-miss costs.

But we are on the magic land anyways, so let's assume there is no context.
Just connections. Just a factorissimo building or smth like that, that is PERFECTLY isolated.

#

that is what your idea looks like at the present point:
no inner state that we care about, just memoize it all away.

#

kinda a nice way to look at things at the macro scale of megabasing you were referring to.

#

there is no definition of a "perfectly correct simulation" btw
there is only a definition of how-it-works on the present version of factorio. how-it-is-done.

for example underground belts + inserters behavior changed not-too-long ago;
and there is a ton of fun details, like what happens if you keep the beacon powered-off most of the time but do power it on once in a while at specific ticks.

#

but as long as we strive to be as close to correct as possible, meet the "scaled recipes"!
the inner state surface is reduced, while still keeping the observed logic in check.
this is basically when you make a single furnace that works like a whole array. neat, right?
that is what you do for the "megabasing" scale when you really want to go that way I guess...

#

and finally
assuming

there is a flow - function that determines the state of the whole system over time
does stand.

there are a few options.

  • that function is static.
    which means we are aware it exists, we know its inputs, we know its' outputs and never ever care how it works inside.
    this is the easiest one by far.
    and by chance, such a black box has a name: assembler.
    there are mods, that do so out of fun or out of curiosity. the one that suggests to train stuff around such super-giga-assemblers as per example.

  • that function is mostly some baked version of a factory block. not known in advance, but still observable in real-time.
    meet the dupeboxes. while one block is simulated "perfectly", you get the rest as a freebie with only minor accuracy loss.

  • that function is supposed to work once some manually-defined-block we assume to behave like a stateless one gets onto some rhythmic loop.
    -> we can't use anything that introduces a state or we accept the loss on such a state
    -> we still have the problem of guessing the cycle size
    ah also
    -> we assume memoization cache invalidation is simple and almost never required

topaz sierra
#

One question. Is following question obvious or complicated (due to float):
We have assembler, which produces 1 item/s. We ensure it always have input and free slots for output
How many items will it produce per second?

rapid karma
#

say it has the productivity bar that only finished each say 10 sec;

#

(say) it would be 1 item per 9 secs, then 2 on the tenth one

#

unless you store that progress bar somehow, or ignore it, or assume you can abstract it with a say "per-minute" or "per-hour" precision

#

well, if you are still motivated to research further,
loss-of-precision for example could be fine; we get the liquids lost all the time in pipes
and multiple state updates, if traced directly to connection changes can be lazily evaluated (though the game does it almost perfectly as it is already)

topaz sierra
# rapid karma this is unsolvable with the "perfect precision"

It's about opinion then. In my opinion it's problem of current engine - it doesn't suit to current ingame wiki (i don't know it's actual name). In other's it may be feature of game
Assuming that current behavior should be preserved - i have no options to prove/refute my assumption, that period isn't too much, other than implement:

I'm going to record flow of some system and check, if my assumption correct
Is there existing mod for that?
Assuming that factorio is continuous and simulation tries to ensure it, i can provide some example of system and calculate period manually

#

Anyway - afaik (writen above), nobody need this optimisation, so i don't think it worths too many time spent on it

rapid karma
#

it is not an optimization, I suppose

#

well

#

looking back on what I written

topaz sierra
rapid karma
#

I think you assume the system to be kinda stateless which is clearly not the case;
(on the second thought it needs not to be "stateless" precisely)
you then suppose that you can replace the cpu-simulated block with a memoized "cycle" of states already-seen, right?

topaz sierra
#

you assume the system to be kinda stateless
Definitely no

#

state[t]=physics(state[t-1], maybe_disturbance[t]), where disturbance if voilation of assumption, that all connections are stable
If cycle appears, it'll continue until distrubance=Some(event)

rapid karma
#

and then, assuming state.length is small enough to be feasible to store them all, state is cached and used instead of the whole simulation stuff

#

or more like the outflow is cached alongside and the box starts to spit the outflow[t], while keeping the state off the chart

topaz sierra
#

you are bold to assume it to be "static" via...
Afaik, callbacks on things like assembler is destroyed aren't costly

we can assume that "the little details are irrelevant" to get https://forums.factorio.com/viewtopic.php?p=484498#p484498. (in the "optimization idea: abstraction" linked above)
Yes, it breaks this idea assuming current behavior should be preserved

behavior changed not-too-long ago
So, behavior can be changed for the sake of ...

there are a few options. ...
Not sure i managed to understand. My problems with English, probably

topaz sierra
rapid karma
#

well still not an optimization sadly

#

close to it though; maybe if if was possible for the inner state update to be directly batched for the whole "area"

#

as it is, it means one have to mutate the inner-state "directly" in-place, without calling physics()

#

a belt that is "completely full" or "completely empty" does sleep.
but this idea seems more like "jit-compiling" an alternative "memoized" version of a physics() for the big-enough isolated-by-"connections" block
(and then using the cheating route, since we are sure nothing would happen the other way than it did before when we recorded it)

topaz sierra
rapid karma
#

nah, I somehow assumed you mean the "set" as in a set of classes (types) of items; as per assemblers with recipe A, assemblers with recipe B etc

#

the "modding" entities are the same as usual ones, they do have the state that one can see ( in runtime stage)

#

but thinking about it that way means one has to copy all the inner states of these ones somehow

#

for the caching purposes

#

or maybe I miss the find-the-cycle idea

#

you never said anything about memozation/caching so far, so it might be not related

topaz sierra
#

but thinking about it that way means one has to copy all the inner states of these ones somehow

It's not necessary. Only side effects need to be cached completely (actually, even them not always)
Only some checkpoints may be cached (every x ticks)

Yes, cycle should be stored somehow. It's possible to find all values of cycle from one state, so it's not necessary to store all states

rapid karma
#

well still have to collect all the inner states into that one state on each test

#

maybe not overly expensive, but feels like the physics_2()

topaz sierra
#

Yes, likely current update function won't work with cache
About alghorithm to find such systems - it's interesting, but complicated and big topic. I have some suggestion on this topic so i think it's possible to implement it, but as i stated earlier - i don't know enough about game implementation, to be sure about it

topaz sierra
#

One thing i forgot to mention. With some assumption (assemblers produce more/less intermediate, than necessary), cycle is stable
I.e. let's assume you have place with several train stations, a lot of assemblers, inserters and belts, which produces advanced circuits from copper, iron and plastic in some cycle. If cycle is broken due to insufficient supply and then you restored supply (built another mining e.g.) - system will return to exactly the same cycle
It means that it's not as "just one thing broken and optimisation gone", it can be easily restored after outer state normalizing

rapid karma
#

consider having cut the iron supply and then restored it here (assuming one gc assembler needs 1.5 wire assemblers here)
even if it does return to the "stable" loop here

image.png
#

but consider the intervals between ore inserters in the smelter array if the ore was not a full belt for some time;
once restored they do get the same outs of a full belt.
but when each-inserter-fires and when-each-prod-bar-gets-full are now "out-of-sync" compared to what it was before

#

the "inner state". it does exist

#

assuming it "either works or wait for full inputs" does seem solve this one though

hollow gale
#

you know, you could probably do all of this with a rate calculator-like thing

#

drag over the stuff and get direct rates, no internal state necessary

topaz sierra
#

I agree with inner state existence
I don't like to consider minor details like floating point behaviour, but i either don't want to see just big assembler
May be my opinion is contradiction by itself though, I'm still not sure

clever moon
#

This is something I considered when thinking about my own game based on factorio