#Project Meteor

179 messages · Page 1 of 1 (latest)

civic quarry
#

Automated harvesting of Blood Magic meteors using AE2 spatial I/O.

Got it almost working, I just need to fix some problems with the control circuitry. Will post a video of it in action when it's finished, just wanted to get the picture here before the deadline.

javaw_NF1hP1uztp.png javaw_DrV5BdMSQ9.png
solemn salmon
#

I am so glad that i can just slap a void miner onto a planet instead of doing this

warm iris
#

motherfu....

#

so you will store one layer in spatial storage and "teleport" that layer to the annihilation panes ?

#

that is smart

#

but stupid

marsh tusk
#

What is the speed on this for cleaning up a whole meteor?

marsh tusk
solemn salmon
warm iris
#

could be fast

#

the annihilation panes are instant and the spatial storage too

#

as long as it not destroy your tps

radiant bear
#

We want video with working process

warm iris
#

this must be so expensive and eat so much power

thin maple
#

we have neutronium power cell for AE so power not matter

#

but this thingy is very epic

#

good job

civic quarry
#

The collection is driven by a timer which is currently set to 1 second (that is per two layers of the meteor). In theory given infinite server power this could go down to 1-2 ticks, but it is getting extremely inconsistent if I set it any lower than it already is.

civic quarry
civic quarry
warm iris
civic quarry
thin maple
#

and they say officials lags xDDD

civic quarry
#

Well don't go building this on the officials :D

thin maple
#

haha yes =D

civic quarry
#

dunno, large spatial ME lags a lot more than I expected it to, for what is essentially a copy/paste of simple blocks.

#

Might try putting an opaque roof over the whole thing so at least it's not doing lighting updates, not sure how much difference would that make.

warm iris
#

I prefer annihilating meteors with frame machines

civic quarry
#

You can't really make good ones in GTNH

warm iris
#

Well you Can

#

But 1.4.7....

#

Unbeatable

civic quarry
#

I mean you can't just make a solid layer of block breakers and sweep that through the meteor; there are no frames that can pass both redstone signal and items through.

#

wtb Create

warm iris
#

You Can do that actually

#

But with block breakers that will just not collect the putput

#

And then use something like ender collectors to collect the items

civic quarry
#

Hmm; I think I tried activating the block breakers from inside the frame but couldn't get that to work; maybe I'm misremembering.

warm iris
#

I love frame machines

#

I Also Saw a frame machines with annihilation panes

civic quarry
#

How would you run channels to them though?

warm iris
#

You could build the whole system on the machines

#

And iirc with cables they are like one block

civic quarry
#

I know, but in this system everything that should get moved needs to be touching a frame. And I don't see any way to run more than 8 channels without having some blocks completely disconnected from frames.

warm iris
#

Ah

#

Hmm

civic quarry
#

(If you want a solid wall of annihilation planes of course)

#

Another thing I was trying was moving dynamism tablets with some area mining pickaxe

warm iris
#

What is dynamism tablet ?

civic quarry
#

TC block that lets you simulate using an item (left or right clicking)

warm iris
#

Ah

civic quarry
#

Hmmmmm I might have an idea how to improve the ME setup. It would make it slower, but potentially a lot less laggy (and therefore faster).

candid rampart
#

🤯

marsh tusk
novel pagoda
deep belfry
#

Don't use annihilation planes, teleport it to an ore drilling rig, and then move the stone to whatever you like.

civic quarry
deep belfry
#

I guess, but it's more efficient.

#

also you can have a meteor being processed while another is being cleared while another is being mined while another is falling.

#

parallelism is useful.

civic quarry
#

If I can gobble up thirty meteors in the time it takes to mine one, no amount of pipelining is going to make it faster.

#

Plus with the current setup I can already kind of do that (I can summon another meteor as soon as the first one is stored in SpME, I don't have to wait for it to be harvested.

#

Assuming perfect TPS and sufficient number of ME cells for storage (or multiple annihilation setups to parallelize that), I am pretty sure I can destroy meteors as fast as the ritual can summon them.

civic quarry
# novel pagoda and what about instead of moving the meteor... move the annihilation planes?

That would be somewhere between far more complex, to unfeasible, to impossible.

Right now I collect slices of the meteor that are 2 blocks thick, store them all in ME cells, and then insert the slices (in arbitrary order) one by one between two layers of annihilation planes. The individual slices/ME cells are really just a pile of resources, if something goes wrong with them no big deal, I can just make more with the press of a button.

Moving the planes instead would mean dealing with:

  • Moving a larger section of blocks (unless you can somehow compress everything to one layer?), which needs exponentially more power.
  • Somehow finding a way to connect the annihilation planes to ore storage. I doubt wireless ME will like being moved; I know transvector dislocators can break it. Haven't tested with quantum rings.
  • Or you could possibly have ME storage with the annihilation setup, and then at the end of the operation "park" it back and retrieve the items, which would likely be slow and definitely need a lot more control circuitry.
  • I don't know if AE that is being moved by spatial updates/connects immediately. As far as I know placing a block in front of an annihilation plane breaks it instantly (either in the same tick or the next). Doing the opposite could probably add several seconds of delay if the ME network needs to update with every step.
  • If anything goes wrong, the entire annihilation layer could end up lost, sliced in half, or embedded into some other machinery horrible-transporter-malfunction-style. Yes all three have happened with meteor slices during development. That is both a non-trivial amount of resources at risk; but more importantly several hours that it took to set it up to be redone.

And here is the "probably impossible" part:

I can move and harvest the meteor in layers two blocks thick, by overlapping spatial ME pylons from two sides of the meteor. Moving the APs would mean having a spatial setup in every layer of blocks. This is a little complicated to explain without playing around with how SpME works for a bit, but due to the needed geometry of the pylons I don't think this is possible.

FWIW I don't think it's a terrible area on its own, and I have briefly considered it as an alternative. But due to all of the above I really don't think it's anywhere near practical; and that is coming from someone using spatial ME to harvest meteors.

The only (very remote) possibility I could think of is having a frame setup that moves the spatial ME gantry one block, then the SpME summons the annihilation planes, which grab one layer of the meteor; the planes are stored in SpME again, and the process repeats with frames moving the gantry one block further. I could see this being at least theoretically possible, if an order of magnitude more complex and slower than the current setup.

#

Something else I have not investigated at all yet is how the spatial ME dimension actually works and is laid out. Would it be somehow possible to either build the annihilation planes in the spatial dimension, or somehow transport them to an appropriate position there, and then teleport the meteor slices to them, without ever having to bring the slices back to the real world?

This could possibly save half the spatial ME operations, which would be a significant improvement in both EU and TPS costs. I have heard that at least some things can be convinced to keep running in the SpME dimension (well if you can make a teleporter with it it is at least habitable), though I don't know to what degree can you actually build working machinery there and store blocks at specific coordinates.

rustic zinc
#

moving the mining heads will likely cause even more server load as you are constantly rebuilding the me network

warm iris
#

Why is there so many pylons ?

#

Like why are the walls of pylons so thicc

civic quarry
# warm iris Like why are the walls of pylons so thicc

Spatial ME has a stat called "efficiency". Efficiency is determined by the number of pylons as a proportion of the surface area of the enclosed space. If the efficiency drops below 100%, the power cost for the spatial operation increases quickly and drastically. With 100% efficiency, each slice costs about 8M EU to transport (pre-nerf numbers). With even 99.5% efficiency, it is something like 20M EU. So if you want this to work at all without the creative cell, you basically need 100% efficiency in every spatial setup. And with a setup as big as this one that means many pylons.

For some more details see this post: https://www.reddit.com/r/feedthebeast/comments/3c2b2r/a_treatise_on_nocheat_smp_ae2_spatial_io/

warm iris
#

Ahaaa

#

Interesting

marsh tusk
#

If you want quick automation, you can also get a robot with a terra shatterer or world breaker 🙂

civic quarry
#

It would take about as much for the robot to just travel the 53 block diameter of the meteor, not to mention mining out the entire volume, as it takes the ME system to process it.

warm iris
#

MBMing the ores and deleting the leftover Stone with filler

#

But that is so boring

#

And manual

midnight haven
civic quarry
#

OK I really need to make a video even in its current broken non-working state, because people aren't getting a good idea of the scale of it.

#

Summon meteor.
Click button.
Meteor gone.

#

(minus waiting for the server to update and your client to sync)

#

As a rough mental math guide, if your thing harvests one block per tick, it will take an hour to harvest the max sized meteor. So take however many blocks you can do per tick and divide one hour by that to get how long it will take you.

inner crane
#

I would like to see a instant meteor slicing and disintegrating

civic quarry
#

Make annihilation planes not stop working when I relog ^^

#

Stupid non-deterministic errors that sometimes work and sometimes randomly unrecoverably break everything really make me lose the desire to play the game right now.

civic quarry
#

Annihilation planes fixed (almost, now ME conduits are not behaving), Project can resume.

random timber
#

Can you not have several large miners used to harvest the meteor all at once?

#

Also, liking that name

#

Planning to make a similar type of setup using something like spammed large miners or a wall of arcane bores on cardinals

#

naturally I'd call mine the Dalamud Machine

rustic zinc
#

goal here seems to be throughput. 20 second vs 20 minute is a big difference

toxic oracle
#

Hey instead of using me storage why don’t you use drawers

#

And max upgrade then with a void upgrade on ones your ok with voiding

#

And for that moving problem cant you just setup a modified perimeter miner that just pushes the anhelation frames back and forth?

#

And have it all connect to the me system at a top cable that stays at the top?

#

And for that problem with channels how big is the max meteor

#

Cause if it’s 16 by 16 then you could just do a whole cable thing going all around it

#

Like you have a for every line you could have 8 normal cables going along it all together but not connected

midnight haven
toxic oracle
#

Laggier then a whole thing with ae2 pylons?

random timber
#

Yeah, the reason I was thinking of EV miner spam was for less lag

toxic oracle
#

Yeah but you also have to think of item storage and transport

#

And power

#

And drilling fluid

civic quarry
#

~4 minutes real time per max-size meteor. A majority of the time is still server delay; going by in-game time it should be about 15 seconds.

rustic zinc
#

that's actually a bit concerning. I don't think moving that amount of blocks would lag that much. are you running of a potato computer or something?

civic quarry
#

Not an issue of my computer, this is on a MP server.

rustic zinc
#

it takes as much time to set full chunk worth of block via worldedit in my game as your machine takes one slice

#

will it work faster if you don't stand in its render range?

civic quarry
#

Well, I'm not exactly sure what extra work SpME does. I tried looking into the code a bit but I don't fully understand everything. And I think the chance of anyone actually caring enough to optimize it decently is really remote.

#

How could it not rendering on my computer help the server process it faster?

#

I mean I can try but I really don't see how that would change anything.

rustic zinc
#

if you don't stand near it, the server will not need to send you the updated chunks

civic quarry
#

Right I see

#

let me find some anchors and try

rustic zinc
#

minecraft is dumb and doesn't do sync in off thread. if too many stuff changes at the same time it can lag just due to you watching it

civic quarry
#

About 3:30 with me out of range. So really about the same.

rustic zinc
#

welp. i guess that's the limit of spatial io then

placid kayak
civic quarry
placid kayak
marsh tusk
#

I think the pylon part is fine. I assume it’s the planes with ores going directly into storage busses/storage creating a delay in calculation from ae2.
Is it currently connected on a sub and going straight into your main?

civic quarry
#

Nah, it's both. Capturing one slice and placing & breaking it take roughly the same amount of real time.

Everything in the setup is fully self-contained in terms of ME networks. Each slice is its own network (27 total), annihilation planes are divided into 14 networks which all feed into each other via storage bus->interface; until they eventually all feed into one more network that only has an ME drive and a terminal.

I might try isolating it even more (for example, giving each of the 14 annihilation networks its own storage); but I'm not expecting miracles.

Moreover, this is right now a bit of a showstopper: This is what happens when your meteor contains BartWorks ores.
https://i.imgur.com/XMaqeYD.png

Imgur
#

After some discussions with other people I'm right now working on a completely different design using ender quarries, which according to preliminary tests will take about 2-3 minutes in-game time per meteor; but that should hopefully also be 2-3 minutes real time, so even though it's slower it's actually faster :3 (And doesn't make the server unplayable while running.)

fringe hare
civic quarry
#

Abdiel_Kavash is my main account; I had some issues with it when I started on the server so I used my alt. By the time I fixed those issues I was too far in progression to switch back so I stuck with it.

random timber
novel pagoda
solemn salmon
random timber
#

Yeah, I'm wondering how you can setup that, I'm considering using EV miners so I can get the pre-macerated ores

#

But not sure how to have them deal with regenerating ores

sage salmon
novel pagoda
sage salmon
#

my point was moreso that the meteor only has collision on the center column, so you only need to move one block out of they way, not the entire ritual shrug

novel pagoda
#

:/

#

So you can only remove that block when the mneteorite lands, so is a nonsense

sage salmon
#

you can remove it after starting the ritual and the meteor will still come down

#

unless the compacted ritual stone works differently for literally no reason whatsoever (I was too lazy to build the full thing in creative beesleep)

novel pagoda
sage salmon
novel pagoda
#

But I think that no, if you break any part of a ritual the ritual will cancel

novel pagoda
sage salmon
#

nah, just tested with the full ritual using a projectred diamond block breaker and it works fine

novel pagoda
#

well

#

easyer

#

😄

#

Like for example the binding one when is doing the lighting thing, if you break a block the item is consumed but not transformed :3

sage salmon
#

luv me some "automation"

image.png
novel pagoda
#

;/

#

I would be using annihilation and formation planes xd

sage salmon
#

I cannot be bothered to set up AE2 stuff for a creative test

sage salmon
random timber
deep belfry
civic quarry
#

Wait if moving the master ritual stone works to drop the meteor under it then that is such GalaxyBrain

#

I had a """working""" setup that moved the ritual stone from under the meteor to let the quarry work, but it was insanely complex (hint: spatial ME can not move the master ritual stone. Good luck figuring out an alternative.)

If I can just poof the ritual stone before the meteor lands this will be one button.

#

Hmm, interesting. So the ritual doesn't actually even check for a non-air block or anything like that. It actually spawns an entity at y=257 that literally falls down until it hits something, then explodes.

#

Corollary, building the ritual at build limit makes it spawn faster PepeGTNH

novel pagoda
#

@civic quarry now you can build the ritual at max height, spawn 3 meteors and then mine the 3 together

civic quarry
#

lmao it works

#

And yes that is a possibility

#

Just a minor issue; you don't want to spawn them literally on top of each other, because then they would clip into each other and lose like 40% of the volume. You want to stagger them a few blocks and drop them to different target heights so they don't overlap.

#

But yeah, the max size meteor is 53 blocks tall, you can fit 4 on top of each other + some space for machinery easily.

civic quarry
#

Still need to figure out how the timing on EQs works; it seems like it is some constant amount of time (sub-1 tick) per block mined in the entire area, regardless of if it's air or an actual block. Things I'm not sure about:

  • Does this mean that a quarry placed higher will take more time to clear the entire area than a quarry placed lower? This would make the stacked setup inefficient if I only want to do one meteor.

  • Does the hardness of the block mined make any difference? It seems by just looking at it that mining a column of stone takes a bit longer than going through a column of air. If this is the case, the timing would be inconsistent for different meteors, as they contain different amounts of blocks.

Unfortunately I wasn't able to find source code for the ExU version that is in GTNH, so I'll have to do some in-world testing.

novel pagoda
short edge
severe marsh
#

Do you mind in sharing your setup? I would like to put it on my todo list 🙂
Maybe also with a part list and which Tier should I be to make it worth for usage at last.

lost heath
#

@civic quarry

Looking into this idea since I'm in a run considering using it (but with neutronium energy cell if at all, I think the energy is a bit too much for the effect). I want to do this because once you have neutronium energy cell this is a zero power alternative to void miner spam and you can be selective enough about the ore to literally pattern your ore processing (crazy, right!?) which would then let you use level maintainers to ensure raw material counts using a combination of proxy items and proxy patterns. I also think there are some ways to improve it (both by modifying code and by changing the in world design a bit) which could make this method virtually instant at mining meteors and probably vastly surpass void miner throughput.

First thing I did, I made a local dedicated server instance on a new map and did a benchmark spark profile (https://spark.lucko.me/IdwNMplcvr) on a 127x100x127 region, going to bedrock, taking a lot of earth with it. Set the cell to get extracted into the IO port and a timer to activate the IO port once a second, alternating between putting the blocks back and removing them. There is a lot of time just spent on file IO due to loading and unloading chunks in spatial dimensions, which is writing NBT to file. Because chunks get unloaded and reloaded at rapid fire. Also a lot of time spent on lighting updates as anticipated.
Spatial IO console logs said:
Block Copy Time: 89778556
Update Time: 60511147

#

Second test (https://spark.lucko.me/V2j8VwGvXZ), I add a big flat wall of blocks above to stop most of the lighting updates (some still happen due to lava and junk) and chunkloaded all the chunks inside tthe spatial dimension by travelling there, server utilities claiming, and leaving using a wr-cbe remote. The MSPT spike on moving these blocks, approximately 1064960 of which are non-air, went from 2690 ms (2.7 seconds) to 239 ms (0.24 seconds)! Looking at the profiler results the remaining performance cost seems to be ... tick updates for liquids and the few light sources that weren't spatial IO'd, including a couple hives and lava pockets, because the test region I picked had some underground pockets of lava.
Block Copy Time: 91317994
Update Time: 32200342
Block copy time is the same more or less, update time is cut in about half. Although I don't think this log is capturing all the going on and I don't know the unit of time used here.

Third test (https://spark.lucko.me/RmypMwveI9), a 50x50x50 (125000 blocks total) square of gregtech iron ore (a little over the worst case of these meteors to my understanding), again chunkloaded and light covered. Looks like the ores are not very smart when it comes to doing their lighting updates on a block update, and that is the main source of delay. It took longer doing this small cross section than it did the 127x100x127 area. A
Block Copy Time: 7509957
Update Time: 12109869

#

Also, my client was getting kicked from the server if I stood in render distance of the IO swap happening. My client got humongous stack overflow errors due to recursive neighborhood updates which is likely the source of all the performance crippling server tps moving these ores to begin with. This is a very fixable issue for the IO ports specifically, and although less present if you only move smaller slices at a time, it is still probably the lion's share of tps usage. One silver lining is if you move very few contiguous blocks at a time (think some clever design using single columns to diagonally slice the meteor) you can almost entirely eliminate this performance issue without a code fix.

[02:38:22] [Client thread/ERROR] [FML]: There was a critical exception handling a packet on channel GregTech
java.lang.StackOverflowError
~[GT_Block_Ores_Abstract.class:?]
    at RFB-Launch//net.minecraft.world.World.func_147453_f(World.java:3875) ~[ahb.class:?]
    at RFB-Launch//net.minecraft.world.World.func_147455_a(World.java:2603) ~[ahb.class:?]
    at RFB-Launch//net.minecraft.world.chunk.Chunk.func_150806_e(Chunk.java:851) ~[apx.class:?]
    at RFB-Launch//net.minecraft.world.World.func_147438_o(World.java:2540) ~[ahb.class:?]
    at RFB-Launch//gregtech.common.blocks.GT_Block_Ores_Abstract.onNeighborChange(GT_Block_Ores_Abstract.java:110) ~[GT_Block_Ores_Abstract.class:?]
    at RFB-Launch//net.minecraft.world.World.func_147453_f(World.java:3875) ~[ahb.class:?]
Stopped watching the game log because the log length surpassed 100000 lines.
You may have to fix your mods because the game is still logging to files and likely wasting harddrive space at an alarming rate!```
#

It would be fairly easy to add a feature to IO ports to disable block updates / neighbor change notifications on affected blocks similar to worldedit, it would be just adding a line to not run the lines which explicitly do so here: https://github.com/GTNewHorizons/Applied-Energistics-2-Unofficial/blob/5a8393bd573869c3ba6bea8c3b7b9af6f6b67953/src/main/java/appeng/spatial/StorageHelper.java#L211
Worth noting traverseEdges only calls updates on the very edges of the bounding box around the spatial IO area, so if you make sure none of the blocks are adjacent to the edges that part of the code won't activate.

The other silver lining to this is moving ae2 multiparts with spatial io is very performant. Responding to your point about using annihilation planes: You can put spatial IO inside spatial IO. Yeah, so you can move the meteor into cell world 1, and while that is gradually annihilatted into cell world 2, summon a second meteor over the same area. Or, let's say we work without an IO port quick fix, we just move two "sandwich" layers of annihilation planes up and down the top and bottom of the meteor, leaving it in-place, to gobble it up. In my testing network recalculation does happen moving an internal network in and out of spatial IO network recalculation is still quite fast.

Sticking with the original strategy of moving the meteor elsewhere, you can cut also the performance cost in almost half again by being smart about the spatial IO overlap. Let's say you have a 20x20x20 meteor and want to capture it in 20x2x20 slices. What you can do is have five spatial IOs take five slices spaced out to leave gaps in between, then a sixth spatial IO take all the five remaining slices. Then you can have a second area with ten layers of annihilation planes in total, send spatial cell #6 there and it's all eaten at once, then have for each of the five sets of two layers, move spatial cells #1-#5 to each of the corresponding five layers in the same overlapping spatial region.

#

There are also a lot of log messages that look like this.

[Server thread/DEBUG] [FML/]: The world 71f7ac2e (World) may have leaked: seen 15 times.
[Server thread/DEBUG] [FML/]: The world acea22e (World) may have leaked: first encounter (5 occurences).
[Server thread/DEBUG] [FML/]: The world 5b436922 (World) may have leaked: first encounter (5 occurences).
[Server thread/DEBUG] [FML/]: The world 3369ab17 (World) may have leaked: seen 15 times.
[Server thread/DEBUG] [FML/]: The world 5fdca031 (World) may have leaked: seen 15 times.

There is a very slow memory leak here, I know this because I OOM crashed a beefy server having many IO ports run over millions of cycles. Could be due to net.minecraftforge.common.DimensionManager not expecting API calls the way spatial IO does it. I don't have forge source code in front of me so I can't confirm this right now.