is it impossible to make my game?

[ripe bridge]

I am making a recreation of minecraft where players can create their own world and other players can join that world too. I am currently making the world generation part of it. I have resized my baseplate to be 700x700 and each block is 4x4x4. Meaning I will generate 30,625 blocks per y axis. I am planning for the world to be generated 10 blocks high, meaning the total blocks would be 306,250. This is an example of a block's data:

"XXXX.YYY.ZZZZ" = "NNN"

XXXX.YYY.ZZZZ represents the block's coordinates in the world and also its id in the database

NNN represents the block id, such as if it is a dirt, stone, etc

which would be 21 bytes. Since each data store key value has a limit of 4.1mb, im planning on setting each chunk to be 38x38x120 blocks, totaling 3.8mb for each chunk if the chunk was filled up

now, the amount of chunks generated will be 25. BUT:

1. there is a limit on reading and writing to data store: 25mb/minute and 4mb/minute

2. to solve this issue, i have decided to store loaded worlds into memory store so that it can be quickly accessed if the world is popular and many people want to join it, but memory stores has a limit of [64KB + 1.2KB * number of players] for the whole experience which is not even enough for 1 chunk to be saved

so now im at a wall. i dont see any solution to get over this storage issue. i could shrink my world further down, but at that point its not even fun to play anymore. imagine playing minecraft but the max building limit is 10x10x20 blocks

[fair lantern]

interesting question, i prob dont have best answer but what id try:

Having map be generated by SEED, generate it each time he loads world so its always the same and only save blocks changed

and maybe force them to private servers?

prob bit cooked tho

[ripe bridge]

does private server count as a different experience?

[fair lantern]

i believe so

i mean

maybe

id have to find out more

[ripe bridge]

because on the creator hub dashboard, it shows a 101mb limit for data store but it only scales by 1 mb for every new user joined

[fair lantern]

yea they made it be 100mb + 1 per person

[ripe bridge]

ur seed suggestion is very helpful

[fair lantern]

which can be bad now as i think but uh

aint this per database

cant u have few

[ripe bridge]

thats shared accross all stores and the whole experience right?

[fair lantern]

not sure about this one

id have to read

[ripe bridge]

same i read thru the docs but couldnt quite figure it out

[fair lantern]

yeaaa the 10mb is per whole game

that can be bad

i do believe u can purchase more (or will be able to in future) but i dont think thats the play

[little verge]

nahh I bet you could deflate it to a maximum of 60kb

[ripe bridge]

per chunk?

[little verge]

nah I did some math and pulled random stats online it would be like 1-2 kb per chunk

[ripe bridge]

damn

[little verge]

you'll have to follow up with recent strategies for data compression tho

[fair lantern]

alright so i got too lazy to read so i got chat gpt
apparently 10mb is only for open cloud (via http)

[ripe bridge]

how did u manage to get those numbers

[fair lantern]

let me actuall find it in docks

[ripe bridge]

how did u manage to compress it to 1-2kb per chunk

[little verge]

https://en.wikipedia.org/wiki/Run-length_encoding

https://stackoverflow.com/questions/62531554/recreating-zlib-from-scratch

do them on top of each other

[fair lantern]

also one of the methods for compression u might be able to use (idk if its helpful) could be just splitting each chunk into lets say 16 smaller ones, saving middle of it (or corner?) and having blocks in that place be relative to that, should save on some big numbers (no idea if that much tho)

[little verge]

its posible cause others did it

[ripe bridge]

r u saying to not save the middle of a chunk, and save its borders and use some sort of algorithm to generate the middle section?

[fair lantern]

not really

basically

if u do normally, thats like middle is 0,0,0

[ripe bridge]

can u provide an example, cus ion really know how to apply this

[fair lantern]

so everything is how far off 0,0,0

[little verge]

yeah I'm thinking and reading

[fair lantern]

but if u maybe split world into smaller chunks, each chunk will have their own middle

and blocks in that chunk are compared to the middle of it

[little verge]

oh but the math I can share for noww...

[fair lantern]

so instead of 1000,0,1000
u might be able to have 50,50

[ripe bridge]

so r u saying to group the same kind of blocks together

[little verge]

38 x 38 x 120 = 173200 about which raw ofc is 169.2 KB (dookie) run length is 2B per run and if the average run is like 100 Bytes then it can get down to as low as 3.4KB. size = (173200 / 100) * 2B then you use the Deflate which halves it so like 1.7KB

[fair lantern]

how would run lenght be 2b

[little verge]

ofc it means ur map shouldn't be too complicated and support those "runs"

[fair lantern]

am i slow or missing smth

[little verge]

the data byte and the .

or like the data length and ID stuff

If there are like a sequence of copies it can reduce

[1, 1,1, 1,1, 2, 2,2]
(5, 1), (3, 2)

so the run is 2 byte

simple strat 🙂

[ripe bridge]

intriguing

so is this converting it to buffer and shortening the binaries because im lost

[ripe bridge]

what do these numbers represent? the position or wut

[little verge]

yeah

the block type and the array index would be position

a standard 3d array proll

or I mean 2d kinda

whatever.

[ripe bridge]

[little verge]

yeah and with the int8Array or whatever you can support already whatever that is

like 64 block?

I'm too lazy for binary count

you don't even need the like Zip compression stuff possibly.

[ripe bridge]

but if its converted like this, would that mean i will need to fill in 0 in positions that do not have 3 characters? ion fully understand but i assume its ordered like [block id, x,y,z, blockid, x,y,z]

[1, 0,0,0, 1, 111,20,2]

if its compressed into

(1, 1)(3, 0)(4, 1)(1, 2)(1, 0)(1, 2)
for (4, 1) would mean 4 number 1s but how would i know, when decoding it, if it is the block id 1 at position x 111 or block id 11 at position 11?

[little verge]

like just use the index of the data in the array, so index 1 of the array would represent the first block so maybe the bottom right corner of the chunk

index 2 would follow to the left of it

and so on, with 36 x 36th index representing the next layer

[ripe bridge]

but thats assuming the user places the blocks in that order, if the user decided to place a block in the middle of the chunk index 1 would be that middle no?

[stiff ocean]

** You are now Level 5! **

[little verge]

yeah you'll have a formula that will give you the index based on the cell the player wants to place it in

[ripe bridge]

[little verge]

where you divide and add some numbers that will return the index in the list, do you want me to go in a lil more detail

cause like i'm buns at explaining stuff 💀 but like this is a cool topic

[ripe bridge]

yea

am i supposed to fill in empty spaces with air blocks, and when a user places a block at an "empty" slot it will replace the index of the air block

[little verge]

yes you'll know what cell the user places that block and you will go to the data structure that holds that chunck, go to that index, and replace the current value with the new block

wait but I see what you mean, are you asking how you will patch the data structure in the middle of a run?

[ripe bridge]

i see

[ripe bridge]

no u answered my question just now, but im still a bit lost because what data structure should i use

like should i use a hashmap and then convert it to the things u linked about

[little verge]

well first off you'll have a serialized and deserialized values, serialized is the compact version for storing and desericalized it what you will use prolly, so basically the loaded chunks

[stiff ocean]

** You are now Level 8! **

[ripe bridge]

i see, like encoding and decoding

[little verge]

for the serialized version it would be a very long string of what would look like random characters or a lil less technical "numbers"

the deserialized prolly would show all the runs in a large table { {len = 99.., blockid = ...}, ...}

then from there you can figure a function for not only figuring what run ur affected cell is in but then a patching function that will "fix" your run

I'm sure minecraft goes through a similar process so you can look it up or find some research papers on other formats, this is a very simple spacial array you could prolly get more creative but idk.

[ripe bridge]

i see, i get the idea now, thanks for ur advice

[little verge]

cause like as ur runs decrease the data starts getting to like for example: avg 10 blocks per run is 33.9KB without Deflate and it is a bit much.

yeah goodluck thats super intersting project

[ripe bridge]

ive been trying to figure out a formula for this, and i have found a way to do this but im not sure if it matches with ur advice:

 local Chunk1 = {}
local X_AXIS_BLOCKS = 4
local Z_AXIS_BLOCKS = 4
local Y_AXIS_BLOCKS = 4

for X = 0, X_AXIS_BLOCKS - 1 do
    for Y = 0, Y_AXIS_BLOCKS - 1 do
        for Z = 0, Z_AXIS_BLOCKS - 1 do
            local BlockId = "1"
            
            Chunk1[X][Y][Z] = BlockId
        end
    end
end

im using a 3d arrray to store the xyz position, i havent figured out a way to "deflate" it from the links u sent me but can u tell me if this was what u had in mind?

ok nvm i think u meant for it to be like this instead:

local Chunk1 = {
    1, 0, 0, 1,
    1, 0, 0, 1,
    1, 0, 0, 1,
    1, 0, 0, 1
}

--(0,0,0) (1,0,0) (0,0,1) (1,0,1)
--(0,1,0) (1,1,0) (0,1,1) (1,1,1)
--(0,2,0) (1,2,0) (0,2,1) (1,2,1)
--(0,3,0) (1,3,0) (0,3,1) (1,3,1)

[little verge]

ur 3d array is looking perfect for our eyes and it seems you understand it. To fully utalize the Runs and later the Deflate, Zip stuff you'll just need to flatten it to a 1D array rather with just the bits representing the run

but thats the "serialized" version, then the unserialized is prolly ur loaded chucks with all the runs = {
{ length = 9, blocktype = 1 }, ...
}

there are stratigies you could prolly use for patching(searching what run to change) and formatting ur map. for examply you could use some sort of binary search for the patching, that would be my first thought/

[ripe bridge]

yeah, i have found the formula to find the xyz position of a block based on its index in a 1d array:

--Example result: (4x4x4 chunk)
local Chunk1 = {
    1, 0, 0, 1,
    1, 0, 0, 1,
    1, 0, 0, 1,
    1, 0, 0, 1
}

--Example position result: (4x4x4 chunk)
--(0,0,0) (1,0,0) (0,0,1) (1,0,1)
--(0,1,0) (1,1,0) (0,1,1) (1,1,1)
--(0,2,0) (1,2,0) (0,2,1) (1,2,1)
--(0,3,0) (1,3,0) (0,3,1) (1,3,1)

--Setting the max amount of blocks on each axis
local Chunk = {
  max_y = 4,
  max_x = 2,
  max_z = 2
}

local function GetPosition (BlockIndex : number)
    local RealIndex = BlockIndex - 1
    local X = RealIndex % Chunk.max_x
    local Y = math.floor(RealIndex / (Chunk.max_x * Chunk.max_z))
    local Z = math.floor(RealIndex / Chunk.max_x) % Chunk.max_z
    
    print(X,Y,Z)
end

GetPosition(1) --0,0,0
GetPosition(2) --1,0,0
GetPosition(3) --0,0,1

and after generating the blocks and storing the data, i will compress it using RLE:

local Chunk1 = {
    0, 0, 0, 0,
    0, 0, 0, 0,
    0, 0, 0, 0, --"0" representing air so this represents the sky
    1, 1, 1, 1,
    1, 1, 1, 1,
    1, 1, 1, 1,
    1, 1, 1, 1 --"1" represents dirt so this represents the ground
}

--Compressed version:
local Chunk1 = {
    {12, 0}, {16, 1}
}

[little verge]

lol yeah thats sick

ofc I didn't like check ur formula but it looks familiar, nice

show pic of it generating a chunk when you get there

[ripe bridge]

idk if ur still reading this but if u are, this is the result of using ur suggested compression methods:

before: 27mb per world
now: 0.063701mb per world

making it so that i can allow players to create over 62 million worlds! thank u very much!

[little verge]

lol wow very very impressive work!

I will definitely be implementing some of these strategies now I know how effective it is

[ripe bridge]

Correction, its actually 100mb / 0.063701mb = 1,569,834,068 worlds, and for every 1 new user my data store storage gets increased by 1mb so i can create another more 15 million worlds

[little verge]

lol that is absolutly crazy

tbh I didn't really know how powerful the application of it would be