Have i thougth wrong here? Data & DB

[rancid stag]

Hey i need some advice in my logical thinking here 😂

i want to create a minecraft mod(so its java and normaly the data get stored in the world files), which sounds kinda boring yeah, but what i want to do is gather tons of player activity data (on serverside only).
that can range from movement data (all few seconds), different triggered events(which will start with a few and maybe get up to a ton of different events) for lets say an example server of 30 daily regular players. There will be probably be soon like 1000 data entrys per minute sended.

I had a argumentation with a well versed MC modder, and they said i should save my data in the world.
Which is kinda not making sense for on this mass(that they dont know).

My plan for this data, a PY program will run on a schedule and use this data to calculate player behaviour, based on that i want to let it trigger different functions/methods via a opend port on the server.
So ingame logics = the mod/s, data analysis and triggering stuff = PY and data saving my plan was to use MySQL (or MariaDB) to make this accesiable via different programs (even when the java server is offline or what i think maybe gets overloaded with data if i would save it there)

Is there any thinking error i did? Problem -> tons of data savings/operations/read outs -> using a DB -> MySQL for security reasons & data mass -> creating API/Interface in Java mod & PY & maybe some other applications later.

[open shore]

I guess the correct term would be a plugin (instead of mod) if it's running exclusively on the server

How much data are you talking about when you say "tons of data"?

Your idea of not saving the data in the world sounds reasonable, I don't see why you'd need to run a separate database for that.

Are there any big reasons to have the data processing done in Python? Does the data processing really matter when the server is offline?

[rancid stag]

Well normaly plugin would be correct.
But i want to create different modules, a few of these will use player activity data, which also some cant be "get" on serverside only, so i need to tinker a lil around there and also send stuff from client only to server (in some specific event cases).

Reason to use PY is because im learning that language now and i want to use it anyhow productive too, i mean i learn python only for data analyses reasons now, so why not trying to use it then ( i know some other languages like ruby are also well served in that, but well my course ive got payed from the country will be py xD)
Data anlyses when the game server is offline -> based on analyses i want to create smth. like a querry for the game server to take input out from to do stuff. -> I want to kinda remove some known problems minecraft has on itself with data, which is why this will be my attempt to "squish some bugs" 😄 which could happen or produce lag/glitches based on how much the gameserver is impacted at that moment.
Also the analyses should run in timed intervals, where i cant gurantee the server runs.
Or maybe later, multiple server run and take their inputs from the database (but thats far future stuff)

[open shore]

Those actions seem to be in real-time, right?

[rancid stag]

yes and no.

It does not have to happen in this specific moment.

[open shore]

Right, then you could throw them in a database, I guess that makes sense

[rancid stag]

like one example:
i want to create logical spawner system in the game,
which has a start coordinate and a range and a "cache" of entitys.
But based on player actions, like beeing in the spawner range, killing inside or outside of the spawner range a entity, the middle point coordinate should change and be saved as data also.
but this should not happen after every action, but after analysing the data all few mins(or hours probably more accurate)

[open shore]

The analysis code will run on another server separate from the actual Minecraft server, right?

[rancid stag]

yip probably, or when its a better system on the same. Based on specs.

for start it will run on the same engine.

but like "killing" an entity which belings to the spawner or spawning one out of the cache are real time changed the java server will do.

[open shore]

If it's on the same server then you probably don't have to worry about the database stuff lol

[rancid stag]

what do you mean? 😄

[open shore]

If both analysis and Minecraft are running on the same machine you don't need a database

(takes away your ability to run analysis while the server is down but I don't see a reason why it should be down)

[rancid stag]

wait... how do i store then the data?
And i want to filter for time ranges also.
(as said could happen that it will be multiserver Data too)

[open shore]

I'd just drop it into SQLite, one (or multiple) table(s) for raw data entries and then one for the processing jobs

Java dumps stuff into the db
Python takes it out, does processing, updates the job and pushes back new events to another table
Java plugin reads from the events table and performs the events as needed

Poor man's job queue

[rancid stag]

Maintaince & Updates, which could happen to run in offline mode for 1-3 hours

[open shore]

But is that actual server downtime or just the Minecraft executable not running

[rancid stag]

That i cant say at this state of the project for certain

[open shore]

If it's actual server downtime you have a lot of other issues to worry about lmao

[rancid stag]

but probably most times only the java exe

[open shore]

In that case you could use any other database of your choice and build the same stuff

(but at that point you might as well just use a proper message queue etc.)

[rancid stag]

yeah i also need to keep in mind, that i need to split the mods a lil, so they should be able to work also for people who play solo, but with other analyses (because that i want to use as a key feature, so people want to join the MP server :D)

[open shore]

Architecture options kind of depend on how professional / big you want the system to be

[rancid stag]

true. thanks for the exchange of infos not rob 😉

[rancid stag]

Another question,
i got "pointed" to check out NoSQL options, like Cassandra /Big Table / Wide-column

But anyhow, from what i read, im not feeling so well with them.
Reasons:

• Unstructured data
• Different languages or none to prepare
• and some things i maybe have not understood correct 😄

I normaly know what kind of data i want to store, the ammount i can guess good and whats coming later too.
Also i allready know before i do a project, what kind of things i want to store maybe later, or i know how to change the data + tables.
What are you thinking about SQL vs NoSQL here?

[open shore]

I don't think it matters much

Cassandra is definitely overkill, you're writing like what 1k events per minute?

[rancid stag]

well 1k is just a small number, to have starters. i need to test what the MC server can handle without making problems and need to make what i save as data configurable.

[open shore]

You'd be surprised how performant databases can be 😛

[unique cargo]

Many SQL DBs have options for unstructured data too, like PostGres where you can throw in JSON for example

[rancid stag]

well TBH 1k requests per minute would be my biggest so far 😄
before i ahd maybe like 1000 in a week for my companys tool xD

[unique cargo]

1000 in a week any DB can handle

[open shore]

I'd guess my Raspberry Pi running Postgres can handle 1k writes per minute

(especially if you batch or something)

[rancid stag]

That was obvious 😂

But imagine you are a lil programmer puppy that started a year ago coding and people dont explain you about "what is much" for databases, there you start thinking thats maybe normal with 1k per week 😂