#Error: ER_DATA_TOO_LONG: Data too long for column 'json' at row 1

1 messages · Page 1 of 1 (latest)

regal mural
#

The error is pretty self explanatory. You have too much data for one id, the database refuses it

sullen sparrow
#

?

#

I need to store an exorbitant amount of data

regal mural
sullen sparrow
#

:/

#

I would like to store millions of data but unfortunately then I will have to use a local database system

regal mural
#

I don't think you understand correctly. You can do it. But not in one id... also if you are using sqlite for this amount of data. It's just a bad idea. I would recommend mysql at least.

Databases are not softwares where you can just put data randomly in anyway you want. They follow stricts patterns for it to be more optimized and within the limits. You will have to do some preprocessing if you want to place it in a database

sullen sparrow
#

I understand, I would need to do something very big for a not so big project

#

I think I'm going to use the simplified way with a local database, which won't change my use

regal mural
#

Your choice but I am just letting you know that sqlite is a local database. That is probably the one you are using with quick.db. so unless you choose something else which is not a real database like a random json file or something you probably won't be able either.

#

Also quick tip, if that database will be used for a project which needs to be accessed fast... then paying the price of preprocessing the data to actually put it in a fast and reliable database will totally be worth it

sullen sparrow
#

I use my own database system, which was developed to access .json files quickly

#

Do you think this is coherent for a project?

#

that will use a lot of data

regal mural
#

Even if you say accessing json files quickly. Having a huge json file will hurt performances no doubt. If you split it in multiple tiny json files then I wouldn't say it's a bad idea really.
There is reasons why databases software exists, json isn't a database, it's a format

sullen sparrow
#

your opinion is to separate into several tiny .json files

regal mural
#

If you want to use json files, yes. Since you will be accessing those individually it will at least have more performances. Json needs to be convert from start to finish before being able to actually use it. So having a huge file will be bad as it needs to parse the whole thing every time. Now if you say you will cache it in your program it's less of a problem but I do sure hope you have a huge amount of ram if your data is as big as you said it was.

One thing to note with json is that you write to a filesystem. Nodejs normal fs module can corrupt files when it is heavily accessed even more if it's threaded. You can search online and you will find a lot of people having problems with json "databases" since nodejs fs module can corrupt them (this means a total loss of data)

#

So if you would ask me for advice, I would totally not use json. Database softwares have implemented ways to make it performant, reliable, less often corruptable (this depends on the database, for example sqlite which is a simple local database on 1 file, even that can corrupt) what you lose for all those good points is that you need to follow their pattern and limits

sullen sparrow
#

right thank you very much

regal mural
#

You're welcome, if you need anything more feel free to mention me, even for advice or choice doesn't need to be a problem with quick.db

#

@sullen sparrow just to maybe point you in a better direction. If your data is already json and that you do search for a database system. I would recommend a nosql one. Like mongodb. Those are more flexible and mongodb uses bson which is essentially binary json so it goes well with json stuff normally

sullen sparrow
#

and another thing, do you know how to change the icon of a nodejs application

regal mural
sullen sparrow
#

to transform into .exe

#

but it has this node icon, when I use an application to change the icon the file stops working

regal mural
# sullen sparrow but it has this node icon, when I use an application to change the icon the file...

Yeah, I looked it up and it doesn't mention anything about changing the icon. It doesn't seem feasible at least with pkg itself with only an option.
There is a discussion about it on their github repo https://github.com/vercel/pkg/issues/91
There are multiple ways people have been able to do it but some are really old so they may not work or even they could all maybe not work if there was an update which changed something important

sullen sparrow
#

Have you ever changed the icon of a node .exe ever?

#

Do you have any other application suggestions to turn into .exe

regal mural
#

I did. I haven't use pkg before though. I mostly did it with electron which if you don't use electron then I really don't recommend it just to change the icon. I searched a bit and found nexe which does support changing icons,

sullen sparrow
regal mural
sullen sparrow
# 
{
    "data1": [
        "value1"
    ],
    "data2": [
        "value1",
        "value2",
        "value3"
    ],
    "data3": [
        "value1",
        "value2"
    ]
}```
#

you know how i can make a console log

#

ranking

#

data2: 3
data3: 2
data1: 1

#

make a rank by numbers of values

regal mural
sullen sparrow
regal mural
#

Mmm, i am not sure how you got this but this is how I would done it

sullen sparrow
#

but idk now how i can make a rank

regal mural
# 
Object.entries(data).map([k, v] => { id: k, value: v.length }).sort((a, b) => a.value - b.value);
sullen sparrow
#

can you give me ready code based on my

regal mural
#

It pretty much is

sullen sparrow
regal mural
#

All you have to change is data

#

It should be database for you

sullen sparrow
#

=> have a error

regal mural
#

Forgot brackets ([k, v])

sullen sparrow
#

solved

regal mural
#

here, I also forgot some brackets somewhere else (applogies for this, I was writing it on my phone)

Object.entries(database).map(([k, v]) => ({ id: k, value: v.length })).sort((a, b) => b.value - a.value)```
I also tested this one
sullen sparrow
regal mural
sullen sparrow
#

oo

#

ok

# 
console.log(Object.entries(arr).map(([k, v]) => ({ id: k, value: v.length })).sort((a, b) => a.value - b.value))```
regal mural
sullen sparrow
sullen sparrow
regal mural
#

you are asking me wrong

#

I didn't do it wrong

sullen sparrow
# 
{
    "data1": [
        "random"
    ],
    "data2": [
        "random",
        "random",
        "random"
    ],
    "data3": [
        "random",
        "random"
    ],
    "data4": [
        "random",
        "random"
    ]
}```
regal mural
#

you wanted the code for thjis ^^^

sullen sparrow
#

i need a rank by datas length

regal mural
#

yes

#

look

#

boom

#

it works

#

you are just not giving it the right structure at all that you gave me at the beginning

#

those are not the same structure

#

you want me to do it for the first structure you gave me or the other one ??

sullen sparrow
#

this is the structure

regal mural
#

then it's different... so I wasn't wrong, you gave me the wrong structure

sullen sparrow
#

and i need make a ID rank by DATA LENGTH

sullen sparrow
regal mural
#

no worries

sullen sparrow
#

I got confused

regal mural
# 
database.map(e => ({ ID: e.ID, data: e.data.length })).sort((a, b) => b.data - a.data)`
sullen sparrow
regal mural
#

yup, now you have the ranking

#

if you want to do it in a string with like data2: 3 ect
you just have to

.map(e => `${e.ID}: ${e.data}`).join("\n");```
sullen sparrow
#

really good

#

thank you

regal mural
#

which should look like.

const ranking = database.map(e => ({ ID: e.ID, data: e.data.length })).sort((a, b) => b.data - a.data);
const ranking_str = ranking.map(e => `${e.ID}: ${e.data}`).join("\n");
#

you can even make it a one liner if you want

sullen sparrow
#

How do I limit an amount of dates?

#

idk how to use the index

regal mural
#

I would use splice in that case, splice takes a sub array of the array

#

slice not splice my bad

#

for example if you want only 10

const ranking_str = ranking.slice(0, 10).map(e => `${e.ID}: ${e.data}`).join("\n");
sullen sparrow
#

thank you so much

regal mural
#

you can change 10 to how many you want

#

you're welcome