import json
# Load JSON data from a file
with open('data.json', 'r') as file:
data = json.load(file)
# Function to check if an element exists in the JSON data
def check_element_exists(json_data, element):
return element in json_data['data']
# Example usage
element_to_check = 'element100000000'
exists = check_element_exists(data, element_to_check)
# Print True or False
print(exists)
#🔒 Any faster approach to check if text exists in file, there can be around 100 Million elements in j
sick bison
edgy forgeBOT
```
import json
# Load JSON data from a file
with open('data.json', 'r') as fil...
@sick bison
Python help channel opened
narrow whale
what size is your json (in disk space ?)
lofty bloom
well, unless you've got an index to that data, I doubt there's a faster way
narrow whale
cause if you're running into ram issues, whether you open it as a file or as an object won't change much
lofty bloom
what, specifically, is this data, and why are you checking it?
sick bison
what size is your json (in disk space ?)
around 120 GB
lofty bloom
a 120GB JSON file sure doesn't seem useful
sick bison
what, specifically, is this data, and why are you checking it?
user query
stark pier
the correct thing to do would be to convert your entire thing to something that isn't json, like mysql
json is not a database format
icy quartz
They might be better off with a nosql solution depending on what days they have tbf
sick bison
I have a section in my app where I will show different pages to user, each page has title. and title of all the page are stored in json file. I need to check If I have showed the page to user, if not I will show that to user. In really there will be max 30 million elements(titles) in json file but I am testing for 1 billion. It will never cross that 30 million mark
lofty bloom
why do you need to store anything other than pairs of users & titles?
Just go through that list to see if you've already displayed it. If not, display it and add it to that list.
I doubt that list will have zillions of elements.
sick bison
max will be 30 million. User can't see more than 30 million pages in his lifetime
sick bison
Just go through that list to see if you've already displayed it. If not, displa...
Yes wanted to do that but in the best way possible
sick bison
They might be better off with a nosql solution depending on what days they have ...
but may be that json file will be required in future so easily it can uploaded to cloud
lofty bloom
Yes wanted to do that but in the best way possible
well I don't understand, but whatever
sick bison
I have a section in my app where I will show different pages to user, each page ...
You understand this, right?
icy quartz
JSON is simply not suited to the operation that you want
It's clear what you're trying to do, it's not clear why you're doing it in that particular way, so it's hard to help imo
sick bison
It's clear what you're trying to do, it's not clear _why_ you're doing it in tha...
have a habit of using json only for all things but mysql also works
narrow whale
well JSON has limits you can't just use it "for all things"
sick bison
JSON is simply not suited to the operation that you want
so I should store in .db file
sick bison
well JSON has limits you can't just use it "for all things"
yes, I am very well aware
narrow whale
you're starting to use huge amounts of data so you'll want some database
and that's not just a size issue but a performance and data availability issue overall for your users too
icy quartz
I don't know the structure of your data, it's impossible to know what to recommend
But some options could be mongodb or mysql, but you should think about why you're choosing a particular option
sick bison
you're starting to use huge amounts of data so you'll want some database
and tha...
I understand that very welll
sick bison
I don't know the structure of your data, it's impossible to know what to recomme...
just a list
nothing else
narrow whale
if you tell us more about what's stored in that json, we could give you pointers towards which database to choose
a list of what ?
narrow whale
and those operations need to be done at which frequency ?
sick bison
['Mukesh Ambani','the wolf of the wall street',' baseball bat', 'shark tank', 'Trump']
list will be like this it can be any entity
narrow whale
is it just a list of strings ?
sick bison
and those operations need to be done at which frequency ?
need to check around 5 times if text in file in 1 seconds(5 different threads)
sick bison
is it just a list of strings ?
yes
narrow whale
and you want fast access to those ?
cause if so i guess a key value database like Redis would work just fine
sick bison
only read to check if text exists
and write to write a title which is visited
sick bison
and you want fast access to those ?
cause if so i guess a key value database lik...
yes
icy quartz
You need to do this per user? How many users are there?
sick bison
You need to do this per user? How many users are there?
It is done locally so it doesn't matter, right?
For now I don't know how many users will use the app but at least a million
icy quartz
Each user will locally store your 120GB data file?
narrow whale
wdym by "it is done locally" ?
sick bison
Each user will locally store your 120GB data file?
I said I was just checking it. The file size will never cross 20mb
narrow whale
maybe you should start from the beginning and explain fully what's your usecase so we can better understand
this feels very much so like a XY problem
icy quartz
^
sick bison
I was just checking on 1 billion elements actually there will be max 30 million elements
sick bison
maybe you should start from the beginning and explain fully what's your usecase ...
sure
I have a list like this ['Mukesh Ambani','the wolf of the wall street',' baseball bat', 'shark tank', 'Trump'] which is consists of 30 million elements. I need to check If a new element is present in a list. If not, Need to add that in list
The operation of check will be performed 5 times per sec using threads
so to check that I am using
import json
# Load JSON data from a file
with open('data.json', 'r') as file:
data = json.load(file)
# Function to check if an element exists in the JSON data
def check_element_exists(json_data, element):
return element in json_data['data']
# Example usage
element_to_check = 'element100000000'
exists = check_element_exists(data, element_to_check)
# Print True or False
print(exists)
string can be of any human language . Need to check that locally (file will be in device storage and file size will be 30 mb). The app is already taking ram of around 700mb. So need a method which will consume less ram
icy quartz
Is there any more context you can add?
sick bison
Is there any more context you can add?
not really, The app is made for Android, ios, mac os, windows and some operating systems of T.V
sick bison
Is there any more context you can add?
I will try
sick bison
Is there any more context you can add?
I have edited with atmost details
icy quartz
But what are you actually trying to do
Like what is the business problem you're trying to solve
sick bison
But what are you _actually_ trying to do
just showing doc/page which is not shown to user. user needs to see all pages of all docs
proven river
@sick bison check mmap ( memory mapping ) in python, it's a module specially used for reading large files which cannot be directly loaded into RAM
I had the same issue with log files of more than 10 GB, which i solved with using mmap
obsidian ibex
I have a list like this ['Mukesh Ambani','the wolf of the wall street',' basebal...
that's not json
json has no single quoted strings.
also apparently your json file contains an object with a "data" key, not a list.
if the file had a specific format (e.g. a json list of strings, with line breaks between the items) you could search the title line-by-line without having to parse the whole file.
additionally, if the list was sorted you could use binary search.
{"data": [
"baseball bat",
"Mukesh Ambani",
"shark tank",
"the wolf of the wall street",
"Trump",
]}```
Python help channel closed