🔒 What better I can do .

[sterile ingot]

In one of my projects, the client was using MongoDB and S3 for segmentation. Their approach was to store data in S3, then whenever they needed to perform operations, they would first store the data in a temporary file, process it, and finally save the results back into MongoDB. This process was very time-consuming.

To improve it, I replaced MongoDB with PostgreSQL and DuckDB. DuckDB allowed me to query data directly from the S3 bucket without saving it locally, which made the output around 50% faster compared to the original approach.

Now, I’m looking for an even more optimized or scalable approach to handle this kind of workflow.

One more thing If I want to showcase user data to frontend to show 5000 rows or more it's taking a lot time.

[floral haven]

@sterile ingot

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[rustic meteor]

you can add caching for user data so it is not fetched everytime user ask for it

[slate mountain]

You can implement Spark + Parquet

And for frontend display, Try chunking + Virtual scrolling

They'll add complexity, but will perform significantly better than your current setup

And yeah, implementation of kafka will definitely yield better speed (if you have certain usecase)

[floral haven]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.