I'm currently working on a datavisualization and manipulation project, it's a desktop application I will be using for myself exclusively so scalability isn't a problem.
My main question as mentioned in the title refers to a couple factors like response time, data modification time in runtime, reliability and to an extent how interesting it would be to pick up said language/framework.
The current backend is written in Java but I am open to rewriting it if I figure another language would work better for the project so please hit me up with your recommendations. ^^
#What is the best language/framework to quickly transfer data between UI and SQL database?
spring hill
broken kiln
I don't think using a different language or framework will actually fix your problems for long
Sure, you could go from Java to C++ or Rust and probably squeeze out a 2x or 10x performance increase, but it won't solve the actual problems
I'd say you probably want to think about the following strategies:
#1 Better database indexing to speed up queries
#2 Caching responses
#3 Precomputing certain KPIs / visualizations ahead of time so you just have to display them (instead of calculating them for every request)
#4 Sending less data over the wire (can you use your database to do joining, etc.?)
But for any concrete tips I'd have to know more about what exactly you're doing
spiral rampart
Fwiw if you're having slow downs around database queries, using a "faster" language won't fix your performance issues.
A database requires an I/O access which is inherently much slower than even the "slowest" programming languages. So like not rob said, you need to optimize your database and how/when you access it.
spring hill
That advice actually is something I didn't know and that might really help but I think you misunderstood the question. The speed is not a serious problem this is just a question of me trying to learn new things.
To reply to @broken kiln I'm making a finance planning tool that's supposed to be the basis for some other finance and market prediction projects.
broken kiln
That advice actually is something I didn't know and that might really help but I...
Sounds fun, most of the stuff in the financial space is precomputed (wherever possible)
The 4 points I gave still apply, but I'd need more info to give you some tips or whatever
supple tendon
I think Rust is the best for that. I will give you detail informations.
rapid maple
That advice actually is something I didn't know and that might really help but I...
The most drastic improvement in SQL performance I've seen was getting the database moved onto an M.2 NVME drive.
But once I optimized that database I doubt the difference would have been so drastic/noticable.
broken kiln
I think Rust is the best for that. I will give you detail informations.
Why would you say that?
broken kiln
The most drastic improvement in SQL performance I've seen was getting the databa...
We aim to keep the entire database in RAM at all times, SSD is a must if you want any kind of performance
rapid maple
We aim to keep the entire database in RAM at all times, SSD is a must if you wan...
To be fair, the original database was storing mime encoded images straight to a table when I got involved.
It didn't take a lot of time target that as a huge problem, and the solution was pretty easy, just record the CRC32 checksum for each image, and don't store an image with an existing checksum. When viewing records just grab the image that matches the recorded checksum. All good.
broken kiln
Ahahahahahaha we did similar bullshit back in the day
We had PDFs in the database as well
rapid maple
Is that better or worse than using PDF as a database?
My city does all their water management records in PDF.
I'm sure there's someone working on the tech side who just said, "This is better than filling out paper forms and scanning them into an OCR!", and then rubber stamped it.
broken kiln
Saving files in a database was the laziest thing I've ever done
Makes other queries slow because files are big, we eventually migrated them out to S3 and saved like $30k a month
RAM on database instances costs more than S3 storage, disk space + replication on db instaces is expensive as well -> S3 big money saver
spiral rampart
Why are we storing images in the DB??? Store them on disk and the path in the DB???
broken kiln
Why are we storing images in the DB??? Store them on disk and the path in the DB...
We did it at the time because we just wanted to get things working and it was very comfy (but switched to S3 rather quickly)