#best way of batch processing and limit SQL query results?

1 messages · Page 1 of 1 (latest)

stiff carbon
#

Needing to process 250k records through a Java application, but unfortunately the environment the application is in is unable to process all 250k at once. I’m trying to determine the best way to split the query and processing process into 30k records at once. I know I can set a SQL limit, but am wondering about offset or even using JPA slice? I still need all 250k to be processed, just in batches instead of all together.

normal mapleBOT
#

<@&1004656351647117403> please have a look, thanks.

normal mapleBOT
#

While you are waiting for getting help, here are some tips to improve your experience:

Code is much easier to read if posted with syntax highlighting and proper formatting.

If nobody is calling back, that usually means that your question was not well asked and hence nobody feels confident enough answering. Try to use your time to elaborate, provide details, context, more code, examples and maybe some screenshots. With enough info, someone knows the answer for sure.

Don't forget to close your thread using the command </help-thread close:1027500463647621170> when your question has been answered, thanks.

naive belfry
#

if the data you are fetching has some sort of ID that you can use to split it in parts then use some scheduled executor to do the querys with an interval that you like

stiff carbon
#

The IDs are randomly generated

celest sapphire
#

use spring's pagination features

#

and your server is gonna need a lot of ram for that many records. You should be able to fetch like 100, 500, 1k or so at a time and run whatever processing on it before fetching more and letting GC kill the old vars. dont know the use case tho

#

30k might work but that is also a ton of things Hibernate will have management over. I'd consider detaching them from the EM if doing massive fetches

normal mapleBOT
#

Closed the thread due to inactivity.

If your question was not resolved yet, feel free to just post a message to reopen it, or create a new thread. But try to improve the quality of your question to make it easier to help you 👍

stiff carbon
celest sapphire
#

Hibernate should be handling the implementation syntax and details for u with pagination