#Processing large amounts of text

1 messages · Page 1 of 1 (latest)

wintry adder
#

try batching to comply with token limits?

wind burrow
#

Multithreading and/or multiprocessing by book. Even with 500 pages per book, it shouldn’t take too long. That said, you should probably output your JSON by book as well. You can aggregate the output later