#V2 runTask() return size limit?

1 messages · Page 1 of 1 (latest)

warm pike
#

I was curious if there is a size limit when running a task on what can be returned.

Suppose I run a Task in which I fetch a huge json object from an API which I wanna return in my task so I can use it in a later task again, is there a size limit for this?

I tested it, with a json object which has roughly 30k objects, I do get a error when trying returning this but the error does not suggest that it is because of the actual size of the json.

Error:

SyntaxError: Unexpected token < in JSON at position 0
    at JSON.parse (<anonymous>)
    at parseJSONFromBytes (node:internal/deps/undici/undici:6662:19)
    at successSteps (node:internal/deps/undici/undici:6636:27)
    at node:internal/deps/undici/undici:1236:60
    at node:internal/process/task_queues:140:7
    at AsyncResource.runInAsyncScope (node:async_hooks:203:9)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:137:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

At the same time if I slice the Object to the first 10 objects for example everything works, which is why I got curious if it could be a limit. I didn't find any limit regarding this in the docs.

fair dock
warm pike
#

Thanks! That makes sense. You specified that thats the limit for a v2 Task, does it mean that using v3, we would not have such limits since runTask also doesn't exist anymore?

fair dock
#

Yeah you can just get all the data you need inside the run function and process it how you need to. Only when you trigger or return from a v3 run do you need to be concerned about size limits. And in those cases we offload data to storage for large payloads/outputs (we use R2 in the cloud).