Should I use one or multiple tasks?

[jaunty dome]

Hey there!

I'm building an app where users can train an AI model using their own images. The workflow would be something like this:

1. Upload user images to a folder in Supabase
2. For each image in the folder:

Convert to PNG -> Run through Claude Vision API -> Generate .txt file with captions

3. Once all images have their corresponding .txt file create a .zip file of the folder
4. Call the training endpoint with the .zip file

From a user point of view, they "only" upload images and get a trained model.

My question is, should I divide this task into sub-tasks? Asking mostly for possible concurrent run limitations in case we use multiple tasks and have multiple concurrent users.

Thank you very much in advance!

[graceful rose]

Hi @jaunty dome. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

This is a great question about structuring your workflow in Trigger.dev! Based on the knowledge sources, I can provide some guidance on whether to use one task or multiple tasks for your AI model training workflow.

Task Structure Options

You have two main approaches:

Option 1: Single Task with Sequential Steps

You could create one task that handles the entire workflow sequentially. This approach is simpler to manage but might not be optimal for performance.

Option 2: Multiple Tasks with Orchestration

You could break this down into multiple tasks:

• A main "orchestrator" task that coordinates the workflow
• Worker tasks that handle specific operations (image conversion, caption generation, etc.)

Considerations for Your Decision

Concurrency Limits

Trigger.dev has concurrency limits based on your pricing tier:

• Free tier: 5 concurrent runs
• Hobby tier: 25 concurrent runs
• Pro tier: 100+ concurrent runs

If you have multiple users triggering workflows simultaneously, you might hit these limits with a single-task approach. Limits documentation provides details on these constraints.

Parallelization Benefits

The knowledge sources show that Trigger.dev supports parallelization patterns that could be beneficial for your workflow. Using batch.triggerByTaskAndWait() allows you to run multiple tasks in parallel and efficiently coordinate their responses, which would be perfect for processing multiple images simultaneously.

As mentioned in the Parallelization documentation, this pattern is "particularly valuable when different parts of a task can be handled independently."

Orchestrator-Workers Pattern

Your workflow fits well with the "orchestrator-workers" pattern described in the knowledge sources. This pattern uses a main task (orchestrator) that breaks down a job into smaller tasks and assigns them to specialists (workers), then puts everything back together.

Recommendation

Based on the information available, I recommend using multiple tasks with the orchestrator-workers pattern:

1. Create a main orchestrator task that:

   • Receives the uploaded images
   • Triggers worker tasks for each image using batch.triggerByTaskAndWait()
   • Creates the zip file once all workers complete
   • Calls the training endpoint

2. Create a worker task that:

   • Converts an image to PNG
   • Runs it through Claude Vision API
   • Generates the caption text file
   • Returns the result to the orchestrator

This approach offers several advantages:

• Better scalability for multiple concurrent users
• More efficient resource usage through parallelization
• Clearer separation of concerns
• Better error handling (if one image fails, others can still complete)

To avoid potential deadlocks when using recursive task triggering, make sure to follow the advice in the queue concurrency documentation about using different queues for parent and subtasks.

Would you like more specific guidance on implementing this pattern with Trigger.dev?

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: