Hi everyone

[trim ivy]

I want someone's that worked with celery before to take a look at my tasks here and see if it's efficient, everything is working but I want to make sure that am not messing it up:

tasks.py:
@shared_task
def send_notification_email(email_to, respond, producer):
    user = get_user_model().objects.get(email=email_to)
    subject = 'You have notifications'
    user_name = user.username

    html_template = 'project/emails/notification_email.html'
    context = {
        'username' : user_name,
        'respond': respond,
        'producer': producer
    }
    task_chain = chain(
        encode_image.s(),
        send_email.s(email_to, subject, html_template, context)
    )

    # execute the task chain
    task_chain.delay()


@shared_task
def encode_image():
    # MUST EDIT IN PRODUCTION
    image_path = os.path.join(settings.BASE_DIR, 'static', 'img', 'email_logo.jpg')

    # encode the image file
    with open(image_path, "rb") as image_file:
        encoded_image = base64.b64encode(image_file.read()).decode("utf-8")

    return encoded_image

@shared_task(bind=True, max_retries=2)
def send_email(self, encoded_image, to_email, subject, html_template, context):

    try:
        # add the encoded image to the context
        context['logo_base64'] = encoded_image

        # render email template with provided context
        html_content = render_to_string(html_template, context)
        text_content = strip_tags(html_content)

        msg = EmailMultiAlternatives(subject, text_content, settings.DEFAULT_FROM_EMAIL, [to_email])
        msg.attach_alternative(html_content, "text/html")
        msg.send()

    except Exception as exc:
        # retry the task twice in case of failure
        raise self.retry(exc=exc)

celery server:
celery -A project_name worker -l INFO --without-gossip --without-mingle --without-heartbeat -Ofair --pool=solo

let's say there is a heavy task that django is running and an email should be sent at the end using celery,
sometimes the whole tasks will take time even though am using celery, and other times django just does his task blazingly fast like there is no email should be sent and celery will send the email (and this is perfect) but I don't know why it's happening, and I can't figure out how to measure how much time the celery tasks is taking seperated.

[simple rock]

I'm just wondering why you split it into multiple tasks, especially the encode image which takes no parameters and does the same thing every time, there is no need for that.

[trim ivy]

I was using one task but it took time so I figured to split it since it's asynchronous tasks, the encoded image will be stored when I deploy so it won't do the same task

[simple rock]

Is it long in the execution of the task or starting it? Your worker might be busy with something else

That's also why 1 task would be easier, you can just call it without delay and debug it to see what is happening/what the reason is for slow execution

I see 3 things that could make it slow: Getting the user from the database, getting the image file from disk or connecting to whatever you have as mail backend and sending the email.

[trim ivy]

if I want to use processes like "--autoscale=4,1" would it be better?

[simple rock]

I don't use the chains, so not sure how it keeps track of the dependencies, but it would be a worker needs to keep track of doing the encode image task and then pass the result to the second one, so if a worker is busy with that you need another one to actually execute those tasks.

[trim ivy]

that is why I choose to split the tasks but now I see that there is no real difference like you said:

@shared_task(bind=True, max_retries=2)
def send_notification_email(self, email_to, respond, producer):
    try:
        user = get_user_model().objects.get(email=email_to)
        subject = 'You have notifications'
        user_name = user.username
        html_template = 'project_manager/emails/notification_email.html'
        
        image_path = os.path.join(settings.BASE_DIR, 'static', 'img', 'email_logo.jpg')
        with open(image_path, "rb") as image_file:
            encoded_image = base64.b64encode(image_file.read()).decode("utf-8")

        context = {
            'username' : user_name,
            'respond': respond,
            'producer': producer,
            'logo_base64': encoded_image
        }

        html_content = render_to_string(html_template, context)
        text_content = strip_tags(html_content)

        msg = EmailMultiAlternatives(subject, text_content, settings.DEFAULT_FROM_EMAIL, [email_to])
        msg.attach_alternative(html_content, "text/html")
        msg.send()

    except Exception as exc:
        raise self.retry(exc=exc)

it takes exactly the same time in one task, i tried "--autoscale=4,1" instead of "--pool=solo" but it just get stuck trying to send the emai,
how does "solo" work btw?

[simple rock]

ok, bit of background: When you start a celery worker it itself is not the one executing the tasks, it basically managers workers that handle tasks, the solo option means it actually does handle tasks on it's own, so it's just a single worker that does it's thing. That's great if you want to scale on a different level (more docker containers for instance) but in general I wouldn't use it

[trim ivy]

for sending emails and make complex reports, what do you suggest to use?

[simple rock]

Either don't specify pool, so you get a prefork worker or one of the other options

Multiple tasks for those emails might be started at the same time, so you want to proces them fairly quickly and even multiple at the same time.

In general you don't want to be processing them one by one, but at the same time