#Serverless worker doesn't run asynchronously until I request its status in local development

13 messages · Page 1 of 1 (latest)

ornate pumice
#

I'm following the docs and created a verey simple handler.py with the following:

import runpod

def handler(event):
    """
    This is a sample handler function that echoes the input
    and adds a greeting.
    """
    try:
        # Extract the prompt from the input
        prompt = event["input"]["prompt"]

        print (f"Prompt: {prompt}")

        result = f"Hello! You said: {prompt}"

        # Return the result
        return {"output": result}
    except Exception as e:
        # If there's an error, return it
        return {"error": str(e)}

# Start the serverless function
runpod.serverless.start({"handler": handler})

This is running inside a Docker container with a custom Dockerfile and executed with

python3 ./handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000

The local server starts:

--- Starting Serverless Worker |  Version 1.6.2 ---
INFO   | Starting API server.
DEBUG  | Not deployed on RunPod serverless, pings will not be sent.      
INFO:     Started server process [1]                                     
INFO:     Waiting for application startup.
INFO:     Application startup complete.                                  
INFO:     Uvicorn running on http://0.0.0.0:8000 (Press CTRL+C to quit)

But when I POST localhost:8000/run with the expected prompt input, I get a id back saying the job is in progress, but nothing shows on the container logs. It doesn't run until I call /status/jobId. Only then it runs.

Is this the expected behaviour? If so, why?

vernal vale
#

Have you tried testing your handler offline by doing the following? It will map a local version of test_input.json inside your application.

docker run -v ${PWD}/test_input.json:/app/test_input.json myimage:v3

This will process your test_input.json through your handler locally. Without having to run Uvicorn at all.

#

In your Dockerfile you can start your handler like this:

CMD ["python", "/app/handler.py"]

ornate pumice
#

I have not, but anyway it should work well by running a local uvicorn server.

vernal vale
#

Can you share your Dockerfile?

#

Are you doing something like this?

CMD ["python", "/app/handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000"]

#

You might get more info by entering your image with a bash shell and then trying to load the ./handler from there.

ornate pumice
#

Yes, as I said, the webserver works fine, I can hit the endpoints from my local machine. The jobs won't simply start if I call /run unless I request the status afterwards.

#

If I do /runsync, they do start immediately.

#

My Dockerfile ends with

CMD python3 -u ./handler.py

But on development I'm launching it from a compose file where I override the command with

services:
  myservice:
    // Omitted
    command: python3 ./handler.py --rp_serve_api --rp_api_host 0.0.0.0 --rp_api_port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
vernal vale
#

I would still use -u in dev to prevent buffering but everything looks good.

somber meadow
#

I'm having the same problem, @ornate pumice did you find a solution to this?

ornate pumice