#Has anyone experienced issues with serverless /run callbacks since December?

1 messages · Page 1 of 1 (latest)

steel dirge
#

We've noticed that response bodies are empty when using /run endpoints with callbacks in the RunPod serverless environment (occurring sometime after December 2nd).

Additional context:

  • /runsync endpoints are working normally
  • Response JSON format appears correct in the "Requests" tab of RunPod console under Status
  • Our last deployment to this endpoint was two months ago

Could anyone confirm if there have been any releases since December that might have introduced this issue? We haven't made any changes to our deployment since two months ago, but are now seeing empty response bodies with callbacks.

Thanks in advance! 🙏

brisk echo
#

@upper widget

unique sedge
#

same here

image.png
#

getting a barrage of 520 and 415 since 4 am

jagged imp
#

same here

fathom isle
#

Same here with two clients of mine simultaneously. Status request immediately after the job's completion returns the correct result. But no webhook

jagged imp
#

getting JSON strings

#

sometimes empty responses

#

badly formatted string

twilit wolf
#

i am not getting any responses

jagged imp
#

it started happening today

twilit wolf
#

getting empty object in request.body and undefined in request.rawBody

#

Yes

jagged imp
#

everything is breaking 😢

twilit wolf
#

yes

#

No one is responding too.

#

my entire flow is disrupted due to this.

coarse magnet
#

Same here, we're only getting the input data, but not the request id anymore so we can't fetch the output of the job

hybrid sand
#

from what I found, they're missing the "content-type" header

#

the webhook POST is missing the content-type header, if you can fix it, just set it to application/json (I am using a middleware in rails)

mossy osprey
#

same here, getting initial "IN_QUEUE" status but not receiving any response , runsync works ok

#

anyone has a fix or know what's wrong?

coarse magnet
#

I think we need to wait for them to wake up, it's currently ~2am in san francisco

unique sedge
#

smth like this fixed it for me in python

twilit wolf
#

Yes, If using nodejs, set the request header to application/json manually and use express json parser to get the parsed body

app.post('/webook', (req, res) => {
    req.headers['content-type'] = "application/json";
    express.json()(req, res, ()=> {
        console.log(req.body);
        // Your code goes here

        res.sendStatus(200);
    })
})
mossy osprey
#

unfortunately my company uses bubble as a front end service, do i have to make a middleware on my end to solve this?

faint tapir
brisk echo
#

Kek

pulsar marsh
brisk echo
#

Anyone can take a screenshot I wanna see?

#

Does it happen on all region? What or what region do you guys use

faint tapir
pulsar marsh
#

fails on every datacenter

brisk echo
#

The job fails or it only returns empty output?

pulsar marsh
#

it returns a valid output (can be seen in requests tab of an endpoint), but the webhook post request body doesn't seem to be a serialized json

event setting content type manually didnt work for me

trying to serialize it manually from chunks sent

brisk echo
#

So it errors in webhook only?

#

But the /status and /run output works well?

pulsar marsh
#

Yup

#

For those who work with node.js and didn't resolve it with manually setting the content-type, here's the custom serializer from chunks into json

(express)

public async handler(req: Request, res: Response, next: NextFunction) {
    req.headers["content-type"] = "application/json";

    let buffer = "";
    req.setEncoding("utf8");

    req.on("data", (chunk) => {
      buffer += chunk;
    });

    req.on("end", () => {
      try {
        req.body = JSON.parse(buffer);
      } catch (err) {
        console.error("Error parsing JSON:", err);
      }
      next();
    });
  }
dusk junco
#

same here 🙌🏻
Sometimes I get empty body (with binary data). Is there any explanations from runpod?

pulsar marsh
#

not yet, i see this happens since 03:00 at night UTC

coarse magnet
#

Is there any way to escalate issues like these to runpod staff, especially if it happens in the middle of the night for them?

pulsar marsh
#

i guess discord is the only place

brisk echo
brisk echo
coarse magnet
brisk echo
coarse magnet
#

Ok, I guess they will notice either way once they wake up. But I guess a company like runpod should also have monitoring set up that screams at them when suddenly a huge amount of webhooks across all customers fail.

pulsar marsh
#

We've recently had an sdk version that didn't work properly for about 2 weeks straight so...

dusk junco
#

But I think runpod should increase their focus on support or deployment management. Because 1 or 2 months ago, runpod sdk was broken and I couldn't see if I checked discord

dusk junco
#

The right one is request from runpod

first.png
#

And Runpod respond as binary, I guess

second.png
#

And that's the binary data

third.png
brisk echo
dusk junco
#

Yes :/ When they will fix u think?

brisk echo
#

When they wake up and working

#

Maybe like few more hours

upper widget
#

Hey, sorry about this! We’re aware of the issue and will have a hotfix in next hour. The response is currently missing the application/json header. As a workaround, you can update your code to parse the body as JSON even if the header is missing.

pulsar marsh
#

Thanks for quick respond 💪

dusk junco
jagged imp
#

@upper widget any updates? 👀

brisk echo
#

Don't worry they will announce it as soon as it's fixed either here , or in #📢｜announcements

jagged imp
#

was in the middle of a huge refactor to make this work but if you are working on it I'll just wait

upper widget
#

We’re running the final tests now, it should be ready soon.

upper widget
#

We’re pushing the change now; it will take about 15 minutes. I’ll keep you posted.

upper widget
#

The release is almost complete, and my testing shows the response looks good. Could you verify it on your end and let me know if you still encounter any issues?

mossy osprey
#

thanks

coarse magnet
upper widget
jagged imp
#

All systems operational now, back to normal

#

Even with priority support this was quite unnerving, would be great if you could have support team for the midnight san francisco hours

upper widget
steel dirge
#

I'd like to confirm that our application has recovered from the above issue. Thank you.

hybrid heath
#

I am having issues right now... Cannot create pods through the python package despite having GPUs available. Keep getting the "There are no longer any instances available with the requested specifications. Please refresh and try again" but if I try to create it with exactly the same settings from the Web UI it all works out...

brisk echo
hybrid heath
#

Its been happening since last week. Is I try to hit the button on the webui at the same time with triggering the script, it doesn't work with the runpod python package but it works with the Web UI.

brisk echo
wary jolt
#

Are any of you still experiencing issues with serverless vllm? I cannot manage to release a working endpoint. I keep getting 500s and even some 502 bad gateway from cloudflare. I don't even know how to further describe my issues, it's days that I'm banging my head on this problem and I'm losing sanity. I tried to rollback to runpod/worker-v1-vllm:v1.6.0stable-cuda12.1.0, without any luck. Lucklily it seems that my old endpoints created in the past few months are not experiencing visible issues

#

502 are coming in strong now and my in progress requests seems to be multiplying according to inprogress counter (without aparent reason)

upper widget
upper widget
wary jolt