#Load tests - Self hosted Novu instance

1 messages · Page 1 of 1 (latest)

tribal river
#

Hello,
We are trying to do some benchmarking in our self hosted Novu instance.
Is there any prior limitations on RPS on different Novu APIs. I do see in official documentation on some rate limiting applied, but for cloud versions.
On increasing the RPS for trigger events, we do see system get hung up a bit on Novu API service (tested with 3 pods), causing timeout errors as well.

tough rampartBOT
#

@tribal river, you just advanced to level 4!

tribal river
#

Hi @frank lintel
Can you give some info on this?

torn pollen
tribal river
#

@torn pollen
I did a bit of changes on my end. I'm only using chat webhooks, to eliminate provider dependencies.

  1. Changed workflow type from BRIDGE to REGULAR. >> This doubled performance as no api calls were made from worker service to api layer.
  2. Document DB CPU, high load caused CPU to be exhausted. Increased cpu limit here.
  3. No of connections open per service. This also had major impact. This requires testing against all services you're running. And concluding a proper value. I'm using 10-50. Could be high. But needs more testing.
tribal river
#

Also set MONGO_AUTO_CREATE_INDEX to True on first run. This will improve read queries in services as certain collection can increase over time.

burnt lance
#

Hey @tribal river , can you elaborate on how you changed the workflow type from BRIDGE -> REGULAR? I am not seeing any workflow type options in the dashbaord or API. What version of Novu are you using?

tribal river
#

@burnt lance
I'm using Api to create Workflows through v2 Api. You will need to add a field called type with REGULAR as value. Steps also has type I think.
When using dashboard to create Workflows. This type is automatically set as BRIDGE. You can check directly in db.

burnt lance
#

Hey @tribal river , can you please share a sample API request? I have tried hitting the v2/workflows POST API with type: "REGULAR" but in the db i still see type is BRIDGE

tribal river
#

@burnt lance
Yeah. I went through the code. For now I'm using v1 api. Which accepts this type and also allows to attach content to the templates unfortunately.
I'm still trying to get some get some clarity of v1 and v2 Workflows what's the differences.