Lavalink scaling considerations

[inner laurel]

Hey all,
I have looked for some more in depth performance metrics/scaling information about LL and haven't found much. Does anyone have any tips, tricks or things I should watch out for when migrating a decent sized LP setup to LL? I wrote a LP based bot in like 2017/18 and it's been sitting for a long time, finally porting it to a solution I can scale out with and don't want to write one myself. LL will be running on a k8s node with 16gb of ram and 6 xeon e-2697v2 cores.

For reference my LL usage metrics are expected to be:

• 200-500 active streams at any given time, depending on time of day
• 30-40k track loads per day, balanced across a couple /64 ranges
• Expecting 30+ qps to the api at peak
• Even though I haven't touched the bot in years voice usage is growing ~20% MoM, so I want to plan for a lot more down the line. For when I don't have time to work on it, it hopefully wont run into issues.

I would imagine this scale is fine for a one node setup for now, I already ported everything to LL and hoping to roll out this week. Does LL seem to like scaling up, out or either? Eventually I am going to at the least spin up basic low spec nodes in each major region. I also typically ran a larger than defualt (800ms - 2000ms) JDA-NAS buffer to help against stuttering, in LL how does that effect resource usage? I would imagine just minimal RAM impact.

Thanks for the help and wisdom :)

[crude tartan]

For bigs bots, you would probably want load balancing, using more than 1 lavalink node

About specific settings to improve performance, I don't know, never had to look much at it

[fickle relic]

200-500 active streams should easily fit on one lavalink node with those specs. A node will occupy a smaller heap compared to using Lavaplayer directly within a bot, thus less pressure on garbage collector etc. Yes, you can scale horizontally or vertically, although ideally you would scale vertically once you're close to the limits of the hardware you're hosting on. You'll probably want lots of smaller nodes rather than a few big nodes, as obviously the larger the nodes the more bandwidth they'd be using, as well as heap space used and garbage created which puts pressure on the GC and by extension, the CPU. Hosting multiple nodes on the same machine can work in this case but seems a little pointless and defeats the purpose of distributed audio nodes.

A larger NAS buffer shouldn't affect resource usage too much, you may notice a slight increase in memory consumption, but also keep in mind that increasing the size of any buffer may also have a knock-on effect of increasing the delay before things like filters etc are applied.

[inner laurel]

Sweet thanks, as long as one beefy node can handle up to 500 streams for the short term. I do want to more efficiently spread out loads and see if I can cut some costs on hosting as well. Is there any blog posts, or threads with benchmarks or example resource usages? I would love to see different comparisons of how arbitrarily sized nodes hold up.
Worst case scenario maybe I will just have to play with it and keep records of how things scale, although I dont think I have the scale to fully saturate many larger VM configs.

[fickle relic]

Is there any blog posts, or threads with benchmarks or example resource usages?
Not to my knowledge, although if you have the hardware already (or know someone with same/similar specs, or even have a friendly hosting provider) you could ask them to run Lavamark which would give you an estimate as to how many players a node could handle

Just keep in mind this spawns a fair few players to simulate a real world load and, if using YouTube as the benchmark track source, could get the request IP banned pretty quickly. I think there's flags you can pass to change what track it uses as a benchmark, though

Edit: this benchmark only has one supported command-line argument, "opus", which tells it to use a single opus track rather than a playlist

[inner laurel]

One last follow up question, do inactive players take up any amount of significant cpu? I would imagine LL does not take any incoming audio from the connections, so I would doubt it.

[drifting osprey]

while lavalink does not process any incoming voice udp packets discord will still send those

to avoid this just server mute the bot

muting yourself isnt enough

it needs to be the server muting

[warm slate]

Ur going to get rate limited with that /64 range

[inner laurel]

I have multiple

And a /48 as a fallback if I hit bans

[warm slate]

dope jus so yk tunnelbroker can result in a some issues tho let that be known