devout lotus
I sometimes (try) to run some really heavy tasks (ai training, data or bitbake). The kind that takes all the ressources available by default.
On my latest attempt, the cpu was always at 100%, avrg load at 180 and 800 ish threads total.
I setted the thing up for 12/32 worker threads max, and still ended up hitting a bottleneck.
Is there a monitoring software or a method to find the honeyspot ensuring max efficiency ?
peak badger
I sometimes (try) to run some really heavy tasks (ai training, data or bitbake)....
Monitor the actual bottleneck you can use htop, btop or sysstat
midnight remnant
I sometimes (try) to run some really heavy tasks (ai training, data or bitbake)....
Honeyspot? Haven't heard that before loll
frank tangle
Honeyspot? Haven't heard that before loll
probably a calque of 'sweet spot'
midnight remnant
probably a calque of 'sweet spot'
That makes sense lol. I had figured that was the case
devout lotus
Honeypot, honeyspot... Be creative 😄
ruby kestrel
I sometimes (try) to run some really heavy tasks (ai training, data or bitbake)....
youre over provisioning your threads. use iostat to check your iowait and if its high your drive is the bottleneck. and drop your worker threads down to your core count, check completion time and slowly increment until performance starts to degrade.
devout lotus
So it isn't the io but too much context switching... Too much threads at once I beleive. Something I can do to prevent that except for reducing the number of workers ?
ruby kestrel
So it isn't the io but too much context switching... Too much threads at once I ...
if you dont wanna reduce worker count you gotta minimize cache thrashing caused by threads jumping between physical cores. use taskset to bind worker PID to specific cores to prevent kernel scheduler from moving them around. taskset -cp [threads to pin, eg 6-7] [PID]
also is your machine multi socket?
you can also switch your task to SCHED_BAATCH using chrt to allow it to run for longer time slice before being preemted chrt -b -p 0 [prcess id]
also you can isolate heavy tasks into its cpuset controller so it reserves some cpu for that task and can prevent the rest of the os from interfering with those cores
old drift
I sometimes (try) to run some really heavy tasks (ai training, data or bitbake)....
you can use nice somewhat
you probably just want your desktop or whatever gui to remain at higher priority, nice should be able to help? what exactly are the things you're running?
ruby kestrel
nice only changes cpu scheduling priority, but doesnt prevent cache thrashing or stop threads from moving across cores. taskset can pin a worker's threads to specific cores or utilize cgroups for max core isolation