#Overhead of Go routines and Rules of Thumb for usage?

22 messages · Page 1 of 1 (latest)

last axle
#

Does anyone have good rules of thumb on when it makes sense to use go routines?

I have been using go routines to speed up large loops of 1000s llm queries and they've been working great. Its an obvious usecase as each llm query may take up to 1 minute and most of the time is just waiting for an external server.

But after seeing how well they worked, I thought about their usage for smaller computational tasks and was wondering when it makes sense to use a goroutine and when it would be better to just use for loops?

Is there a way to determine when it is more efficient to use goroutines versus loops, other than just benchmarks/experimentation?

For instance if we were reading and parsing a large text file with long lines of text?

  1. We could increment over each line using a traditional for loop.

  2. Run a goroutine to parse each line

  3. Split the file into x1 segments of size x2 and then have each parse their own section

Currently my intuition tells me that:
1 Would be best for a text file of single words where we do a very simple calculation
2. would be good for VERY long lines of text with a COMPLEX computation for each
3. would be best for medium lines and medium tasks

So I see situations in which these are all options that may be useful.

Does anyone have good rules of thumb on when it makes sense to use go routines or will I just develop this intuition the more I use/experiment with them?

void shoal
# last axle Does anyone have good rules of thumb on when it makes sense to use go routines? ...

The problem of using goroutines for computation is that it's always less efficient.
When IO is not at play you use more total CPU resources to compute the same result. However if for example you add a 1.2 overhead but spread it over 4 cores it's 4/1.2 = ~3.33 times faster overall.

So what happen for most go programs is that you scale unrelated tasks that you would use different goroutines anyway for IO reasons.
Like a webserver, net/http uses a goroutine per HTTP connection. So even if goroutines could make the latency of one request faster you would be able to handle less users since each user use more CPU resources.

Tl;Dr: if you don't have any data to do otherwise only use goroutines for IO reasons (imagine completing an HTTP query requires querying 3 databases, it make sense to use a goroutine per database).

Now the fun part, imagine you have a piece of code that doesn't scale to multiple users. You only ever do one job.
Creating and destroying goroutines requires handling memory allocations which is expensivish, 100ns~10µs range usually.
Synchronization is very inefficient, so avoid writing to shared memory someone is about to take (this include mutexes, atomics, channels, ... all the sync primitives).
If you write bad code it's more efficient to stand up on the scheduler's shoulders, that means sparying goroutines and hopping for the best.
If you profile, benchmark your code and improve your code, it's not extremely hard to do better than the scheduler, that means you use runtime.GOMAXPROCS(0) to fetch the maximum allowed CPU count and only start that many goroutines, then you use your own algorithm to distribute work among theses goroutines.

#

Let's say you were tasked to count the number of if byte in a really huge []byte slice func CountByteValues([]byte) [256]uint.
What I would do:

  • Split the slice into runtime.GOMAXPROCS(0) slices.
  • Spawn a goroutine per subslice which is gonna write into it's own [256]uint.
  • The original goroutine then sum up all the tallies into a final [256]uint you return.
#

You would use benchmarking to determine what is « really huge » and avoid using the multithreaded implementation for slices where you find there is no benefit.

#

The bullet points above would avoid becoming too inneficient by limiting the memory caches invalidation.
A good example of what not to do is [256]atomic.Uint and using x[subslice[i]].Add(1) to increment a shared array of counting buckets. The CPU will have to synchronize the caches all the time.
While with my solution it only does once at the end when doing the summation.

#

||Also I wouldn't do exactly what I described you could optimize that a tiny bit more but chances this is not worth it.||

true cypress
#

The problem of using goroutines for computation is that it's always less efficient.
[...]
Tl;Dr: if you don't have any data to do otherwise only use goroutines for IO reasons

maybe there are nuances later but that's an odd summary, there are many pure compute tasks that benefit from parallel execution, at least up to the number of hyperthread the host machines supports (ie gomaxprocs)

void shoal
true cypress
#

it's still more efficient to use cpu than having sit idle because a job is single threaded while it could be parallelized

void shoal
true cypress
#

and yes there is a cost to scatter and gather (split the work and reassemble) so the unit of work has to be worth it - but that 'nuance' should be included in TL;DR;

#

it's certainly not 20% for what would make sense to split

void shoal
#

The tldr doesn't even talk about efficiency, I think if you aren't writing the 5 lines of code needed to benchmark it why you are spending time paralelising it ?

void shoal
#

It's a random number to show the math, in the wild you can find bad code with losses > 100% and ones where it's insignificant

true cypress
#

anyway sure, measuring always makes sense, though there are stuff that clearly intuitively would benefit from using more than 1 core, even with 0 IOs (also IOs actually don't really need more cores... by definition they just sit there waiting/unscheduled)

void shoal
#

Measuring it can be as simple as time ./cpu-compute-work if you are gonna observe 2X performance improvements

true cypress
#

problem is you have to write the code to split the work, that's harder than measuring (harder to pick the right way, guess the likely benefit) but yeah

void shoal
#

I don't get what you mean

#

Regardless of the conclusion of this question you should learn how to do that anyway at some point given it's useful if for IO or compute bound apps.
Once you did it once or twice this is routine.

true cypress
#

you're argument was the OP doesn't know... then my argument is if you don't know; the "benchmarking/measuring" is the easy part