Need help with concurrency

[pure gorge]

// Return the word frequencies of the text argument.
// Split load optimally across processor cores.
func WordCount(text string) map[string]int {
    var wg sync.WaitGroup
    // wg := new(sync.WaitGroup)

    var mu sync.Mutex
    freq := make(map[string]int)
    text = regexp.MustCompile(`[^a-zA-Z0-9 ]+`).ReplaceAllString(text, "")
    text = strings.ToLower(text)
    arr := strings.Fields(text)
    wg.Add(len(arr))
    for _, v := range arr {
        // wg.Add(1)
        // defer wg.Done()

        mu.Lock()
        go func(v string) { freq[v]++ }(v)
        wg.Done()
        mu.Unlock()
    }

    wg.Wait()

    return freq
}

func main() {
    // read in DataFile as a string called data
    data, err := os.ReadFile(DataFile)
    if err != nil {
        log.Fatal(err)

    }

    numRuns := 100
    runtimeMillis := benchmark(string(data), numRuns)
    printResults(runtimeMillis, numRuns)
}

So basically this is how it looks like, I have 2 other functions called (printResults and benchMark). So anyways the question is about using concurrency to count the number of words in a map or honestly I don't even know where I should use concurrency and I'd really appreciate it if someone could help me out here.

[still timber]

This is a big misunderstanding of goroutines

The locking, unlocking, and wg.done should be inside the goroutine

And I don't think making stuff concurrent can help much in this situation

Especially if it's with a mutex

@pure gorge Did you do the tour about goroutines? https://go.dev/tour/concurrency/1

[pure gorge]

they were meant too be inside the goroutine lol I didn't realise that they were outside. I'm dumb. But does look good other than that because I highly doubt that. BTW thanks for the tip, I will check that out!

[still timber]

This code does not benefit from concurrency though

[pure gorge]

how else do I od it then?

[still timber]

In this context, I don't think you can

[pure gorge]

"Update the WordCount function with the Map and Reduce tasks, using goroutines to parallelise and a channel to gather partial results"

The question is long but this is the part I'm working/ struggling with.

[still timber]

I dont' know what the original question is, but I dont' think this is a good target for concurrency, maybe someone can correct me

[pure gorge]

tbh it is stated in the question that the most challenging part is "identifying the different tasks that can be made parallel" so I've literally been stuck on this the whole day

[still timber]

You could instead of putting the regex and the tolower at the start put them isnide the goroutine

[pure gorge]

then what, each goroutine need to lowercase them? Would it make it faster?

[copper sage]

right now you are only making map insertion concurrent, but the majority of work is reading the file, splitting the words, lowercasing em

you could get the size of the file, split it into n chunks and do all the stuff in goroutines, merging the results and handling edge cases (e.g. mid-word chunk boundaries) later.

[pure gorge]

okay but how do I know the size of the file if I don't split it?

[copper sage]

I meant the size in bytes, so you split the work before tokenizing

[pure gorge]

oh, I've not learned anything about bytes yet.

Sorry I misunderstood you.

What do I do with the size of the file, I mean how does that help me (srry new beginner!).

[copper sage]

What's the end goal here? If it is to learn about concurrency - its probably not the best example for it. Go tour has web crawler exercise , that is much cleaner.

[pure gorge]

Basically this is what I am asked to do Implementation for the WordCount function Reading a text file into a string in the main function Filtering duplicate versions of words in different cases, removing punctuation marks, etc Check that the unittest passes Log the runtime performance in the table below Once you are satisfied with the singleworker, move into mapreduce/words.go and parallelise the program in order to improve the performance.

So basically the question has two parts, one to do it sequentially which I think I have done right. The second part is to somehow use concurrency to make it faster.

[tawny canyon]

I wouldn't create a new gorouting to just increment key of a map. Creating the goroutine cost you probably more resources than incrementing that key in the map. First of all I would use sync.Map from the go "sync" package instead of a map. Your mutex locking doesn't make any sense either, data races can still occur since the mutex is instantly unlocked after wg.Done()
So you have to pass a reference from the WaitGroup to your gorouting lock/unlock and then call Done()
This is still very unefficient and I would use worker pools for this. It may be a little complex to implement but it's definitely faster and done with sync.Map. You could check the length of the arr field and divide it by x to create small groups of the array and afterwards creating x goroutines that process simultaneously.

[jagged mulch]

Main problem you're having is that writing to a map, or any other write operation for that matter, can be a very delicate operation and it can break your program. You must identify which processes can be done concurrently without crashing into each other.

What I would do is:

1. Have a process identify and classify the words (using the word as a key in a map and a unique id as a value). Then pass this classification_id (value) to a channel.
2. Have another process read from that channel and use a map[<classification_id>]<count>, and add up sequentially

That way you're classifying stings and counting them concurrently

I'm a bit distracted right now. I don't know if this is the best solution, but I can guarantee it works.

[sick orbit]

KoltPenny's approach is the way.

[tawny canyon]

My approach is faster 🙏 but probably not best practise

[sick orbit]

Oh I was talking about for the map reduce algorithm. I kinda left that part out tho.