I'm trying to work on a golang project, and I need to store a hash map that attaches a key to a value. I don't need it to persist across restarts, and I need it to be able to be grown or shrunk dynamically (add/remove by key). I need to be able to query it from any function in the program's code while it is running. Is that possible?
#Need to store globally available hash map
high kindle
clear crane
why cant you use sync.map? or the map data type
lavish plank
they mean use a global variable (generally not great to have but if that’s your need, that’s your need)
high kindle
they mean use a global variable (generally not great to have but if that’s your ...
The reason is that i have a webserver, one endpoint handling requests to add/remove to the list and another endpoint that makes use of the list
lavish plank
ok so if you need it concurrent across go routine (like would happen in a web server) the map indeed need to be either protected by a mutex or use an existing type if suitable (like sync.Map though that may or may not be suitable)
there are many, including my own $[fortio.org/smap]
mellow falconBOT
high kindle
ok so if you need it concurrent across go routine (like would happen in a web se...
And that is able to be used globally?
high kindle
The reason is that i have a webserver, one endpoint handling requests to add/rem...
Both funcs in same file
I could use sqlite but the data is transient so id rather not
lavish plank
you could put an instance in a package yes eg
package global
import "fortio.org/smap"
var Map = smap.New[YourKeyType, YourValueType]()
then everywhere else in your code you can use
global.Map.Set()/Get() etc
Id rather not introduce a dependency, is there a reason sync.Map won't work?
lavish plank
well it says
The Map type is specialized. Most code should use a plain Go map instead, with separate locking or coordination, for better type safety and to make it easier to maintain other invariants along with the map content.
and
The Map type is optimized for two common use cases: (1) when the entry for a given key is only ever written once but read many times, as in caches that only grow, or (2) when multiple goroutines read, write, and overwrite entries for disjoint sets of keys.
will it be disjoint set of keys ?
also it's ... not using generics so it has "any" instead which requires casting values etc
high kindle
well it says
> The Map type is specialized. Most code should use a plain Go map ...
Okay so what i have is
String key corresponds to string value
Occasionally the value will be gotten, the value will be set every minute, and every 5 minutes the key will be removed if not updated within that timeframe
But this may be happening for multiple items
lavish plank
hmm so it's more like an LRU ?
high kindle
LRU?
lavish plank
Least Recently Used items evicted from a cache
high kindle
No no
Inactive items removed
Not based on recency
If theres no update with 5 mins, client is deemed dead
So it should be removed
lavish plank
well you say "no no" but ... that's one of the typical LRU eviction (either last read or last write) but whichever - if you have some time thing that does mean you would iterate over and delete entries
high kindle
The entry that hasnt updated in 5 mins yes
As in, been set
Doesnt matter if gotten within the 5 mins, if it hasnt been set again
But the value usually wont change
So it may be set back as same value
lavish plank
yes so you need to record when it was last set, in the value (or in your data structure) as well as have periodic eviction (or eviction at time of get/insert)
high kindle
Correct, eviction at time of insert
So whenever its updated a 5 minute timer for that item is reset
Well, doesnt need to work exactly like that, but in practice anyway
It doesnt need to strictly be 5 minutes
Whatever is less performance impacting
lavish plank
that's what I do here for instance https://github.com/fortio/tsync/blob/main/tsnet/tsnet.go#L225-L239
high kindle
I just dont want it growing forever
Im also wondering if sync.map fits my usecase or if i should use rwmutex map
high kindle
that's what I do here for instance <https://github.com/fortio/tsync/blob/main/ts...
Thanks
lavish plank
pretty sure you need a rwmutex like mine but ... you can try sync.Map too - my point being that correctness is likely more important that concurrent access / time to acquire a lock; unless you'd do many 10s of thousand per second
high kindle
Oh no way
rwmutex definitely
Im at most gonna have 10 sets and 1 get per second
Maybe double or triple that
Im very new to golang and the syntax is confusing
I usually work with java
But ill try to get this working
high kindle
that's what I do here for instance <https://github.com/fortio/tsync/blob/main/ts...
So i just need to periodically call some func like this to perform cleanup?
Maybe i could call it in the http route where im getting the value of this map
Or would that be too frequent
lavish plank
well clean up every time is maybe a bit much (if you look the link above, that clean up is called inside a ticker (periodic))
high kindle
@lavish plank is there a way to store a 2nd value to lookup by?
so you look up by 2 keys and get 1 value
lavish plank
you can have the value a pointer and set 2 keys I suppose?
lavish plank
but you'd need to clear both keys
high kindle
then store that
lavish plank
well if you do know both, I thought you meant access 2 different ways
lavish plank
I think struct {ip string, port int} is still comparable
!go return 42
hmm bot not here
high kindle
I think `struct {ip string, port int}` is still comparable
not really necessary
im already validating its an int prior
seems simple enough to split by :
well im gonna try implementing this now
high kindle
@lavish plank will this work?
type saltEntry struct {
salt string
timestamp time.Time
elem *list.Element
}
var (
saltMap = make(map[string]*saltEntry)
saltMapMutex sync.RWMutex
lruList = list.New()
lruMutex sync.Mutex
lruTTL = 5 * time.Minute
)
value doesn't need to be pointer to entry, and key probably would be nicer as typed ip/port
also you would want to Expose the entry and some function as you don't want to have all the callers having to remember to do the correct locking - what's that list?
lavish plank
put these function here then (anyway you need as with lowercase it's all private)
high kindle
alr
lavish plank
by here I mean same package you put the above 🙂 not here here
high kindle
its in same file
lavish plank
glad! (don't forget to run with go run -race btw)
high kindle
Whats that do lol
lavish plank
it'll tell you if your sync is correct
high kindle
I just go build and ./foo
lavish plank
yes so for test / dev mode, do
go test -race ./...
and ditto go run -race .
high kindle
Whats ditto
Im running go test -race ./...
Do i need to interact with my map thing or is it gonna do that for me
lavish plank
Whats ditto
means "and same"
well you do if you you don't actually have much tests
so run it manually with go run -race . and use the app
but you should have e2e automated tests too
high kindle
Whats the -race do?
Race conditions?
Its supposed to throw loads of errors if something is wrong right
When i add, or when its removed?
I had no issues when adding
But idk im not doing a lot of requests at once
I'll try that
lavish plank
yes it will complain
you can remove 1 lock / unlock pair to see
(confirm it is indeed finding these bugs)
high kindle
you can remove 1 lock / unlock pair to see
Nope, no issues
lavish plank
well that’s not good
high kindle
?
It didnt complain isnt that the point
I did get an error about invalid memory address when i added like 30k items to the list at once
But it was only for one of them
And i was making all those requests from localhost, my bandwidth would never allow that much
lavish plank
i was saying to remove (temporarily / comment out) some locks to see the race errors