I'm in a situation where I may initially want the top 10 values, but later on want the top 100, and then later the top 10, and so on. I don't think I can re-call reserve with different parameters on the same key, so I'm thinking I will have to create functionality to do the following
// lock, if true, execute. If false, go on with your life (distributed system) // read the current values with count // delete the key // reserve the new key with the new values // write the old values in the new key
I'm wondering if someone else has a better way to do this. There's also the problem that in the small interim between deleting and rewriting, that clients could read the wrong value.
#What is the best way to recreate a TopK key?
atomic rune
lament perch
I would just do a TOPK with the largest size I'd expect to need, and then just use TOPK.LIST and chop the list down to what you actually need.
atomic rune
Is there a performance downside to that?
I will have 1000's of keys. We want a size of 10 for 90% of them. When a custom configuration (semi rare) comes in, then we want to allow for more, up to 100.
so as you recommended, that would work if I just got the top 100.
lament perch
What is the maximum number of keys you'd have to list?
atomic rune
super prompt response btw, I just saw you on youtube and saw the discord shout out. Appreciate the quick help.
Max keys would be 100K
atomic rune
TOPK.INCRBY mykey item1 5 item2 10 item3 15 item4 10000
In the above example, I will have 100K "mykey"s at max.
I will have millions of item1, item2, item3
lament perch
So you'll want to list up to 100k keys?
atomic rune
Ah I see the confusion. I will only ever want to list
10-100
atomic rune
Gotcha, do you expect there to be a performance hit? I see the performance degrades by a factor of k
lament perch
It will not be substantive.
atomic rune
I see, at 100 it's too small to matter.
lament perch
WIth the top-k there's a min-heap in Redis so you're just getting the 100 records from the min-heap
It's more performant than having to add to multiple top-k's in all likeihood (e.g. if you want to do one with the top 10, 20, 30, etcl . . .)
If your top k set was only going to have a cardinatily of 100k, I would actually recommend using a sorted set instead, as it's more accurate, and that's not a scale that's likely to any issues with your memory footprint (unless of course you have absurdely large member strings)
atomic rune
two reasons I wanted to go with topK instead of sorted set. I really like the fact there's a decay built in. Alternatively I'd have to figure out how to rotate sets so that as my topK changes, I can change as well.
lament perch
but if you have a cardinality in the millions, then yeah you're just better off with a top-k for sure.
atomic rune
The second reason is the high cardinality/space concerns.
lament perch
the decay is just a probablistic mechanism, it doesn't do any sort of member expiration or anything
atomic rune
understood, but my understanding is if the topK changes, it will adjust right?
lament perch
Yeah, so as you insert records into Redis, the heavy keeper will (as it encounters conflicts in it's map) will probablistically decay items in there.
atomic rune
so one question, if I my incrBy tps is regularly something like:
100K item1
50K item2
and later it starts looking like
50K item1
80K item2
and my topK only lists the top 1, AND the count is still higher for item1 overall, will it still show item2?
ideally it shows item2, but that may take playing with the decay to probalistically "delete" count from item1 fast enough.
i think I answered my own question. decay directly affects count. So whatever count is highest, will be shown.
is there documentation I can see to read on how decay operates in redis?