🔒 Is Combining a Queue with a Set a good idea

[low shoal]

@ocean anchor

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[bronze dirge]

you cannot send file that are not image/video etc.

(assuming that was the reason

!paste

[low shoal]

Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

[ocean anchor]

I had the idea of combining a queue with a set for something like URL enumeration. I will be sure my code gets dumped on this site first luckily it's very small.

[bronze dirge]

err... some more explanation?

[ocean anchor]

Yeah I sure can. But basically I had the idea of making it so that a queue can only carry unique items with it

https://paste.pythondiscord.com/247A

Ok Finally I was able to do this. I like to have proper static typechecking and this is how I usually write my code incase something screws up.

Hopefully now this answers your questions.

[bronze dirge]

I mean, the only downside of combining is just it's unordered

or computational if you done it other way

(which shouldn't be too big of an issue

[ocean anchor]

yeah it has to do with the was a hash map works in C if I remeber correctly. But this would be a good replacement to something like an lru_cache for example.

[bronze dirge]

let me see if I could write something that could take more advantage on set while maintaining order

oh nvm, you are doing it good enough

just one thing

[ocean anchor]

Yeah go for it.

I am willing to turn this into a python library on one of my other aliases if other people really want it. I also made asyncio versions.

[bronze dirge]

-   def _put(self, item: T) -> None:
-         if item not in self.__set:
-             self.queue.append(item)
-             self.__set.add(item)
+   def _put(self, item: T) -> None:
+       prev_len = len(self.__set)
+       self.__set.add(item)
+       if len(self.__set) > prev_len:
+           self.queue.append(item)

you can try this which take more advantage on the set probably

but you have to do some preformance test

[ocean anchor]

Yeah I am aware. I think I know how to make that better I was trying to make it so that the item would check if it's already apart of the set and then append it but it seems as though you just optimized it futher so I thank for that.

[bronze dirge]

just a thought because I think that would take advantage on more C level computation which is faster than python

Don't think that would give any optimize if its a compile language or maybe other language

[ocean anchor]

I agree. I was originally thinking about combining deque with a set in CPython but I have not undertaken the project yet due to lack of intrest, willpower or since I have a job where I work for half the day. I would've written it in cython but I feel like it would hinder it's own performance if I did that.

I could write a concept in rust though but even I struggle with rust.

[bronze dirge]

It shouldn't be too much I guess?

like it should be completely fine with like a few thousand item in queue and 2k in and out/s

[ocean anchor]

I'm gonna try writing this in PyO3 First since I'm trying to get good at rust and then I'll write a cython version and C Version and then perform some benchmarks.

[low shoal]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.