🔒 Optimising the creation of 50000 objects

[slim parcel]

@unkempt ferry

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

Please react with ✅ to upload your file(s) to our paste bin, which is more accessible for some users.

[unkempt ferry]

@rustic pumice got a notification from @slim parcel saying that a html attachment inst allowed? should i put it on the paste bin?

[rustic pumice]

if it'll fit, yeah

otherwise there's always gist I suppose

[unkempt ferry]

no clue if it can fit, helix was crapping out when i was trying to remove my username

yeah 5mb is insane

[rustic pumice]

or take a screenshot of the flamegraph and post it here

[unkempt ferry]

not sure why it says 9 seconds now, i swear it was 5 seconds without whatever this is

python pastebin has been loading for quite a while now

[rustic pumice]

some overhead is expected when profiling

[rustic pumice]

which function does the creation of all those objects happen in? update_file_list?

[unkempt ferry]

yes

i profiled only that part

[rustic pumice]

can you link your code? I kinda jumped into the middle of this

[unkempt ferry]

https://github.com/NSPC911/rovr/blob/master/src/rovr/core/file_list.py#L98-L219
the main part is lines 134-144 which makes the class, 122-124 which scans the directory (leading to a function at https://github.com/NSPC911/rovr/blob/master/src/rovr/functions/path.py#L126-L167 ), and line 163, which actually adds the options

i did a quick fix to it to add the pyinstrument profiling and remove duplicate lzstring mention, reminder that it will open the browser every time the file list gets updated

ill be heading off shortly, laptop battery running low + it's quite late into the night already

[rustic pumice]

same here 😄
parting thoughts: maybe you could cache objects? that way at least the cost is one-time, so on reloading the directory you don't have that problem
or lazy-load somehow? like only load the N objects +/- of what is being displayed
I haven't looked at your code or app that much so sorry if these don't make any sense

[unkempt ferry]

nope, i was wanting to make it interactive unlike other tui file explorers, so you can use your mouse to scroll, theres a scrollbar to scroll (all thanks to textual)

reloading the directory has no issues, unless something changed in the directory, then it recreates all the objects again

so if you were to fuzzy search for a file, no problem, 50k items takes just under 0.4 seconds to mount

okay, ill check in once i wake up tomorrow

[worthy surge]

If it's memory usage have you considered reducing the space of each object using __slots__ ? Not sure that will help but it shouldn't be too hard to test.

[static crown]

__slots__ should speed things up a bit, too

[slim parcel]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.