🔒 How to expand a path with wildcards in pathlib?

[rotund igloo]

Lets say i want to work with any CLI path input e.g. one or more file(s) and/or dir(s) to my py-code.

If i get a path like e.g. path/to/some/directory/**/*.txt and i cannot change the string before it goes in my module, how could i work though this just with pathlib and buildins?

I discovered the hard way pathlibs (r)glob methods cannot work such wildcards because the needed input there is split. (I know the glob.glob can do it but thats not wanted here!) So which way could i make this work?

[formal wagon]

@rotund igloo

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[rapid sundial]

to clarify, are you saying you want to, in that example, return all the texts files in every subdirectory of path/to/some/directory?

i'm thinking you could iterate over [f.path for f in os.scandir(path) if f.is_dir()] and take all the text files within it

[rotund igloo]

Yes, all files with a extension, in this example txt.

And with pathlib or buldins. Not OS or anything else.

It's just a restriction i set, not a codewise need. Just want to know if its possible because i stumpled over this with Path.glob().

[rotund igloo]

No other way?

[pulsar bloom]

Can you explain more about the need for splitting the input? Why can't you do Path().rglob("path/to/some/directory/**/*.txt")? Why is glob.glob not wanted?

@rotund igloo

rglob works just fine with things that DON'T have any wild cards in them, so it should be fine to stick all paths including ones without wild cards just into the .rglob() part.

[rotund igloo]

As i said: rglob and glob in pathlib cannot do it.

In [249]: for item in pt().rglob("/home/olli/**/*.txt"):
...:     print(item)
...:
---------------------------------------------------------------------------
NotImplementedError                       Traceback (most recent call last)
Cell In[249], line 1
----> 1 for item in pt().rglob("/home/olli/**/*.txt"):
2     print(item)

File /usr/lib/python3.10/pathlib.py:1045, in Path.rglob(self, pattern)
1043 drv, root, pattern_parts = self._flavour.parse_parts((pattern,))
1044 if drv or root:
-> 1045     raise NotImplementedError("Non-relative patterns are unsupported")
1046 selector = _make_selector(("**",) + tuple(pattern_parts), self._flavour)
1047 for p in selector.select_from(self):

NotImplementedError: Non-relative patterns are unsupported

Thats why i said the input would "need to be splitted":

In [250]: for item in pt("/home/olli").rglob("**/*.txt"):
...:     print(item)
...:     
/home/olli/.config/falkon/profiles/default/adblock/customlist.txt
# ...etc...

The same with Path()glob() as the ** wildcard makes it recursive.

[pulsar bloom]

Oh, I see, absolute paths cause a problem

Okay, so what about splitting it on the first *?

[rotund igloo]

Yes. But foolproof...

I must admit some of my trys with different input variants did not really work out.

[pulsar bloom]

Hmm... looks like using rglob actually puts in an automatic "**", so if you do Path("/home/olli").rglob("*.txt"), it'll return "/home/olli/**/*.txt" which wouldn't be what you want in that case.

Oh, that is what rglob is, glob with **/ at the front

[rotund igloo]

Yes, a recursive shorthand to glob.

[pulsar bloom]

Do you have edge cases that Path("before the first *").glob("* and everything after") wouldn't work for? You said that isn't fool proof, and I believe you, but are we talking about a limited number of edge cases we can address?

Okay, just realized it has to split at a folder name

What about just splitting the absolute part of the URL? Doing:

Path("/").glob("home/olli/**/*.txt")```
if it starts with a `/`

[rotund igloo]

I cannot really say why my trys banged sometimes and then again not. As it was some days ago i don not have my code trys anymore or i would have shown it.

Hm.

[pulsar bloom]

You could try to grab as many folders as you can without hitting wild cards, but that may not be needed at all with that

[rotund igloo]

Strangely it worked just now in ipython.

The left side wants a starting point and the right side something to look for. Means: Path("left side").glob("right side")

Who the heck coded this for pathlib?

[pulsar bloom]

Why is that odd to you? "/home/olli/**/*.txt" converts to Path("/").glob("home/olli/**/*.txt") which both preserves the order of the original and also makes sense as a method acting on the Path("/") path object to try to find things within that path.

[rotund igloo]

This may be but you cannot put a whole path in one side and i think thats a bit cumbersome.

Now i need to split this on the right position...hm.

[pulsar bloom]

Fair enough. Does glob.glob do that though? You didn't say why you didn't want glob.glob

[rotund igloo]

Just for curiosity how this would work only with pathlib. I found it strange ancient glob can do it, and the new(better?) pathlib (r)glob is unable.

I thought this cannot be, there must be a way i dont see.

[pulsar bloom]

You still don't really need to do any splitting though, just grab the one / in the case of absolute paths:

for pathtxt in userinputs:
    if pathtxt[0] == '/':
        pathgen = pathlib.Path('/').glob(pathtxt[1:])
    else:
        pathgen pathlib.Path().glob(pathtxt)```
Or could even leverage the `.is_absolute()` method.

[rotund igloo]

Hm, true.
.is_absolute() returns a True if he finds a WIN drive or in *nix a slash, or?

[pulsar bloom]

I mean, just because its newer doesn't mean every single use-case will be easier and more intuitive to use. In this case absolute directories don't work because Path().glob(absolute) isn't a method acting on Path() which when blank is just your current working directory.

[pulsar bloom]

Yes, but if you wanted to support win drives, you'd have to do it in a more complicated way than just:

    if Path(pathtxt).is_absolute():
        pathgen = pathlib.Path('/').glob(pathtxt[1:])```
Since you have to deal with cases such as the slashes going the other direction or UNC paths `\\something` or multiple character starts like `C:\`

It may make more sense to see if glob.glob will work and then just convert the outputs from glob.glob to Path objects if you want the extra path functionality in that case

[rotund igloo]

Ok. SO this would be one way what you posted...

In [5]: pt("home/olli/**/*.txt").parent
Out[5]: PosixPath('home/olli/**')

In [8]: print(*pt("/home/olli/**/*.txt").parents)
/home/olli/** /home/olli /home /

In [9]: print(*pt("/home/olli/**/*.txt").parts)
/ home olli ** *.txt

Thats what i get with some pathlib....

Funny. The easiest way is still the good old glob module. Eats it simply up. ARGH!

[pulsar bloom]

Yeah, using the last entry of .parents as your starting point would be a smarter way to handle all those edge cases

or the first entry of parts

But yeah, its all kinda hacky compared to glob.glob

[rotund igloo]

My question was just to see what pathlib can actually do with diff path thrown in, not a real usecase.

As said i stumbled on it on found the limitation strange.

Now we know. Thanks a lot you both! 👍🏼

!close

[formal wagon]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.