🔒 Import class from module in relative folder

[grave horizon]

My folder structure is
main/program.py
main/plugins/utils.py
main/plugins/plugin1/main.py

I'm using importlib.import_module( 'plugins.plugin1.main' ).my_class() to import a class from main.py in program.py. I'd like to import a class from utils.py in main.py. However, I can't see to use the same call.

I see a lot of references online to simply injecting a path into the system PATH variable to load a distant module, but I'd like to avoid that if possible and keep the environment as clean as I can. Is there any other good way to load that class?

[worn wing]

@grave horizon

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[warped dagger]

indeed, messing with PYTHONPATH is usually a smell since local imports can often be solved with a better project structure or by running submodules with python -m path.to.mod, but in your case it looks like from plugins.utils import MyClass should work

[grave horizon]

how does python "know" that the relevant "plugins" folder is two steps up? Is it iteratively searching upwards in the folder tree until it finds a "plugins" folder?

[warped dagger]

absolute imports like import foobar will search sys.path, a list of directories that python iterates through to find modules
https://docs.python.org/3/library/sys.html#sys.path

[grave horizon]

python script.py command line: prepend the script’s directory. If it’s a symbolic link, resolve symbolic links.
ah

[warped dagger]

the way you run the python interpreter determines how sys.path is configured, so in your case running python program.py prepends the parent directory of program.py to sys.path, which means all absolute imports can access program itself and plugins

or in other words, you would be permitted to write the following imports: py import program import plugins.utils import plugins.plugin1 import plugins.plugin1.main

that holds true regardless of the file you write those imports in, since sys.path is shared by all of them

[grave horizon]

so in particular this is not robust to me using program.py as an import I call from elsewhere

[warped dagger]

what do you mean from elsewhere?

[grave horizon]

for example, if I write corvus_stuff.py in the directory above program and within corvus_stuff.py, import 'this_program.program'

home/corvus_stuff.py
home/this_program/program.py
home/this_program/plugins/utils.py
home/this_program/plugins/plugin1/main.py

corvus_stuff.py:

import this_program.program

[warped dagger]

so this_program/ can be used both as a library you import, and as an application where you run the script inside it?

[grave horizon]

It's basically a library that has a if name == __main__: do a demo of the library

I think the fundamental problem is I don't have a good understanding of how to structure python code The Right Way, is what it feels like

I suppose in most sane cases, if this_program is being treated as a library, it will be installed somewhere where it is accessible in the PATH

[warped dagger]

in sys.path yeah

(not to be confused with the PATH environment variable that determines what commands you can run from the terminal)

[grave horizon]

fair

and if this_program IS in sys.path, then the import you mentioned above will work fine

[warped dagger]

but anyway, i would structure your program as a proper package with the assumptions that:

1. the parent directory of this_program/ will be in sys.path
2. none of its submodules will be run with python path/to/script.py, only python -m path.to.mod (equivalent to importing path.to.mod using sys.path and assigning that module the __main__ name)

[grave horizon]

this is really helpful. I'll try to get my head around structuring like this.

[warped dagger]

for example, if your project was structured like this: py home/ ├── this_program/ │ ├── plugins/ │ │ ├── plugin1/ │ │ │ └── __init__.py # contents of main.py │ │ └── utils.py │ ├── __init__.py │ └── __main__.py # contents of program.py │ from this_program.plugins.plugin1 import MyClass └── corvus_stuff.py from this_program import xyz these would be the correct commands for running your two entry scripts: sh /home $ python corvus_stuff.py /home $ python -m this_program (correct in that home/ is added to sys.path, so the imports stay the same)

[grave horizon]

and those __init.py__ are weirdly very important right?

[warped dagger]

__init__.py is used to mark a directory as a package, but also allows providing an "interface" to a package by defining classes and functions inside one, which lets you import stuff from the package directly: py package/ ├── __init__.py │ from package.foo import Foo │ from package.bar import Bar ├── foo.py │ class Foo: ... └── bar.py class Bar: ... ```py
import package
foo = package.Foo(1, 2, 3)
bar = package.Bar("Hello", "world!")

or:

from package import Foo, Bar```

it's standard for packages and subpackages to have an __init__.py, but nowadays it's easier to forget it because of "implicit namespace packages", where directories that don't have an __init__.py can still be treated as if they were a package

see also the official tutorial and documentation:
https://docs.python.org/3/tutorial/modules.html#tut-packages
https://docs.python.org/3/library/__main__.html#main-py-in-python-packages

if you want some project structure hints, here's two example layouts in my sys.path guide: https://thegamecracks.github.io/python/syspath.html#how-should-i-structure-my-project
an application written as an installable package:
https://github.com/thegamecracks/sqlitediff
a library written as an installable package:
https://github.com/thegamecracks/berconpy
and of course you can look at packages from popular projects:
https://github.com/psf/requests/tree/main/src/requests
https://github.com/Textualize/rich/tree/master/rich
https://github.com/pytest-dev/pytest/tree/main/src

[grave horizon]

(ina work meeting at the moment but appreciate the help!)

[warped dagger]

the whole "making stuff installable" is a separate thing from structuring packages, but if you want to learn more about that (and optionally uploading packages to PyPI):
https://packaging.python.org/en/latest/tutorials/packaging-projects/
https://setuptools.pypa.io/en/latest/userguide/quickstart.html#basic-use
https://packaging.python.org/en/latest/guides/writing-pyproject-toml/
and some extra material about src vs. flat layout, --editable mode, and entry points:
https://packaging.python.org/en/latest/discussions/src-layout-vs-flat-layout/
https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#automatic-discovery
https://setuptools.pypa.io/en/latest/userguide/development_mode.html
https://setuptools.pypa.io/en/latest/userguide/entry_point.html

[grave horizon]

clearly I should just put everything into a single 6000 line module 😄

but really, I appreciate the guidance

I feel like I have to write everything several times

• The first to figure out how to make it run
• The second to structure it so I say everything once
• The third to figure out what code goes where

[warped dagger]

i don't often start with a fleshed out idea of my project layout either, instead i begin with a basic layout, write my code, and i'll naturally find reasons to move that code around, like:

• "I have a lot of functions for handling my network protocol, let's move that into a protocol.py module"
• "My protocol.py module is getting too big and has a lot of related types, let's turn it into a package and organize those types into submodules"
• "I have exception classes in client.py shared across unrelated modules, let's have a dedicated errors.py module so they don't need to import client.py"
  try to name your modules so that it reflects specific responsibilities of your program, for example a GitHub REST API wrapper might look like: githubapi/ ├── actions/ │ ├── __init__.py │ ├── secrets.py │ └── workflows.py ├── __init__.py ├── auth.py ├── http.py ├── ratelimits.py ├── releases.py └── repos.py
  this article also has relevant points about naming modules:
  https://breadcrumbscollector.tech/stop-naming-your-python-modules-utils/

[grave horizon]

so if I'm writing http.py, and I need a function from auth.py, I would just write import auth. But should I be worried that the presence of an auth.py module might trample an installed module/package (via pip, etc) called auth? I've been naming all my files corvus_main.py, corvus_utils.py, etc., but maybe that's too conservative

[warped dagger]

since it's part of the githubapi package, import auth shouldn't work unless githubapi/ was somehow added to sys.path, such as by accidentally running python githubapi/releases.py

[grave horizon]

sure, I more mean: later on, as I'm adding features to githubapi, I realize I need a new package. It's in a pip package called auth. Now I have to rename my auth.py and go and fix everywhere in my script where I use it.

of course, maybe the answer is "yeah, you have to"

[warped dagger]

in normal use cases you'd only expect the parent directory of githubapi/ to be in sys.path, in which case you're protected from that name collision by being forced to fully qualify your import name as import githubapi.auth

but yeah, it would be bad practice to make a library consisting of a bunch of modules not inside a package: src/ ├── auth.py ├── http.py └── repos.py pyproject.toml if auth.py, http.py, and repos.py were copied straight into site-packages/, you'd need to write your imports as: py import auth import http import repos now it's no longer obvious that they are all from the same project, and it would make it much easier for a third-party library to overwrite one of your modules and break your project

[warped dagger]

to clarify, if you had the following modules that could be discovered from sys.path: auth.py githubapi/ ├── __init__.py └── auth.py there would not be a name collision here as import auth would always refer to auth.py, even if written inside a submodule of githubapi/

in contrast, if the githubapi/ directory was incorrectly added to sys.path, import auth might resolve to either module depending on the order in which their parent directory appears in sys.path

there's a somewhat common issue that comes up in this server related to the development of discord bots, where OP installs the "discord.py" library from PyPI, adding a discord/ package to site-packages/, but then they start work on a discord.py script, and once they run their script it tells them discord.Client doesn't exist or similar

in their case the command python discord.py prepended their script's directory ahead of site-packages/, meaning import discord resolves to their own discord.py module instead of the discord/ package they meant to import

[grave horizon]

I'm sorry, I'm still having trouble fulling getting it.

so I created

/corvusproj/__init__.py
/corvusproj/mymodule.py
/corvusproj/main.py

In main.py I wrote
import corvusproj.mymodule
then typed
python main.py
and got
ModuleNotFoundError: No module named 'corvusproj'

I expected this to work, because python should know about /corvusproj/ in my sys path

[warped dagger]

that's where python -m path.to.mod is necessary

python main.py added corvusproj/ to sys.path directly, which is different from adding the parent directory of corvusproj/ to sys.path

[grave horizon]

python -m corvusproj.main
run from the parent directory. I see

but I can't do
python -m .corvusproj.main
from the /corvusproj/ directory: python.exe: Relative module names not supported

so I should write all my files assuming they are being called in a python -m context, and can do imports as
import corvusproj.fully.qualified.path.to.module

[warped dagger]

let's say you had this file structure: project/ ├── pkg/ │ ├── sub1/ │ │ ├── __init__.py │ │ └── mod.py │ ├── sub2/ │ │ ├── __init__.py │ │ └── mod.py │ ├── __init__.py │ └── sub3.py └── main.py here would be the list of valid, absolute imports as you change your python command: ```py

(CWD is set to project/)

python pkg/sub1/mod.py (sys.path = ["project/pkg/sub1/", ...])

import init
import mod

python pkg/sub2/mod.py (sys.path = ["project/pkg/sub2/", ...])

import init
import mod

python pkg/sub3.py (sys.path = ["project/pkg/", ...])

import init
import sub1
import sub1.mod
import sub2
import sub2.mod
import sub3

python main.py (sys.path = ["project/", ...])

import pkg
import pkg.sub1
import pkg.sub1.mod
import pkg.sub2
import pkg.sub2.mod
import pkg.sub3

python -m pkg.sub1.mod (sys.path = ["project/", ...])

import pkg
import pkg.sub1
import pkg.sub1.mod
... # same as python main.py```

[grave horizon]

I guess this is how I should structure stuff up - have all my libraries and things people might actually care about in /project/pkg/, and have my "demo scripts" in /project/

[warped dagger]

yup, that would be the simplest layout you could do that doesn't require python -m pkg.mod, setting up pyproject.toml for installation, or PYTHONPATH/sys.path.insert(0, ...) hackiness

[grave horizon]

So now I have the instinct that in sub2/mod.py I could either write
import pkg.sub1.mod
res = pkg.sub1.mod.do_something()
or
import pkg.sub1.mod as sub1
res = sub1.do_something()

The second one seems tempting since if I change project structure, or someone wants to rip out some of my code for their stuff, it might involve less fixing?

[warped dagger]

if you dont like putting said scripts in the project root, moving all of it to a subdirectory is fine too: project/ ├── app/ │ ├── pkg/ │ │ ├── __init__.py │ │ └── ... │ ├── demo1.py │ └── demo2.py ├── .gitignore ├── CHANGELOG.md ├── LICENSE └── README.md sh /project $ python app/demo1.py /project $ python app/demo2.py

[warped dagger]

if it's possible that sub1/mod.py could be removed, i would suggest exposing its contents in sub1/__init__.py and have external code access it from sub1 directly, e.g. ```py

pkg/sub1/init.py

from .mod import MyClass

pkg/sub2/mod.py

from pkg.sub1 import MyClass``` that way sub2 doesn't rely on the presence of sub1/mod.py at all

it also might be worth prefixing mod.py with an underscore as a hint that end users shouldn't need to import it, which is what some of the projects i linked previously like rich does

[grave horizon]

that makes sense

[warped dagger]

httpx is a pretty aggressive example of this convention, literally every submodule in their package has that underscore prefix
https://github.com/encode/httpx/tree/master/httpx
(or rather all their top-level submodules have the prefix, but they skip it in _transports's submodules)

[grave horizon]

I suppose that also goes along with the concept of public features of the sub1 subpackage which are exported from __init__ and private ones that are used among the scripts in the /sub1/ directory but not exported out

Anyways, this was tremendously helpful, thank you so much!

[worn wing]

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.