🔒 Imports and packages

[steep jacinth]

How do you guys solve this (I assume very common) problem?

I have a project with a few different functional units that I break into packages. I want to develop and test the packages one by one.

I end up with a structure something like

   package
   main.py
   subpackage1/
      module1.py
      module1_functions/
      module1_utils/
   subpackage2/
      module2.py
      module2_functions/
      module2_utils/```


If I structure the imports in module1.py as **import module1_functions.functions** then I can run module1.py to develop it.  But it will break when I'm finished and want to run from main.py.

If I structure everything as **import subpackage1.module1_functions.functions** then I can run the finished version, but not the module alone.  I have to add something like a test.py as a sibling to the main.py and have to click backwards and forwards between the file I'm actually editing and my test.py to develop.

What am I doing wrong?

[coral mural]

@steep jacinth

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[radiant turtle]

Can you put a snippet of your code?

[steep jacinth]

I've reduced it down to a toy example. I'm literally working with the structure you see there (as in I have made this example in vs code). The only difference is I have some print_hello_world functions to import in the module_functions/utils folders.

[glossy steppe]

Are you just asking how to import the code in the files or am I missing something?

[steep jacinth]

My issue is that I want to be able to run the package on it's own to develop, and then run it as part of the overall program called from main.py in production. module1.py needs to import functions from module1_functions. I can structure the imports so that they work when I run module1.py directly, or when I run main.py which calls module1.py. But I don't know how to structure them so they work in both cases. However I structure the imports, they will break in one of the cases.

[viscid sparrow]

the way I did this myself, which imo is ugly was by using try/except for the imports.

try:
    import subpackage1.module1_functions.functions
except Exception:
    import module1_functions.functions```
the more proper way is to package your program as an actual library and import it yourself that way you can reference your modules via a normal method and also works for importing things from upper level folders too

[glossy steppe]

Another way could be to try if and else though I just have to think of a way to it , though I prefer the method above

[steep jacinth]

Oh that's interesting. I hadn't thought of that. RE: packaging and importing as a libary, do you mean like a pip install -e? Would you do that for each package separately? Maybe that's the piece I need more detail on

I'm happy to stick to the "proper" way. But I think I'm missing some of the steps to fully understand it lol

[viscid sparrow]

yes the pip install -e and do the whole application then you can just do imports like from myprogram import module1_functions.functions no matter what

and personally I dont fully understand it yet myself nor do I feel I am ready for it so I havent done it the 'proper' way. but once I am done testing the sub modules I just revert their imports to how they should function at the root level as opposed to the try/except stuff

[steep jacinth]

So then when you are developing module1, can you just run module1 from that file window (i.e. the run button in vscode when you are inside the module you are developing)?

[viscid sparrow]

yes

[steep jacinth]

The way I've solved it so far is having a test.py sibling to main.py that just targets the module I am developing, but that is also ugly and slow to work with

[viscid sparrow]

though personally I havent setup much of any unit testing over all, this is what I do when directly and manually testing modules as I create them but the principle is the same

[steep jacinth]

I would love to see a guide/official documentation on how this is supposed to be done if one exists

[viscid sparrow]

iirc the packaging and importing is the 'official' way or one of them. I was handed a guide for this recently when I had a similar issue let me see if I can find that link again

[steep jacinth]

It isn't really unit testing. Just a test.py with a single line to call the right module (as if you navigated to the module itself and hit run - but with all the import structure in the right place)

[steep jacinth]

That would be really helpful. Thanks!

[viscid sparrow]

https://pip.pypa.io/en/stable/cli/pip_install/#install-editable
https://packaging.python.org/en/latest/tutorials/packaging-projects/

[fair jewel]

looks like you're missing init files?

[viscid sparrow]

ah for this I do it on a module by module basis using the main() function with the needed setup. because generally I havent needed to retest something I have already gotten working, well until today but that was for other reasons.

__init__.py would be another requirement for this to be working as well but since it seems it works otherwise they have those already. its just a matter of where they are trying to do imports from

[fair jewel]

no, you can import from sibling files in the and directory, one directory to and you need an init or the directory take recognized as a module

[viscid sparrow]

yes I know and that is not what we were referring to. it was about the methodology of importing modules universally depending on what directory you are trying to run them from for testing.
In their example

   package
   main.py
   subpackage1/
      module1.py
      module1_functions/
      module1_utils/
      test.py

if I am wanting to test module1_functions from main.py then it is from subpackage1.module1 import module1_functions but if I want to test from test.py then I would need to from module1 import module1_functions

[steep jacinth]

I do have __init__ files. I didn't put them in my example for space reasons

It sounds like pip install -e is the way to go. I'll try it with a simple example tomorrow. Thanks ❤️

[viscid sparrow]

just to clarify what I did for my use case

└── MyProject/
    └── main.py/
        └── utilities/
            ├── handlers.py/
            │   ├── try:
            │   ├── import utilities.db_handlers as db_handlers
            │   ├── except Exception:
            │   └── import db_handlers as db_handlers
            └── db_handlers.py```
now I can run `main.py` OR `handlers.py` directly for testing and both function the same way when testing the handlers.py module and without having to do any changes

but I would agree that pip -e is the way to go functionally

[steep jacinth]

Yes that seems like a good second best if installing as editable fails me

It's neater than mine anyway. I lose a lot of time clicking between files

[light scroll]

have you checked out uv and its workspaces concept? https://docs.astral.sh/uv/concepts/projects/workspaces/

[steep jacinth]

Ok I couldn't wait lol. I added a pyproject.toml and venv to my toy example and pip install -e . I then swapped all imports to absolute references from the top level package e.g. from package.subpackage... . It now works whichever file I run. So I guess this was the way all along. I'll try it with my real project tomorrow. Thanks all and @viscid sparrow especially.

[steep jacinth]

I haven't. I may look into it later, but more tools seems like overkill to manage this. It's only a couple of steps above Hello World in complexity.

[light scroll]

sure, but i have gotten so used to uv that i use it for projects of any size, to manage the venv, dependencies, package installations and even my python installation that i use for my projects

[coral mural]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.