2

The problem

I've found dozens of articles and tutorials about the basics of using import in Python, but none that would provide a comprehensive guide on setting up your own Python project with multiple packages.

This is my project's structure:

  • codename/
    • __init__.py
    • package1.py (has class1 and is a script)
    • package2.py (has class2)
    • package3.py (has function1 and is a script)
  • test/
    • __init__.py
    • test_package1.py (has unit tests for package1)
    • test_package3.py (has unit tests for package3)

How do I setup my imports to have the following requirements met (and do all of them make sense?):

  1. class1, class2 and function1 are in namespace codename, i.e. this works:

    import codename
obj = codename.class1()
codename.function1(obj)

  2. they may be imported the same way using from codename import * or from codename import class1

  3. function1 can easily access class1 (how?)
  4. package1 and package2 are executable scripts
  5. so are test_package1.py and test_package3.py
  6. tests are also executable via python -m unittest discover
  7. scripts are also executable via python -m codename.package1

For some reasons I'm having issues with having all of these met and when I try to fix one issue, another one pops out.

What have I tried?

  • Leaving codename/__init__.py empty satisfies almost all of the requirements, because everything works, but leaves names like class1 in their module's namespaces - whereas I want them imported into the package.

  • Adding from codename.package1 import class1 et al again satisfies most of the requirements, but I get a warning when executing the script via python -m codename.package1:

    RuntimeWarning: 'codename.package2' found in sys.modules \
after import of package 'codename', but prior to execution of \
'codename.package2'; this may result in unpredictable behaviour

which sort of makes sense... Running the script via python codename/package1.py functions, but I guess I would probably like both ways to work.

  • I ran into an answer to a similar question that stated that internal modules should not also be scripts, but I don't understand why we get the -m switch then? Anyway, extracting the mains into an external scripts directory works, but is it the only canonical way of setting all of this up?
Community
  • 1
  • 1
mikosz
  • 393
  • 1
  • 8

1 Answers1

1

  1. you'll need to add the parent directory of codename/ to the PYTHONPATH environment variable (or write/use a setup.py file, or modify sys.path at runtime)

  2. You'll need to import all names that you want to export in codename/__init__.py

  3. from .package1 import function1 if you write/use a setup.py file, otherwise from codename.package1 import function1

  4. You should use a setup.py file for scripts/executables since it makes everything much cleaner (and you'll need a setup.py file sooner or later anyway)

  5. (and 6.) I would suggest using py.test it will find all tests for you automagically (and can run them in parallel etc.)

  6. That should work out-of-the-box, but if you've written a setup.py then you can run them from anywhere (and on any platform) as just package1.

thebjorn
  • 26,297
  • 11
  • 96
  • 138
  • 3. this doesn't work. I've added a setup.py file, used both install and develop and I'm getting `No module named '__main__.package1'; __main__ is not a package` when running the script normally and `Cannot import name 'package1'` on the `from .package1 import...` line. But anyway, by looking through setup.py's documentation I guess it's a standard procedure to have the scripts separated from the modules and not use -m: is that true? – mikosz Feb 21 '17 at 12:28
  • You'll need to use the `entry_points` feature of `setup.py`, e.g. https://github.com/thebjorn/pydeps/blob/master/setup.py (the syntax is command_name = importable.dotted.path:function_name). – thebjorn Feb 22 '17 at 01:45
  • Ahh, now it works! Thanks a bunch. I've also found a very useful article on setting up executables in a python project: https://chriswarrick.com/pl/blog/2014/09/15/python-apps-the-right-way-entry_points-and-scripts/ – mikosz Feb 22 '17 at 10:34
  • Nice resource. Thx. – thebjorn Feb 22 '17 at 10:44