24

Since mid 2022 it is now possible to get rid of setup.py, setup.cfg in favor of pyproject.toml. Editable installs work with recent versions of setuptools and pip and even the official packaging tutorial switched away from setup.py to pyproject.toml.

However, documentation regarding requirements.txt seems to be have been also removed, and I wonder where to put the pinned requirements now?

As a refresher: It used to be common practice to put the dependencies (without version pinning) in setup.py avoiding issues when this package gets installed with other packages needing the same dependencies but with conflicting version requirements. For packaging libraries a setup.py was usually sufficient.

For deployments (i.e. non libraries) you usually also provided a requirements.txt with version-pinned dependencies. So you don't accidentally get the latest and greatest but the exact versions of dependencies that that package has been tested with.

So my question is, did anything change? Do you still put the pinned requirements in the requirements.txt when used together with pyproject.toml? Or is there an extra section for that in pyproject.toml? Is there some documentation on that somewhere?

sinoroc
  • 18,409
  • 2
  • 39
  • 70
Bastian Venthur
  • 12,515
  • 5
  • 44
  • 78
  • My understanding is that information that used to go in `setup.py` is now split between `pyproject.toml` and `setup.cfg`. – chepner Nov 20 '22 at 14:37
  • 4
    I don't think the setup.cfg is much needed anymore. Maybe some 3rd party packages still insist on putting their config there, but regarding packaging, all the things from setup.cfg can be moved to pyproject.toml. – Bastian Venthur Nov 20 '22 at 14:44
  • I vaguely remember earlier this year trying to move somethings from `setup.cfg` to `pyproejct.toml`, and `setuptools` didn't recognize or find it. I agree that moving things is the *intent*, but I thought it was still a work-in-progress. – chepner Nov 20 '22 at 14:50
  • 1
    I opened a feature request for pip to support requirements in pyproject.toml: https://github.com/pypa/pip/issues/12100 – Stefan Jun 23 '23 at 23:40

5 Answers5

10

Quoting myself from here

My current assumption is: [...] you put your (mostly unpinned) dependencies to pyproject.toml instead of setup.py, so you library can be installed as a dependency of something else without causing much troubles because of issues resolving version constraints.

On top of that, for "deployable applications" (for lack of a better term), you still want to maintain a separate requirements.txt with exact version pinning.

Which has been confirmed by a Python Packaging Authority (PyPA) member and clarification of PyPA's recommendations should be updated accordingly at some point.

Bastian Venthur
  • 12,515
  • 5
  • 44
  • 78
7

This is the pip documentation for pyproject.toml

...This file contains build system requirements and information, which are used by pip to build the package.

So this is not the correct place. Looking at the side bar we can see there is an entry for Requirements File Format which is the "old" requirements.txt file

OranShuster
  • 476
  • 2
  • 12
3

So my question is, did anything change? Do you still put the pinned requirements in the requirements.txt when used together with pyproject.toml? Or is there an extra section for that in pyproject.toml? Is there some documentation on that somewhere?

Nothing changed. Pinned dependencies do not belong in the project's packaging metadata, which means that pinned dependencies do not belong in the dependencies list of the [project] section of the pyproject.toml file. As of today there is no standardized section of pyproject.toml that I know of for pinned dependencies. This document even though quite outdated still applies in the spirit: "abstract dependencies" belong in the package metadata (for example in the [project] section of pyproject.toml) and "concrete dependencies" can be listed in a requirements.txt file if needed (or in any other file that serves a similar purpose like Poetry's poetry.lock, or Pipenv's Pipfile.lock to name only two of them, depending on the project's development workflow and tooling).

sinoroc
  • 18,409
  • 2
  • 39
  • 70
2

Context

I found this SO question "Is requirements.txt still needed when using pyproject.toml" and have a different use case compared to the one described by the above question. Instead of opening an extra SO question with the same title, I decided to add this answer as kind of a "long comment", supporting users in similar situations.

  • In my use case I do not install my code as a library/package. Therefore, I do not need both, pinned and unpinned versions.

  • Unfortunately, the option --only-deps has not been implemented by pip, yet.

  • I already got rid of extra config files for pytest, pylint, ... and use pyproject.toml as a central configuration file in combination with a GitLab CI pipeline (gitlab-ci.yml). => Why should I still use an extra, redundant requirements.txt file if not only for historical reasons?

My Solution:

I got rid of the extra requirements.txt by defining the requirements inside pyproject.toml and installing them with

pip install -e .[dev]

or just

pip install .[dev]

The corresponding sections in pyproject.toml (min example):

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = 'my_dummy_project_name'
version = '0.0.1'
requires-python = ">=3.11.0"
dependencies = [
    'cryptography==41.0.1',
]

[project.optional-dependencies]
lining = [
  'pylint==2.17.4'
]
formatting = [
  'black[d]==23.3.0'
]
dev = ['my_dummy_project_name[linting, formatting]']

On the path to that solution I faced several issues.

a) Since I do not want to install my code as a package but only its dependencies, I first tried without name and version. However, those properties are mandatory if one wants to use the install commands. Without name and version you'll get corresponding error messages.

b) I also tried without specifying a build-backend. Depending on the python version, you will get an error message and/or setuptools will be used as default build-backend. The setuptools has the disadvantage, that it creates an unwanted directory ./src/micat.egg-info. My workaround of specifying hatchling as build-backend solved all those issues for me (even if I currently do not explicitly need hatchling).

c) Python console commands like pytest, pylint might not be found by the GitLab pipline if they are installed this way.

Instead of

pytest src

you might need to use

python -m pytest src

(If you know how to fix this, e.g. with some hatchling option, please let me know.)

d) If you prefer a shorter variant

pip install .

you could reorganize pyproject.toml and list all your requirements in a single dependencies list. However, I would suggest to keep the runtime- and dev-time- dependencies separated.

e) If you would prefer the even shorter variant (similar to npm install)

pip install

see feature request https://github.com/pypa/pip/issues/12100

f) An alternative approach would be a custom script, reading requirements information from pyproject.toml, see https://github.com/pypa/pip/issues/11440#issuecomment-1615211774.

Related:

https://github.com/pypa/packaging.python.org/issues/685

https://peps.python.org/pep-0631/

https://stackoverflow.com/a/75503961/2876079

https://pip.pypa.io/en/stable/cli/pip_install/#options

https://github.com/winpython/winpython/issues/1128

How to write a minimally working pyproject.toml file that can install packages?

Stefan
  • 10,010
  • 7
  • 61
  • 117
  • If you down-vote this answer, please let me know why and let me know of better solutions. – Stefan Jun 27 '23 at 09:24
  • This answer is a bit all over the place. The point **b)** and **c)** seem quite irrelevant to the question. Regarding the **b)** point, a project should always have a `[build-system]`, if it is missing from the project, then it is a misbehaving project. Regarding **c)**, this is an issue with your linter. The directory name is perfectly fine, this is how it should be, it does not have to be snake case, your linter should not care about this name anyway. – sinoroc Jun 27 '23 at 17:44
  • Thank you for your opinion. I do want to enforce snake_case naming convention in my python project. If you know how to disable/avoid the creation of the egg file in the src folder, please let me know. I guess I do neither need the build system nor the egg file in my use case. – Stefan Jun 27 '23 at 22:03
  • 1
    Thinking about it, `*.egg-info` might be a thing specific to *setuptools*, you might be able to get rid of that directory by switching to a different build back-end. Not sure. -- Anyway in principle it is a build artifact that you can and should ignore. I doubt you can do anything about the way it is named. – sinoroc Jun 27 '23 at 22:14
  • That is good to know. For my use case it would be helpful, if the default build-backend would not be setuptools but a none build backend, where no artifacts are built but only the wanted dependencies are installed. (Both, specifying my requirements as dependencies for the build-system as well as for the project would be fine for me... as long as they are installed. My preference is the second one because it allows for optional (dev) dependencies.) – Stefan Jun 28 '23 at 07:02
  • I switched the build-backend to hatchling and improved the answer accordingly. Hatchling does not create the annoying egg file. – Stefan Jun 28 '23 at 07:41
  • You might want to try something like Pipenv if you only need something to manage the dependencies installed in the virtual environment but have no need for packaging and redistributing your project. – sinoroc Jun 28 '23 at 17:41
  • Thank you. pipenv does not seem to support pyproject.toml (and I don't use virtual environments.) – Stefan Jun 28 '23 at 18:26
  • I see. `pip-tools` maybe then. – sinoroc Jun 28 '23 at 20:13
  • Hm. Why should I generate a requirements.txt from pyproject.toml and then install the requirements from requirements.txt ... if I am able to install them from pyproject.toml directly? I am afrait I still don't get the point. – Stefan Jun 28 '23 at 21:10
  • There are use cases for something like `requirements.txt`. I do not know what your use case is. If you don't need any of this, then don't use it. I am not sure what you want to achieve. And I am not sure it really does relate to the original question. -- I guess it is fine if you do a slight misuse of `pyproject.toml` by using it only as a place to list dependencies that are easily installable with `pip install .`. It is unusual so others might be surprised, but it should work fine I think. – sinoroc Jun 28 '23 at 21:20
  • As stated above, I use it as central configuration file, including settings for pylint, pytest, black, etc. ... instead of using the old individual files having different formats ( .pylintrc, pytest.ini, ....). Thank you for your thoughts. – Stefan Jun 28 '23 at 22:03
-2

I suggest switching to poetry, it's way better than a standard pip for dependency management. And because it uses pyproject.toml your dependencies and configs are in one place so it's easier to manage everything

Volodymyr Pivoshenko
  • 204
  • 5
  • hatch is maintained by pypa and I hope that the programs will get compatible regarding support of pyproject.toml. Also see https://github.com/pypa/pip/issues/12100 and https://hatch.pypa.io/latest/config/dependency/ – Stefan Jun 24 '23 at 06:32
  • poetry is not portable but stores an absolute path to python.exe – Stefan Jun 24 '23 at 23:01