Thoughts and notes on how to set up pyproject.toml files
Published in
[
Better Programming
6 min readJul 26, 2022
Background image by Klára Vernarcová on Unsplash
Around three years ago, I wrote a guide on designing clean Python package structures that rely on setup.py and setup.cfg files, and the Setuptools build system. The Python ecosystem continuously evolves, and, nowadays, developers have more flexibility to choose build systems that better fit their needs — e.g., Bento, Flit, or Poetry.
PEP 517 proposes a standard way to define alternative build systems for Python projects. And thanks to PEP 518, developers can use pyproject.toml configuration files to set up the projects’ build requirements.
PEP 517 also introduces the concepts of build front and backend. According to the specification, a build frontend is a tool that takes arbitrary code and builds wheels from it. The actual building is done by the build backend. In a command like pip wheel some-directory/, Pip is acting as a build frontend. In this sense, a fundamental change promoted by PEP 517 is that Pip is not “hardcoded” to use Setuptools as the build backend anymore. Then, as mentioned in the first paragraph, new build tools are arising, and there is an official programming interface that ensures compatibility among them.
The present piece describes how to use pyproject.toml and Flit to get similar results as the former package setup guide. I am using Flit for two different reasons:
- Demonstrate how to use another build system than Setuptools. Flit claims easier usage.
- It complies with PEP 517 and PEP 660, supporting editable installs — which is very useful for developers — without further configuration. Setuptools requires a small workaround at the time of this writing in July 2022.
The suggested package structure is presented below:
project-root
├──src/
└──package_name/
└──__init__.py
├──tests/
└──package_name/
├──.gitignore
├──LICENSE
├──pyproject.toml
└──README.md
You will notice that pyproject.toml replaces setup.py and setup.cgf. The following code snippet brings an initial yet working version of the file:
pyproject.toml cheat sheet — initial version
Despite the TOML format, the content is very similar to setup.py, except for the [build-system] section (or table, as such key/value pairs are called in the TOML specification). It is used to specify the build system for the project — in this case, Flit as the backend.
The pyproject.toml file is handled by the build system, and its content may vary depending on the backend you choose. For example, Poetry requires the authors array elements as name <email> strings (source), while Flit requires a list of tables with name and email keys (source). Please always refer to the build backend documentation when writing your pyproject.toml files to avoid syntax issues. You can easily find such docs by googling <build backend> pyproject.toml (e.g., flit pyproject.toml or poetry pyproject.toml).
Additional Features
In the coming sections, you will see how to add nice-to-have features to the project, such as development dependency management, automated tests, and linting. A fully working example is available on the companion repository:
[
GitHub - ricardolsmendes/pyproject-toml-cheat-sheet: Thoughts & notes on clean pyproject.toml…
Thoughts & notes on clean pyproject.toml design. pyproject.toml: the package’s descriptor file, in which multiple…
github.com
To use Flit also as the build frontend (flit commands), as I will show next, you need to install Flit from PyPI. Run pip install flit~=3.7.1 to do so, replacing 3.7.1 with your preferred version just in case. This step is not required if Pip is used as the frontend.
Development dependencies
Flit’s pyproject.toml documentation describes how to add optional dependencies to a given project. Optional, in this context, means dependencies not required by the package to do its core job but for testing or documentation purposes, for instance.
The
[project.optional-dependencies]table contains lists of packages needed for every optional feature. (source)
The code below is used in the companion project to enumerate the development dependencies:
[project.optional-dependencies]
dev = [
"pylint ~=2.14.0",
"toml ~=0.10.2",
"yapf ~=0.32.0",
]
test = [
"pytest-cov ~=3.0.0",
]
To install them along with your package, just run flit install --deps=develop.
Automated tests
Support for pyproject.toml was introduced in Pytest 6.0. Since then, one can use the [tool.pytest.ini_options] table to set up Pytest for a given project. In the companion repository, I used the below lines to set the default options:
[tool.pytest.ini_options]
addopts = "--cov --cov-report html --cov-report term-missing --cov-fail-under 95"
And the following table to determine which directory pytest-cov should take into consideration when calculating test coverage:
[tool.coverage.run]
source = ["src"]
Then, just run pytest from the root folder.
Code formatting
YAPF is an example of a code formatting tool that already supports [pyproject.toml](https://github.com/google/yapf/issues/708), providing an alternative to setup.cfg. The following code snippet shows how to use it:
[tool.yapf]
blank_line_before_nested_class_or_def = true
column_limit = 88
Run yapf --in-place --recursive ./src ./tests from the project’s root folder, and voilà!
Please refer to your preferred formatter’s documentation to make sure they also support pyproject.toml.
Linting
Flake8 is usually my first choice for linting, given its simplicity. Unfortunately, it still did not support pyproject.toml for configuration when I wrote this blog post, so I decided for Pylint:
[tool.pylint]
max-line-length = 88
disable = [
"C0103", # (invalid-name)
"C0114", # (missing-module-docstring)
"C0115", # (missing-class-docstring)
"C0116", # (missing-function-docstring)
"R0903", # (too-few-public-methods)
"R0913", # (too-many-arguments)
"W0105", # (pointless-string-statement)
]
Run pylint ./src ./tests from the root folder to see the results.
And again, refer to your preferred linter’s documentation to ensure they support pyproject.toml.
Editable installs
Developers using Pip rely on pip install -e . (aka editable install or development mode) to mimic installing their packages from source code and test changes on the fly. It will work if you have Flit as the build backend because Flit is compliant with PEP 660.
Flit’s CLI provides a similar feature through flit install -s. You can find more about flit install in the official docs.
Sidenote: I have noticed that when using Flit to install the package,pytest-cov requires the project’s source code to be installed with the -s flag to report coverage accordingly, which means the flag should be used even when running tests on CI pipelines.
Build and Publish
Building with Flit is straightforward, and you can find the available options in the official docs. Let’s do a local build and install the resulting package to see what it looks like.
- Remove working packages from previous installs:
pip uninstall -y pyproject-toml-cheat-sheet stringcase(I did not find a way to do this using the Flit CLI). - Build a wheel and a sdist (tarball) from the source:
flit build. When the build finishes, you should see a message similar toBuit wheel: dist/pyproject_toml_cheat_sheet-1.0.0-py3-none-any.whlin the console. - Install using Pip to ensure the package is Pip-compatible:
pip install dist/pyproject_toml_cheat_sheet-1.0.0-py3-none-any.whl.
Now, let’s call the pyproject_cheat_sheet.StringFormatter.format_to_snakecase method using the Python Interactive Shell:
python
>>> from pyproject_cheat_sheet import StringFormatter
>>> print(StringFormatter.format_to_snakecase('FooBar'))
foo_bar
>>> exit()
As you can see, foo_bar is the output for StringFormatter.format_to_snakecase('FooBar'), which means the package works as expected.
Publishing is also very simple. Please refer to the Flit documentation for further details.
Still Want to Use Setuptools?
We are good with Flit, but what if you still want to use Setuptools for any reason, such as avoiding breaking changes to existing build pipelines? No worries… it is just a matter of configuring the build system through pyproject.toml, starting with something like:
[build-system]
build-backend = "setuptools.build_meta"
requires = [
"setuptools ~=63.2.0",
"wheel ~=0.37.1",
]
Then you probably want to check Setuptools-specific configurations, and there is plenty of information in their docs.
Closing Thoughts
The more features and dependencies we add to the code base, the more complex it gets, but we can do it reasonably. Some key benefits of keeping code as clean as possible are onboarding new team members and promoting long-term maintenance with less burden for involved folks. I truly believe it starts with how we set up our packages, which is why I share such thoughts.
I hope you enjoyed it!
References
- carlosperate/awesome-pyproject: An Awesome List of projects using the
pyproject.tomlPython configuration file.