pip-tools

December 08, 2020 • 8 min read

For years managing pip packages has been a bit of a pain when it comes to keeping everything up to date.

A prime example that springs to mind is compatibility with html5lib where, thanks to their unique versioning, I’ve seen things like html5lib!=0.9999,!=0.99999,<0.99999999,>=0.90

Last year (2019) I started to play around with pip-env because I was starting a new project & decided it was time to try something beyond vanilla pip. I looked at the likes of poetry as well, but pip-env sounded like it’d help. It worked, and that particular project is still in production with pip-env but I went off that way of managing the “pip workflow”. The way it manages your venv wasn’t something I ever got on with.

So the next time I was beginning a project I looked at what else was available to manage packages. This time I came across a superb github account, jazzband. A quick look through their repos & I was pleased to find lots of helpful django related bits, and I’d used django-axes before. They also maintain pip-tools and since discovering this I’ve started every project with it.

Essentially, it creates requirements.txt for you.

$ source /path/to/venv/bin/activate
(venv)$ python -m pip install pip-tools

Then you create a requirements.in file which contains the packages you want to get started with.

django
sentry-sdk
django-feature-policy
django-referrer-policy

# dev-packages
django-debug-toolbar
django-extensions
pytest
python-dotenv

Then simply get pip-tools to do the hard work by running pip-compile and it creates a requirements.txt from the above project requirements.

#
# This file is autogenerated by pip-compile
# To update, run:
#
#    pip-compile
#
attrs==20.3.0             # via pytest
certifi==2020.12.5        # via sentry-sdk
django-debug-toolbar==3.2  # via -r requirements.in
django-extensions==2.2.5  # via -r requirements.in
django-feature-policy==3.5.0  # via -r requirements.in
django-referrer-policy==1.0  # via -r requirements.in
django==2.2.17            # via -r requirements.in, django-debug-toolbar, django-feature-policy, django-referrer-policy
importlib-metadata==3.1.1  # via pluggy, pytest
iniconfig==1.1.1          # via pytest
packaging==20.7           # via pytest
pluggy==0.13.1            # via pytest
py==1.9.0                 # via pytest
pyparsing==2.4.7          # via packaging
pytest==6.1.2             # via -r requirements.in
python-dotenv==0.15.0     # via -r requirements.in
pytz==2020.4              # via django
sentry-sdk==0.19.4        # via -r requirements.in
six==1.15.0               # via django-extensions
sqlparse==0.4.1           # via django, django-debug-toolbar
toml==0.10.2              # via pytest
urllib3==1.26.2           # via sentry-sdk
zipp==3.4.0               # via importlib-metadata

This is good, but remember the issue I had was with when packages release updates, and you then need to manage compatibility between everything? Well you simply run pip-compile -U which is the upgrade flag. This will go through all the packages and find the upgrades available while also maintaining compatibility with oneanother.

If you took a trip back in time and your requirements were something like this;

django<2
sentry-sdk
django-feature-policy
django-referrer-policy

# dev-packages
django-debug-toolbar
django-extensions
pytest
python-dotenv

Then compiling this, you’d get the following error;

(venv)$ pip-compile
Could not find a version that matches django<2,>=1.11,>=2.2 (from -r requirements.in (line 1))
Tried: 1.1.3, 1.1.4, 1.2, 1.2.1, 1.2.2, 1.2.3, 1.2.4, 1.2.5, 1.2.6, 1.2.7, 1.3, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.3.5, 1.3.6, 1.3.7, 1.4, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.5, 1.5.1, 1.5.2, 1.5.2, 1.5.3, 1.5.4, 1.5.5, 1.5.6, 1.5.7, 1.5.8, 1.5.8, 1.5.9, 1.5.10, 1.5.11, 1.5.12, 1.5.12, 1.6, 1.6, 1.6.1, 1.6.1, 1.6.2, 1.6.2, 1.6.3, 1.6.3, 1.6.4, 1.6.4, 1.6.5, 1.6.5, 1.6.6, 1.6.6, 1.6.7, 1.6.7, 1.6.8, 1.6.8, 1.6.9, 1.6.9, 1.6.10, 1.6.10, 1.6.11, 1.6.11, 1.7, 1.7, 1.7.1, 1.7.1, 1.7.2, 1.7.2, 1.7.3, 1.7.3, 1.7.4, 1.7.4, 1.7.5, 1.7.5, 1.7.6, 1.7.6, 1.7.7, 1.7.7, 1.7.8, 1.7.8, 1.7.9, 1.7.9, 1.7.10, 1.7.10, 1.7.11, 1.7.11, 1.8, 1.8, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.8.3, 1.8.3, 1.8.4, 1.8.4, 1.8.5, 1.8.5, 1.8.6, 1.8.6, 1.8.7, 1.8.7, 1.8.8, 1.8.8, 1.8.9, 1.8.9, 1.8.10, 1.8.10, 1.8.11, 1.8.11, 1.8.12, 1.8.12, 1.8.13, 1.8.13, 1.8.14, 1.8.14, 1.8.15, 1.8.15, 1.8.16, 1.8.16, 1.8.17, 1.8.17, 1.8.18, 1.8.18, 1.8.19, 1.8.19, 1.9, 1.9, 1.9.1, 1.9.1, 1.9.2, 1.9.2, 1.9.3, 1.9.3, 1.9.4, 1.9.4, 1.9.5, 1.9.5, 1.9.6, 1.9.6, 1.9.7, 1.9.7, 1.9.8, 1.9.8, 1.9.9, 1.9.9, 1.9.10, 1.9.10, 1.9.11, 1.9.11, 1.9.12, 1.9.12, 1.9.13, 1.9.13, 1.10, 1.10, 1.10.1, 1.10.1, 1.10.2, 1.10.2, 1.10.3, 1.10.3, 1.10.4, 1.10.4, 1.10.5, 1.10.5, 1.10.6, 1.10.6, 1.10.7, 1.10.7, 1.10.8, 1.10.8, 1.11, 1.11, 1.11.1, 1.11.1, 1.11.2, 1.11.2, 1.11.3, 1.11.3, 1.11.4, 1.11.4, 1.11.5, 1.11.5, 1.11.6, 1.11.6, 1.11.7, 1.11.7, 1.11.8, 1.11.8, 1.11.9, 1.11.9, 1.11.10, 1.11.10, 1.11.11, 1.11.11, 1.11.12, 1.11.12, 1.11.13, 1.11.13, 1.11.14, 1.11.14, 1.11.15, 1.11.15, 1.11.16, 1.11.16, 1.11.17, 1.11.17, 1.11.18, 1.11.18, 1.11.20, 1.11.20, 1.11.21, 1.11.21, 1.11.22, 1.11.22, 1.11.23, 1.11.23, 1.11.24, 1.11.24, 1.11.25, 1.11.25, 1.11.26, 1.11.26, 1.11.27, 1.11.27, 1.11.28, 1.11.28, 1.11.29, 1.11.29, 2.0, 2.0, 2.0.1, 2.0.1, 2.0.2, 2.0.2, 2.0.3, 2.0.3, 2.0.4, 2.0.4, 2.0.5, 2.0.5, 2.0.6, 2.0.6, 2.0.7, 2.0.7, 2.0.8, 2.0.8, 2.0.9, 2.0.9, 2.0.10, 2.0.10, 2.0.12, 2.0.12, 2.0.13, 2.0.13, 2.1, 2.1, 2.1.1, 2.1.1, 2.1.2, 2.1.2, 2.1.3, 2.1.3, 2.1.4, 2.1.4, 2.1.5, 2.1.5, 2.1.7, 2.1.7, 2.1.8, 2.1.8, 2.1.9, 2.1.9, 2.1.10, 2.1.10, 2.1.11, 2.1.11, 2.1.12, 2.1.12, 2.1.13, 2.1.13, 2.1.14, 2.1.14, 2.1.15, 2.1.15, 2.2, 2.2, 2.2.1, 2.2.1, 2.2.2, 2.2.2, 2.2.3, 2.2.3, 2.2.4, 2.2.4, 2.2.5, 2.2.5, 2.2.6, 2.2.6, 2.2.7, 2.2.7, 2.2.8, 2.2.8, 2.2.9, 2.2.9, 2.2.10, 2.2.10, 2.2.11, 2.2.11, 2.2.12, 2.2.12, 2.2.13, 2.2.13, 2.2.14, 2.2.14, 2.2.15, 2.2.15, 2.2.16, 2.2.16, 2.2.17, 2.2.17, 3.0, 3.0, 3.0.1, 3.0.1, 3.0.2, 3.0.2, 3.0.3, 3.0.3, 3.0.4, 3.0.4, 3.0.5, 3.0.5, 3.0.6, 3.0.6, 3.0.7, 3.0.7, 3.0.8, 3.0.8, 3.0.9, 3.0.9, 3.0.10, 3.0.10, 3.0.11, 3.0.11, 3.1, 3.1, 3.1.1, 3.1.1, 3.1.2, 3.1.2, 3.1.3, 3.1.3, 3.1.4, 3.1.4
Skipped pre-versions: 1.8a1, 1.8b1, 1.8b2, 1.8rc1, 1.9a1, 1.9b1, 1.9rc1, 1.9rc2, 1.10a1, 1.10a1, 1.10b1, 1.10b1, 1.10rc1, 1.10rc1, 1.11a1, 1.11b1, 1.11rc1, 1.11rc1, 2.0a1, 2.0b1, 2.0rc1, 2.1a1, 2.1b1, 2.1rc1, 2.2a1, 2.2a1, 2.2b1, 2.2b1, 2.2rc1, 2.2rc1, 3.0a1, 3.0a1, 3.0b1, 3.0b1, 3.0rc1, 3.0rc1, 3.1a1, 3.1a1, 3.1b1, 3.1b1, 3.1rc1, 3.1rc1
There are incompatible versions in the resolved dependencies:
  django<2 (from -r requirements.in (line 1))
  Django>=1.11 (from django-referrer-policy==1.0->-r requirements.in (line 4))
  Django>=2.2 (from django-debug-toolbar==3.2->-r requirements.in (line 7))
  Django>=2.2 (from django-feature-policy==3.5.0->-r requirements.in (line 3))

It is rather verbose, but tells you what it’s tried to do, and where the incompatibility comes from with the requirements from each of the packages in use.

Once you’ve generated your requirements.txt you simply run pip-sync and pip-tools will ensure your .venv matches your requirements. It even gets rid of old packages, like most people wouldn’t when using vanilla pip!

So go on, install pip-tools and make your development easier.

pip-tools

Liked it? Share it