Generalities

Repository :
https://pypi.org/

List installed modules :
python3 -c "help('modules')"
or :
pip3 list

List a depth tree of dependencies of a python environment:
pipdeptree.
It is a third-party dependency. To install it:
pip install pipdeptree

List available versions for a package in the repository:
pip install foo-package==

Module installation location :
the __file__ attribute of the imported module gives the information.
For example to know where the requests module is installed :
python3 -c "import requests; print (requests.__file__)"

Pip module

pip or python -m pip

In theory these produce the same result.
For example that  :
pip3 install requests python-dateutil ...
is often just an alternative to  :
python3 -v -m pip install requests python-dateutil ...

But in some cases, the pip command may fail or emit warning and so the python 3 -m pip way have to favored.
The problem is often related to updates of pip or the coexistence of user(s) and system pip installations on the machine.
Note : only inside a virtual env, pip3 is always a safe usage.

install command

Install a package (or multiple) in its last version From Repository :
pip install requests python-dateutil ...

Install a package (or multiple) in a specific version or a range of versions :
pip3 install 'pycrypto==2.7.1'
or with a range:
pip3 install 'pycrypto>=2.7.1,<2.9'

Force the reinstall of a package (or multiple and with a specified version or not) :
pip3 install 'pycrypto==2.7.1' --force-reinstall

Install specific packages/versions from a requirements file :
pip install -r requirements.txt

Update a package to its last version :
pip3 install --upgrade foo-package
We could even update pip itself :
pip3 install --upgrade pip

system vs user install :
Python installation are generally performed as root (system) but the python packages may be installed at system level or at user level.
--system flag : Install using the system scheme (default in all distros but debian)
For example with python 3.6 it would be in /usr/local/lib/python3.6.
--user flag : Install to the Python user install directory for your platform. Typically ~/.local/lib/pythonX.X, or %APPDATA%\Python on Windows. (default in debian)

Install the requirements with pyproject.toml and pip-compile

Suppose you have this requirements.in file:

flask == 2.3.1
flask_cors==4.0.0
requests == 2.31.0
gunicorn==21.2.0
models-lib

And the pyproject.toml file:

[build-system]
requires = ["setuptools", "setuptools-scm"]
build-backend = "setuptools.build_meta"
 
[project]
name = "service-lib"
authors = [
    {name = "David"},
]
version="0.1"
description = "My package description"
#readme = "README.rst"
requires-python = ">=3.10"
#keywords = ["one", "two"]
classifiers = [
    "Framework :: Flask",
    "Programming Language :: Python :: 3.10",
]
 
dynamic = ["dependencies"]
 
 
[tool.setuptools]
py-modules = []
 
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
#optional-dependencies = {dev = { file = ["requirements-dev.txt"] }}
 
 
#[project.scripts]
#my-script = "my_package.module:function"
 
# ... other project metadata fields as specified in:
#     https://packaging.python.org/en/latest/specifications/declaring-project-metadata/

Create the env:
python3.10 -m venv venv

Active it:
. venv/bin/activate

Install pip-tools:
python -m pip install pip-tools

Compile requirements.in (to generate requirements.txt):
pip-compile

Install the requirements.txt dependencies:
pip install .

Build the distribution files:
python -m build

Upload the package service-lib in the repository:
service-lib$ twine upload --config-file .pypirc --repository david-repository dist/*

install command in multiple packages development

Suppose our project consists of two packages service-lib and models-lib, located at the root of the project.
service-lib depends on models-lib.
During development, we don’t want to install models-lib at each change. It is cumbersome.
To address the problem, we can add a reference of models-lib package in the virtual environment of service-lib.
The idea is to create a file with the extension pth in the directory service-lib/venv/Lib/site-packages and to add in this file the location of the source code.
For example we can specify it in a file named models_lib_local.pth:
C:\programmation\workspace-python\python_blog_examples\devpi\models-lib\src

Performing this operation manually can be error-prone and tedious. To avoid that, we can use the editable flag (-e) of pip install, such as that from the service-lib directory :
pip install -e ../models-lib/
Using this command creates a link from the virtual environment to the package directory source.
The link is implemented in the way that pip prefers.
It may be exactly what we just did by creating the file models_lib_local.pth.
For example on windows with git bash this fie is created: __editable__.models_lib-0.11.pth

Use a proxy and disable ssl check of repos with pip3

python3 -v -m pip install moduleToInstall otherModuleToInstall --proxy http://user:pass@myproxy:8080 --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org

Requests lib

– from pip provided by python :
python3 -m pip install requests

– from linux packet managers :
yum install python36-requests

python-dateutil lib

– from pip :
pip3 install python-dateutil

Redis-py lib

Install redis-py by using proxy and without ssl check for download repos :
pip3 install -v --proxy http://user:pass@myproxy:8080 --trusted-host pypi.org --trusted-host pypi.python.org --trusted-host files.pythonhosted.org redis

Ansible

See the Ansible page.

pip wheels commands

Build Wheel archives for project dependencies and store them into a directory named ‘wheels’:
We can execute the command at the level of the by pyproject.toml or setup.py:
pip wheel . -w wheels

Alternatively we can execute the command with a requirements.txt file:
pip wheel -r requirements.txt -w wheels

Install packages located in the wheels directory :
pip install --no-cache-dir wheels/*.whl

Common issues

Problem :
Pip command fails, emits a warning or still doesn’t use the expected version.

Symptoms :
* pip3 --version outputs an « unexpected » version. For example it outputs the system rather than the user version

* After installing or updating pip as user mode (from the 9 to the 21.2.4), the pip execution produces that warning :

WARNING: pip is being invoked by an old script wrapper. This will fail in a future version of pip. Please see https://github.com/pypa/pip/issues/5599 for advice on fixing the underlying issue. To avoid this problem you can invoke Python with ‘-m pip’ instead of running pip directly.

Solution :
1) Cumbersome way, use the python3 command :
python3 -m pip ...

2) If you rely on an user install of pip, check the location of the pip command used.
For a system install of pip you expect /usr/bin/pip3 while for a user install of pip you expect ~/.local/bin/pip3
You can do that check with :
which pip3

If the location is not the expected one but ~/.local/bin, you can fix it.
First, ensure that the user install of pip exits at ~/.local/bin.
If it exists, it means that the path is not in the PATH of the user or so that it occurs after another folder that contains the system pip command.
Usually, the ~/.local/bin is added in the PATH in the .profile file.
But with bash and the presence of ~/.bash_profile that file is just ignored.
So we could add the path manually at the end of ~/.bash_profile:
export PATH="$HOME/.local/bin:$PATH"

Problem :
Pip doesn’t work any longer after a python update.

Symptoms :
Pip may work as system but fails as user or fails everywhere.
The Pip command may fail because of a module not found such as :

Traceback (most recent call last):
  File "/home/ansible/.local/bin/pip3", line 7, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip._internal'

Solution :
Upgrade the pip version by passing by the python3 wrapper :
Does that where pip fails : at system level or at user level :
python3 -m pip install --system --upgrade pip
or
python3 -m pip install --user --upgrade pip

Problem :
pip3 –version or pip3 list works but pip install doesn’t work any longer after a python update to 3.9.

Symptoms :
Pip install fail because of a missing attribute in a setuptools module:

File "/usr/lib/python3/dist-packages/setuptools/py33compat.py", line 54, in <module>
   unescape = getattr(html, 'unescape', html_parser.HTMLParser().unescape)
AttributeError: 'HTMLParser' object has no attribute 'unescape'

Solution :
Upgrade the setuptools module (for user or system, according to the issue) :
pip3 install --upgrade --user setuptools

Problem :
Creation of venv with python 3.9 update/install fails because ensurepip is not in the pip modules list : python3.9 -m venv foo

Symptoms :
Pip creation venv fails such as :

Error: Command '['/home/david/workspace-gitlab/spring-boot-prometheus-example/script-prom-requests-env/bin/python3.9', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1.

Solution :
Install the venv package with the linux package manager :
apt install python3.9-venv