How to setup two PyPI indices

Goal

  1. pip install should install/update packages from GitLab as well as PyPi repo. If same package is present in both, PyPi is preferred.
  2. pip install should support authentication. Preferred, if somehow we can make it read from a config file so that we don't need to specify it repeatatively.

Theory

  1. pip install supports --extra-index-url to specify additional PyPi indexes. The same can also be provided via pip.conf file.
  2. pip uses requests which supports ~/.netrc as config file (docs).

Steps

  1. Create a pip.conf (pip.ini if on Windows) in any of the locations suggested by pip config -v list.
  2. Add your GitLab PyPi index URL to pip.conf.
[install]
extra-index-url = https://gitlab.com/api/v4/projects/12345678/packages/pypi/simple
  1. Create or update your ~/.netrc file and add your auth details for GitLab.
machine gitlab.com
    login <token-name>
    password <token-pass>
  1. We can now install packages as simply as pip install <package-name>. pip will now look at both indexes to find your packages, with preference provided to the one pointed by index-url.

Additional info

  1. The same could have been possible for pip search too, had there been support for multiple indexes. Till then, one needs to manually specify which PyPi index URL should be used. GitLab does not seem to support pip search since it throws 415 Client Error: Unsupported Media Type when specified as the PyPi index.
  2. As for your doubts, each section in pip.conf points to that particular command, [install] provides configuration for pip install, [search] for pip search and so on. [global] probably refers to parameters that can be specified for all the commands be it pip install or pip search.
  3. .pypirc file is made specially for configuring package indexes related to upload (used by twine/flint), where as pip.conf is associated with configuring pip which manages python packages on your local system.

Try this (based on information from https://github.com/pypa/pip/issues/6797 and Can pip.conf specify two index-url at the same time?):

[global]
index-url = http://pypi.org/simple
trusted-host = pypi.org
               gitlab.company.de
extra-index-url= https://username:[email protected]/api/v4/projects/2142423/packages/pypi

Tags:

Python

Pip

Pypi

Gitlab

Gitlab Api

Recent Posts