Apple - How to use pip after the OS X El Capitan upgrade?

A quick solution is to use homebrew to install python into /usr/local/bin so that your pip can run against a user-modifiable python framework.

brew install python
pip --version

Disabling System Integrity Protection is also an option, but I don't recommend that for anything but professionally managed and fire walled servers where you have the manpower to manage intrusion detection or if you are a developer/sysadmin and need to test things with and without SIP.

ls -lO /System/Library/Frameworks/Python.framework/Versions/2.7/
csrutil status

You will see that the restricted flag is set which cannot be removed even as root while SIP is engaged.

Using homebrew makes it possible to manage pip and python separately than the system provided version. As a bonus, the homebrew framework is designed to ease maintainance and patch/chores via automation.

Another viable option without a need to disable SIP or install other Python versions, is to install the modules only for the current user using

pip install --user <modulename>

If it is just your personal machine, this would be the simplest and safest solution.

This problem often arises when pip tries to install a manpage for IPython on El Capitan. The quick fix is to use a pip command like this:

sudo -H pip install --install-option '--install-data=/usr/local' <package>

However, System Integrity Protection (SIP) on El Capitan blocks several bad practices with pip that used to slide by, so you will probably need to make some more changes to get pip running smoothly on El Capitan.

SIP on El Capitan exposes three problems with using pip with the Apple-supplied version of Python on OS X:

1. distutils does not set two important variables correctly on Macs, so pip tries to write headers and other shared files (e.g., manpages) under /System/Library/Frameworks/Python.framework/Versions/2.7/. This is a bad idea, but in earlier versions of OS X, it succeeded if pip was run with sudo. However it fails on El Capitan due to SIP. This is the error you ran into. It gives messages like OSError: [Errno: 1] Operation not permitted: '/System/Library/Frameworks/Python.framework/Versions/2.7/share'

2. Apple installs outdated versions of some packages in /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/ (e.g., six). On previous versions of OS X, when you installed a package that needed a newer version of one of these, sudo pip would silently remove the old version from the /System/ folder and install a newer version in /Library/Python/2.7/site-packages. This was also a bad idea, and is no longer possible with SIP. But now pip will crash with an error message while trying to remove the old package. That message is also OSError: [Errno: 1], but it comes after a message like Uninstalling six-1.4.1:. See, e.g., https://github.com/pypa/pip/issues/3165 .

3. The Apple version of Python adds several directories under /System/Library/Frameworks/Python.framework/Versions/2.7/ to the python search path above the standard user-accessible package installation locations. So if you install a newer version of a package elsewhere (e.g., sudo -H pip install --ignore-installed six), you will get a message that the installation succeeded, but then when you run python you will get the older version from /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/. This also makes it impossible to use new packages that have the same name as modules from the standard library.

You can work around these issues, but the method depends on your answers to three questions.

1. Do you want to continue using the Mac OS X version of Python or install your own? Installing your own is the safest option, and can be done via the official Python installer, Homebrew or Anaconda. This is also what Apple recommends, as pointed out by @Sacrilicious. If you install your own version of Python, you should probably uninstall anything currently installed in /Library/Python/2.7/site-packages and any scripts that were installed in /usr/local/bin for those packages (including pip). Otherwise you will have the annoying experience of some scripts accessing the system-installed version of Python and some accessing your own installation.

If you want to stick with the system-installed Python, then you have to make two more decisions:

2. Do you want to install packages for all users, or just for yourself? Installing for all users ensures that every program that uses Python (including possibly administrative scripts) will have access to all the packages you install. However, there's a distant chance that it will interfere with El Capitan's own use of Python. (I would hope that Apple uses python -S to ensure they always get the packages they expect, but I have no way to test this.) Installing just for your own user account eliminates the possibility of interfering with the system Python installation. Note: if you are going to switch from system-wide installation to user-only, you should probably take this chance to uninstall anything currently installed in /Library/Python/2.7/site-packages and related scripts in /usr/local/bin.

3. Do you want to hide the extra packages that are installed with the OS X version of Python (under /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/), or keep them in the search path? I recommend hiding them, so that the newest versions of these packages will automatically be installed in user-accessible locations when needed. If you don't hide this directory, then you will occasionally get messages that pip was unable to remove an existing package in order to upgrade it to a later version (needed by a different package you are installing). In that case, you will need to run pip install --ignore-installed <package>, which will install the newer version and hide the system-installed version. However, if you hide the whole /System/.../Extras/... directory, you will lose access to some Apple packages that are not available via pip, i.e., CoreGraphics and bonjour. (If you need these, you may be able to get access by symlinking them into your site-packages directory.)

Now, here are the workarounds. These would be good practice on all versions of OS X, to avoid accidentally replacing or removing Python packages used by the operating system; however they are essential if you want to use user-installed packages with the Apple-supplied version of Python on OS X El Capitan (10.11).

Install pip

You probably did this already, but if not, you can use the following command to install pip for all users:

sudo -H easy_install pip
# pip script will be installed in /usr/local/bin

Or use this command to install pip for your own user account only:

easy_install --user pip
# pip script will be installed in ~/Library/Python/2.7/bin

Manage Shared File Locations

If you are installing packages for all users, create a file called .pydistutils.cfg with these lines (from https://github.com/pypa/pip/issues/426):

[install]
install-data=/usr/local
install-headers=/usr/local

If you usually use sudo -H pip ..., then you should put this file in /var/root (home directory for the root user). If you usually use sudo pip ..., then you should put this file in your own home directory (~).

These settings will prevent pip from trying to write shared items like headers and manpages under /Library/System. (The command at the top of this answer is a quicker version of the same thing.) These settings are needed because darwin-specific code in /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/distutils/command/install.py fails to set these variables to root-writeable locations (although it sets other variables correctly). There is more information on this at https://github.com/pypa/pip/issues/3177 .

If you install packages for your own user account only, shared items will automatically be installed under ~/Library/Python/2.7/. But you should add the following lines to your ~/.profile so the shared items will be found when you need them:

export PATH=~/Library/Python/2.7/bin:$PATH
export MANPATH=~/Library/Python/2.7/share/man:$MANPATH

Note: you will need to start a new shell or run these on the command line for the changes to take effect. You may also want to run hash -r if you've recently removed old scripts from the path.

Manage Python Path

You will need to ensure that the packages you install are higher in Python's search order than system-installed packages. The easiest way to do this is with .pth files. This follows @Sacrilicious's suggestion elsewhere on this page, but ensures that your user site-packages directory is searched before your system-wide site-packages directory, and both are searched before the standard library and Apple's Extras directory (both under /System/...). It also omits /System/.../Extras from the search path if desired.

Create a file called fix_mac_path.pth, with the text below. If you are installing packages for all users, fix_mac_path.pth should be placed in /Library/Python/2.7/site-packages. If you are installing only for your own user, fix_mac_path.pth should be in ~/Library/Python/2.7/lib/python/site-packages. (This file can have any name you want, but it has to be placed in one or both of these locations, and it has to end with .pth; also, all the text in this file has to be on one line.)

If you want to hide the Apple-installed packages in /System/.../Extras:

First run one of the following commands to get a working copy of pip/setuptools independent from the Apple-supplied version:

pip install --ignore-installed --user setuptools   # your account only
# or
sudo -H pip install --ignore-installed setuptools  # all users

Then put the following code in fix_mac_path.pth at the location specified above:

import sys; std_paths=[p for p in sys.path if p.startswith('/System/') and not '/Extras/' in p]; sys.path=[p for p in sys.path if not p.startswith('/System/')]+std_paths

If you want to keep using the Apple-installed packages, you don't need to install another copy of setuptools. Just put the following code in fix_mac_path.pth at the location specified above:

import sys; std_paths=[p for p in sys.path if p.startswith('/System/')]; sys.path=[p for p in sys.path if not p.startswith('/System/')]+std_paths

After this, you can use python -m site to make sure the path search order makes sense.

Install Packages

After this, you should be able to install new packages using one of the following commands.

For all users:

sudo -H pip install <package>

For your own user:

pip install --user <package>