Failing to get PyCharm to work with remote interpreter on docker

If you're still seeing this in PyCharm 2020.2 then do this:

close PyCharm
try @peterc's suggestion:

docker ps -a | grep -i pycharm | awk '{print $1}' | xargs docker rm

launch PyCharm again

Was having a near identical error and was able to get past it. I did two things though I'm uncertain as to which was the actual solution:

Made sure the mappings were correct under both (a) Preferences -> Project -> Project Interpreter -> Path mappings and (b) Run -> Edit Configurations -> <Your_Configuration> -> Path mappings
Removed/deleted any containers that looked to be related to PyCharm (believe this is more than likely what solved things).

Hope this helps. PyCharm docker-compose seems to work for some and be a real PITA for others.

One other note. I downgraded from PyCharm 2018 to 2017.3 as there's known docker bugs in 2018.

EDIT: And it would seem a docker-compose down from CLI reintroduces the error -_-

TLDR:

The {project_name}_pycharm_helpers_{pycharm_build_number} volume has been removed or is corrupted.

To repopulate it, run:

docker volume rm {project_name}_pycharm_helpers_{pycharm_build_number}
docker run -v {project_name}_pycharm_helpers_{pycharm_build_number}:/opt/.pycharm_helpers pycharm_helpers:{pycharm_build_number}

The pycharm_build_number can be found in the about section of your pycharm (mac OS: Pycharm > About)

Long story

I struggled a lot with PyCharm suddenly not finding the helpers any more or any related bugs, sometimes because I was clearing my containers or volumes. For instance, running

docker rm -f `docker container ps -aq`
docker volume rm $(docker volume ls -q)

will almost surely get pycharm into troubles.

AFAIK about how PyCharm works, there is:

a PyCharm base image named pycharm_helpers with tag corresponding to your pycharm build number (for example: PY-202.7660.27)
the first time you create docker related things, PyCharm creates volumes that get data from this image for later use in your containers. For instance, after a first attempt at running a remote docker-compose interpreter, I see the newly created myproject_pycharm_helpers_PY-202.7660.27 volume when doing docker volume ls.
when running the docker interpreter, PyCharm adds this volume into the /opt/.pycharm_helpers directory by adding at some point a -v myproject_pycharm_helpers_PY-202.7660.27:/opt/.pycharm_helpers to your command. For instance using docker-compose, you can see the addition of the -f /Users/clementwalter/Library/Caches/JetBrains/PyCharm2020.2/tmp/docker-compose.override.1508.yml and when you actually look into this file you see:

version: "3.8"
services:
  local:
    command:
    - "python"
    - "/opt/.pycharm_helpers/pydev/pydevconsole.py"
    - "--mode=server"
    - "--port=55824"
    entrypoint: ""
    environment:
      PYCHARM_MATPLOTLIB_INTERACTIVE: "true"
      PYTHONPATH: "/opt/project:/opt/.pycharm_helpers/pycharm_matplotlib_backend:/opt/.pycharm_helpers/pycharm_display:/opt/.pycharm_helpers/third_party/thriftpy:/opt/.pycharm_helpers/pydev"
      PYTHONUNBUFFERED: "1"
      PYTHONIOENCODING: "UTF-8"
      PYCHARM_MATPLOTLIB_INDEX: "0"
      PYCHARM_HOSTED: "1"
      PYCHARM_DISPLAY_PORT: "63342"
      IPYTHONENABLE: "True"
    volumes:
    - "/Users/clementwalter/Documents/myproject:/opt/project:rw"
    - "pycharm_helpers_PY-202.7660.27:/opt/.pycharm_helpers"
    working_dir: "/opt/project"
volumes:
  pycharm_helpers_PY-202.7660.27: {}

You get into troubles when this volume is not correctly populated anymore. Fortunately the docker volume documentation has a section "Populate a volume using a container" which is exactly what PyCharm does under the hood.

For the record you can check the content of the pycharm_helpers image:

$ docker run -it pycharm_helpers:PY-202.7660.27 sh
/opt/.pycharm_helpers #

you end up into the pycharm_helpers directory and find all the helpers here:

/opt/.pycharm_helpers # ls -la
total 5568
drwxr-xr-x   21 root     root          4096 Dec 17 16:38 .
drwxr-xr-x    1 root     root          4096 Dec 17 11:07 ..
-rw-r--r--    1 root     root           274 Dec 17 11:07 Dockerfile
drwxr-xr-x    5 root     root          4096 Dec 17 16:38 MathJax
-rw-r--r--    1 root     root          2526 Sep 16 11:14 check_all_test_suite.py
-rw-r--r--    1 root     root          3194 Sep 16 11:14 conda_packaging_tool.py
drwxr-xr-x    2 root     root          4096 Dec 17 16:38 coverage_runner
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 coveragepy
-rw-r--r--    1 root     root         11586 Sep 16 11:14 docstring_formatter.py
drwxr-xr-x    4 root     root          4096 Dec 17 16:38 epydoc
-rw-r--r--    1 root     root           519 Sep 16 11:14 extra_syspath.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 generator3
-rw-r--r--    1 root     root             8 Sep 16 11:14 icon-robots.txt
-rw-r--r--    1 root     root          3950 Sep 16 11:14 packaging_tool.py
-rw-r--r--    1 root     root       1490666 Sep 16 11:14 pip-20.1.1-py2.py3-none-any.whl
drwxr-xr-x    2 root     root          4096 Dec 17 16:38 pockets
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 profiler
-rw-r--r--    1 root     root           863 Sep 16 11:14 py2ipnb_converter.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 py2only
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 py3only
drwxr-xr-x    7 root     root          4096 Dec 17 16:38 pycharm
drwxr-xr-x    4 root     root          4096 Dec 17 16:38 pycharm_display
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 pycharm_matplotlib_backend
-rw-r--r--    1 root     root        103414 Sep 16 11:14 pycodestyle.py
drwxr-xr-x   24 root     root          4096 Dec 17 16:38 pydev
drwxr-xr-x    9 root     root          4096 Dec 17 16:38 python-skeletons
drwxr-xr-x    2 root     root          4096 Dec 17 16:38 rest_runners
-rw-r--r--    1 root     root        583493 Sep 16 11:14 setuptools-44.1.1-py2.py3-none-any.whl
-rw-r--r--    1 root     root         29664 Sep 16 11:14 six.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 sphinxcontrib
-rw-r--r--    1 root     root           128 Sep 16 11:14 syspath.py
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 third_party
drwxr-xr-x    3 root     root          4096 Dec 17 16:38 tools
drwxr-xr-x    5 root     root          4096 Dec 17 16:38 typeshed
-rw-r--r--    1 root     root       3354133 Sep 16 11:14 virtualenv-16.7.10-py2.py3-none-any.whl

to make these helpers available again, following the docker documentation, you have to fix the volume. To do so:

docker rm {project_name}_pycharm_helpers_{pycharm_build}
docker run -v {project_name}_pycharm_helpers_{pycharm_build}:"/opt/.pycharm_helpers" pycharm_helpers:{tag}

et voilà

The option invalidate cache -> Clear downloaded shared indexes will also repopulate the Pycharm volumes. (At least in 2021.1)

Failing to get PyCharm to work with remote interpreter on docker

TLDR:

Long story

Tags:

Docker

Pycharm

Docker Compose

Related

Recent Posts