Struggling with understanding the reason why Python needs Virtual Environments

Virtual environments make possible for you to encapsulate dependencies by project.

Python has no node_modules equivalent. When you install something with pip it goes to your site-packages folder. To find out this folder you can run python -m site and it will print out the folders where it will search for packages.

Example on Fedora 29:

➜  ~ python -m site
sys.path = [
    '/home/geckos',
    '/usr/lib/python27.zip',
    '/usr/lib64/python2.7',
    '/usr/lib64/python2.7/plat-linux2',
    '/usr/lib64/python2.7/lib-tk',
    '/usr/lib64/python2.7/lib-old',
    '/usr/lib64/python2.7/lib-dynload',
    '/usr/lib64/python2.7/site-packages',
    '/usr/lib/python2.7/site-packages',
]
USER_BASE: '/home/geckos/.local' (exists)
USER_SITE: '/home/geckos/.local/lib/python2.7/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

pip vs package manager

If you don't use virtual environments you may end up with packages being installed side by side with operating system python packages, and this is where the danger is. Packages may be overwritten and things get messy fast. For example you install Flask with pip then try to install Jinja2 from with package-manager, now you remove Jinja2, and breaks Flask, or you update your system, Jinja2 got updated but not Flask. Or even simpler, you install something with package manager and remove with pip, now the package manager is in a broken state.

Because of this we always use virtual environments, and even separate virtual environments by project.

Creating and maintaining virtual environments

Nothing prevents you from maintaining you virtual environment in the same folder as your project. This way you will have the same felling that you have with node_modules. You can create it with

virtualenv <SOME_FOLDER> for python 2 or python3 -m venv <SOME_FOLDER> for python 3

Conventions that I've seen

If you're keeping virtual environments as a subfolder of your project, I usually call then env or venv

Other options is keeping all then in the same folder inside your home, I've been using ~/.venv/<PROJECT>

Pipenv

Finally there is an alternative that I like more than pure pip. Pipenv is a tool that manages virtual environments automatically for you. It feels more close to yarn and has more features

To create a virtual environment for a project just pipenv --tree or pipenv --two in your project folder. It will create and manage the virtual environment and write dependencies to Pipenv file. It also support development packages, I really think is worth trying. Here is the docs: https://pipenv.kennethreitz.org/en/latest/

I hope this helps, regards