How to dockerize Python environments

Inspired by this article, I also wanted to “dockerize” my Python environments.

Why dockerize? 🐳

The advantages of using Docker are:

• Reproducability: No “worked on my machine” problems.
• Isolation: No (negative) interferences between projects.
• Cleanliness: Keep your base system clean from any project specific stuff.
• Ease of use: You will not even realize that you are using Docker.

Source: imgflip Meme Generator

Pure Python

Let’s first talk about plain vanilla Python. If you want to run Python in a container, you may just run:

docker run -it python:3 python

This will open an interactive Python shell within the container:

Python 3.10.2 (main, Feb  8 2022, 05:20:04) [GCC 10.2.1 20210110] on linux

Type "help", "copyright", "credits" or "license" for more information.

>>>

Nice!

But this approach brings two problems:

Problem #1: No local file access

You cannot access local files, because the container has its own filesystem.
This problem can be solved using Docker bind mounts:

docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app python:latest python

The current directory (from the environment variable$PWD) is mounted (-v) to /usr/src/app in the container and the working directory (-w) is also set to /usr/src/app. So now, you can access all files and folders in the current directory from within the container!

But to be really useful in our daily work, we can put all of this into shell aliases.
So, for all Python versions you want to use you can have shell alias:

alias python3.8='docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app python:3.8 python'
alias python3.9='docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app python:3.9 python'
alias python3.10='docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app python:3.10 python'
alias python3.11='docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app python:3.11 python'
alias python='docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app python:latest python'

Source: imgflip Meme Generator

If you need to run a script in Python 3.8, you just run python3.8 my_script.py and it will spin up a Docker container and run your script within this container! As a result, you have all Python versions you can think of at your fingertips!

Problem #2: Ephemeral dependencies

If you install dependencies (i.e. via pip), you will lose them if you restart the container.
This problem can be solved using Docker volumes:

docker run --rm -it -v my_volume:/usr/local/lib python:latest python

This will create a volume my_volume (if not already existing) and mount it to /usr/local/lib in the container. If you now install dependencies via pip, they will be persisted in the volume and also be available after a container restart.

So the combination of the two solutions looks like:

docker run --rm -it -v "${PWD}":/usr/src/app -w /usr/src/app -v my_volume:/usr/local/lib python:latest python

In this case, for every environment you want install dependencies in, you will need a different Docker volume.

But how could you automize this to have automagic 🪄 Python environments?

Python environments

You probably want an isolated Python environment for each project you are working on. And each project probably has its own folder. So why not use the project path as the Docker volume name?
For better readability, the path should be slugified. So for a project path /Users/JohnDoe/Documents/dev/my-cool-python-project the Docker volume name would be users-johndoe-documents-dev-my-cool-python-project.
This can be automized using shell functions:

slugify() {
    echo "$1" | iconv -t ascii//TRANSLIT | sed -E 's/[^a-zA-Z0-9-]+/-/g' | sed -E 's/^-+|-+$//g' | tr A-Z a-z
}

env_name() {
    echo $(slugify ${${1:-$PWD}: -200})
}

The shell function slugify slugifies a given input. The env_name function returns the slugified version of the current directory ($PWD) with a maximum lenght of 200. The output of env_name can now be the name of our Docker volume for the Python environment! 👍

In your bashrc file (i.e. .zshrc or .bashrc) you just add (together with slugify and env_name from ahead):

py-env() {
    docker run --rm -it -v "$(env_name $1)_python":/usr/local/lib -v "${PWD}":/usr/src/app -w /usr/src/app python:latest bash
}

So now, you just run py-env from within your project directory and your Python environment is ready to go! 🚀

You can install dependencies pip and run Python as you did it before! The only difference is, that this runs in a Docker container and you have a clean and isolated environment where you can install any dependencies without worrying about messing your entire system!

But what if you also have projects using older Python versions? No problem, we just define a shell function for this! For Python version 3.8 you would have:

py-env-3.8() {
    docker run --rm -it -v "$(env_name $1)_python":/usr/local/lib -v "${PWD}":/usr/src/app -w /usr/src/app python:3.8 bash
}

Delete Python environment 🧹

Using the following shell function you can delete a Python environment:

py-env-del() {
    docker volume rm "$(env_name $1)_python"
}

This will delete the corresponding Docker volume.

Complete Code

The complete code is part of the accompanying git repository:

GitHub - FranzDiebold/dockerize-datascience: Dockerize DataScience

And don’t forget to ⭐️!

This article is part of a multi-part series “How to dockerize [x]”.
The next article is “How to dockerize Data Science”.

If you appreciate this post, here are a few things you can do to support my work:

1. Give this story a clap. 👏
2. Subscribe to my upcoming stories.
3. Follow me on GitHub: https://github.com/franzdiebold