Unable to pip install in Docker image as agent through Jenkins declarative pipeline

I have found what I myself would think is the prettier solution:

stage("Python Test") {
  agent { 
    docker {
      label "docker && linux" 
      image "python:3.7"
    }
  }
  steps {
    withEnv(["HOME=${env.WORKSPACE}"]) {
      sh "pip install -r requirements.txt --user"
      # python stuff
    }
  }
  post {
    cleanup {
      cleanWs()
    }
  }
}

This workaround steers completely around the issue itself, installing the packages at user level. The issue here was that the HOME-directory was not initially writeable either, thus overwriting the HOME directory.

I had a very similar pipeline that I was running right after setting up Docker agents on my Jenkins system, so I thought my setup was wrong. Using the comments in your thread, I cooked up this solution:

First, you'll need to be root inside your container, so change your agent declaration to be similar to this:

agent {
    docker {
        image "python:3.7"
        args '--user 0:0'
    }
}

Now I was able to use pip install! However, subsequent runs of the job would try to run git clean and fail since the built files inside the container were created by root. To fix that, I ran the clean command inside the container as my last step:

steps {
    sh 'git clean -fdx'
}

Update:

I found a problem where a failed build wouldn't clean up and killed all of the builds after it. To fix this, I put the clean action as a post-build task that always runs:

post {
    cleanup {
        cleanWs()
    }
}