Building Dockerfile fails when touching a file after a mkdir

This is currently investigated in docker/docker/issues/3639, and summarized in this comment:

Okay, I did little research and it seems that volume is non-mutable between Dockerfile instruction.
Here even smaller Dockerfile for testing:

FROM busybox

RUN mkdir /tmp/volume
RUN echo "hello" > /tmp/volume/hello
VOLUME ["/tmp/volume/"]
RUN [[ -f /tmp/volume/hello ]]
RUN rm /tmp/volume/hello
RUN [[ ! -e /tmp/volume/hello ]]

On each instruction we create new volume and copy content from original volume.

Update April 2019:

Use DOCKER_BUILDKIT=1
The new builder does not exhibit this behavior.

Example from dominikzalewski:

That's a very simple Dockerfile that I'm using:

FROM wordpress:latest
ARG UPLOAD_DIR=/var/www/html/wp-content/uploads

RUN mkdir -p $UPLOAD_DIR
RUN ls -lhd $UPLOAD_DIR

Cf. Build Enhancements for Docker

Docker Build enhancements for 18.09 release introduces a much-needed overhaul of the build architecture.
By integrating BuildKit, users should see an improvement on performance, storage management, feature functionality, and security.

• Docker images created with buildkit can be pushed to Docker Hub and DTR just like Docker images created with legacy build
• The Dockerfile format that works on legacy build will also work with buildkit builds
• The new --secret command line option allows the user to pass secret information for building new images with a specified Dockerfile

Looking at https://registry.hub.docker.com/u/library/jenkins/, it seems that /var/jenkins_home is a volume. You can only create files there while the container is running, presumably with a volume mapping like

docker run ... -v /your/jenkins/home:/var/jenkins_home ...

The docker build process knows nothing about shared volumes.