Running cron python jobs within docker
Several issues that I faced while trying to get a cron job running in a docker container were:
- time in the docker container is in UTC not local time;
- the docker environment is not passed to cron;
- as Thomas noted, cron logging leaves a lot to be desired and accessing it through docker requires a docker-based solution.
There are cron-specific issues and are docker-specific issues in the list, but in any case they have to be addressed to get cron working.
To that end, my current working solution to the problem posed in the question is as follows:
Create a docker volume to which all scripts running under cron will write:
# Dockerfile for test-logs # BUILD-USING: docker build -t test-logs . # RUN-USING: docker run -d -v /t-logs --name t-logs test-logs # INSPECT-USING: docker run -t -i --volumes-from t-logs ubuntu:latest /bin/bash FROM stackbrew/busybox:latest # Create logs volume VOLUME /var/log CMD ["true"]
The script that will run under cron is
#!/usr/bin/env python # python script which needs an environment variable and runs as a cron job import datetime import os test_environ = os.environ["TEST_ENV"] print "Cron job has run at %s with environment variable '%s'" %(datetime.datetime.now(), test_environ)
In order to pass the environment variable to the script that I want to run under cron, follow Thomas' suggestion and put a crontab fragment for each script (or group of scripts) that has need of a docker environment variable in
/etc/cron.d with a placeholder
XXXXXXX which must be set.
# placed in /etc/cron.d # TEST_ENV is an docker environment variable that the script test.py need TEST_ENV=XXXXXXX # * * * * * root python /test.py >> /var/log/test.log
Instead of calling cron directly, wrap cron in a python script that does does things: 1. reads the environment variable from the docker environment variable and sets the environment variable in a crontab fragment.
#!/usr/bin/env python # run-cron.py # sets environment variable crontab fragments and runs cron import os from subprocess import call import fileinput # read docker environment variables and set them in the appropriate crontab fragment environment_variable = os.environ["TEST_ENV"] for line in fileinput.input("/etc/cron.d/cron-python",inplace=1): print line.replace("XXXXXXX", environment_variable) args = ["cron","-f", "-L 15"] call(args)
Dockerfile that for the container in which the cron jobs run is as follows:
# BUILD-USING: docker build -t test-cron . # RUN-USING docker run --detach=true --volumes-from t-logs --name t-cron test-cron FROM debian:wheezy # # Set correct environment variables. ENV HOME /root ENV TEST_ENV test-value RUN apt-get update && apt-get install -y software-properties-common python-software-properties && apt-get update # Install Python Setuptools RUN apt-get install -y python cron RUN apt-get purge -y python-software-properties software-properties-common && apt-get clean -y && apt-get autoclean -y && apt-get autoremove -y && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* ADD cron-python /etc/cron.d/ ADD test.py / ADD run-cron.py / RUN chmod a+x test.py run-cron.py # Set the time zone to the local time zone RUN echo "America/New_York" > /etc/timezone && dpkg-reconfigure --frontend noninteractive tzdata CMD ["/run-cron.py"]
Finally, create the containers and run them:
- Create the log volume (test-logs) container:
docker build -t test-logs .
- Run log volume:
docker run -d -v /t-logs --name t-logs test-logs
- Create the cron container:
docker build -t test-cron .
- Run the cron container:
docker run --detach=true --volumes-from t-logs --name t-cron test-cron
- To inspect the log files of the scripts running under cron:
docker run -t -i --volumes-from t-logs ubuntu:latest /bin/bash. The log files are in
Adding crontab fragments in
/etc/cron.d/ instead of using root's
crontab might be preferable.
- Let you add additional cron jobs by adding them to that folder.
- Save you a few layers.
- Emulate how Debian distros do it for their own packages.
Observe that the format of those files is a bit different from a crontab entry. Here's a sample from the Debian php package:
# /etc/cron.d/php5: crontab fragment for php5 # This purges session files older than X, where X is defined in seconds # as the largest value of session.gc_maxlifetime from all your php.ini # files, or 24 minutes if not defined. See /usr/lib/php5/maxlifetime # Look for and purge old sessions every 30 minutes 09,39 * * * * root [ -x /usr/lib/php5/maxlifetime ] && [ -x /usr/lib/php5/sessionclean ] && [ -d /var/lib/php5 ] && /usr/lib/php5/sessionclean /var/lib/php5 $(/usr/lib/php5/maxlifetime)
Overall, from experience, running cron in a container does work very well (besides cron logging leaving a lot to be desired).
Here is a complement on rosksw answer.
There is no need to do some string replacement in the crontab file in order to pass environment variables to the cron jobs.
It is simpler to store the environment variables in a file when running the contrainer, then load them from this file at each cron execution. I found the tip here.
In the dockerfile:
CMD mkdir -p /data/log && env > /root/env.txt && crond -n
In the crontab file:
* * * * * root env - `cat /root/env.txt` my-script.sh