download file via http only if changed since last update

Consider using curl instead of wget:

curl -o "$file" -z "$file" "$uri"

man curl says:

-z/--time-cond <date expression>

(HTTP/FTP) Request a file that has been modified later than the given time and date, or one that has been modified before that time. The date expression can be all sorts of date strings or if it doesn't match any internal ones, it tries to get the time from a given file name instead.

If $file doesn't necessarily pre-exist, you'll need to make the use of the -z flag conditional, using test -e "$file":

if test -e "$file"
then zflag="-z '$file'"
else zflag=
fi
curl -o "$file" $zflag "$uri"

(Note that we don't quote the expansion of $zflag here, as we want it to undergo splitting to 0 or 2 tokens).

If your shell supports arrays (e.g. Bash), then we have a safer and cleaner version:

if test -e "$file"
then zflag=(-z "$file")
else zflag=()
fi
curl -o "$file" "${zflag[@]}" "$uri"

The wget switch -N only gets the file if it has changed so a possible approach would be to use the simple -N switch which will get the file if it needs to but leaves it with the wrong name. Then create a hard link using the ln -P command to link it to a "file" with the correct name. The linked file has the same metadata as the original.

The only limitation being that you cannot have hard links across file system boundaries.


Python 3.5+ script for wrapping curl command:

import argparse
import pathlib

from subprocess import run
from itertools import chain

parser = argparse.ArgumentParser()
parser.add_argument('url')
parser.add_argument('filename', type=pathlib.Path)
args = parser.parse_args()

run(chain(
    ('curl', '-s', args.url),
    ('-o', str(args.filename)),
    ('-z', str(args.filename)) if args.filename.exists() else (),
))