Launching Multiple Queries With Bash Script

Due to https://mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P (output from parallel jobs in xargs risks being mixed), I would use GNU Parallel instead:

cat infile |
  parallel -P0 -q curl {} -o /dev/null --silent --head --write-out "%{http_code} {}\n" > outfile

In this particular case it may be safe to use xargs because the output is so short, so the problem with using xargs is rather that if someone later changes the code to do something bigger, it will no longer be safe. Or if someone reads this question and thinks he can replace curl with something else, then that may also not be safe.

This may be faster:

doit() {
  while read LINE; do
    curl -o /dev/null --silent --head --write-out "%{http_code} $LINE\n" "$LINE"
  done
}
export -f doit
parallel -j0 --pipepart -a infile --block -10 doit > outfile

Tags:

Http

Bash

Curl