Can multiple mv commands be run in the same directory (race condition)?

There is indeed a race condition (one that wouldn't cause harm, though).

The * is expanded on entry to the loop. If you run a second instance of this script simultaneously then it will probably do nothing because all files it tries to move have already been moved. If no files are created in the source directory during the moving operation then the error messages should be your biggest problem.

But in general this structure is a very bad idea. * expands to a sorted list. AFAIK it is not possible to deactivate that. Obviously, the sorting alone is a nightmare with 400K files. See man bash, section "Pathname Expansion":

After word splitting, unless the -f option has been set, bash scans each word for the characters *, ?, and [. If one of these characters appears, then the word is regarded as a pattern, and replaced with an alphabetically sorted list of file names matching the pattern.

Furthermore you should not run one mv instance per file as you can move several files at once.

This is a better solution (in the GNU world):

find . -mindepth 1 -maxdepth 1 -exec mv --target-directory=DIRECTORY {} +

An even better solution would be to use GNU Parallel to insert multiple arguments. By default, Parallel will run n jobs simultaneously, with n being the number of cores your CPU has.

When moving a lot of files like this: mv * destdir you will sometimes get the error:

bash: /bin/mv: Argument list too long

because there are too many files. You can instead do:

ls -1 | parallel mv {} destdir

This will run mv for each file. It can be done faster if mv gets as many arguments that will fit on the line:

ls -1 | parallel -m mv {} destdir


The -m option is really cool for moving or copying files in parallel:

-m      Multiple arguments. Insert as many arguments as the command
        line length permits. If multiple jobs are being run in
        parallel: distribute the arguments evenly among the jobs.
        Use -j1 to avoid this.

Tags:

Shell

Bash

Mv