Concurrent writing to a log file from many processes

When a file is opened in append mode, the OS guarantees that all writes take place at the end. So the data from one writer will not overwrite the data from another writer.

This only applies if the file is opened in append mode, i.e. with >> in the shell. If the creator of the file opens it with > then that guarantee won't apply and it's possible to have this sequence:

• Process 1: >out; now at position 0
• Process 2: >>out; now at position 0
• Process 1: write hello, now at position 6
• Process 2: write world, this is written at position 6 and process 2 is now at position 12
• Process 1: write oops, this is written at position 6 because process 1's file position has not changed.

On Debian (since 2001 or thereabouts), the file .xsession-errors is created by `/etc/X11/Xsession and it is opened in append mode, so everything is fine:

 exec >>"$ERRFILE" 2>&1

I don't know if that's the case on all distributions that log to ~/.xsession-errors.

As long as everybody opens the file in append mode, all output will be present. The output may be fragmented however. In practice, small enough writes to a regular file are atomic. Anything less than 512B should be small enough everywhere, and I think Linux guarantees more than that¹. So each log line should appear intact, even with multiple concurrent writers, assuming that the writers use line-buffered output and that the lines are not overly long.

¹ _{Note that POSIX does not guarantee anything except for pipes.}

Using >> in a POSIX shell guarantees that the file will be opened with O_APPEND.

The Open Group Base Specifications Issue 7 states:

If the O_APPEND flag of the file status flags is set, the file offset shall be set to the end of the file prior to each write and no intervening file modification operation shall occur between changing the file offset and the write operation.

POSIX defines the mimimum number of bytes that can be requested to be written in a single write(2) call (SSIZE_MAX = 32,767). The return value is the number of bytes actually written (guaranteed atomic).

However

Not all filesystems comply. Appending to a File from Multiple Processes says:

The caveat is that not all filesystems are POSIX-compatible. Two famous examples are NFS and the Hadoop Distributed File System (HDFS). On these networked filesystems, appends are simulated and subject to race conditions.

Not everyone plays nice

While you might open using O_APPEND, the other processes writing to the file may not. You can check for any given file:

lsof +fg <file>

Worryingly, when I run lsof +fg ~/.xsession-errors, I see no AP (append) flag, suggesting that the 22 processes in my listing (Manjaro Linux based on Arch) are not opening the file safely.

Only when I run cat >> ~/.xsession-errors in another shell does the final output line include the AP flag:

cat       3099 ravi    1w   REG   W,AP,LG   0,48      963 1926479 .xsession-errors

If anyone knows where this issue should be raised upstream, please comment.

Practically

If all processes open the file locally with either:

• open(2) and flag O_APPEND
• fopen(3) and the "a" flag
• POSIX sh >> or bash &>>

Then no data should be overwritten via race conditions.

Thanks to @Gilles answer for pointing me in the right direction.