How to prevent dd's progress from being meaningless on Linux?

From the first line we can tell dd has read and written 1.5GB in one second. Even an SSD can't write that fast.

What happened is that the /dev/sdc block device accepted it (writeback), but didn't send it to disk but buffered it and started writing to disk at the rate the disk can take it. Something like 3MiB/s.

The system can't buffer data indefinitely like that, there's only so much data it will accept to hold in that non-committed dirty state. So after a while (in your case, after more than 1.5GB have been written but less than 2 seconds have passed (as progress lines are written every second)), dd's write() system call will block until the data has been flushed to the disk (during which it cannot write progress messages). When it gets through, dd can send the few extra missing megabytes, and that happens within less than a second, so you get only one extra progress line.

To see a different behaviour, you could force the writes to be synchronous, that is not to return unless the data has been committed to disk. For instance by using oflag=sync or oflag=dsync or oflag=direct (not that I would advise doing that though).


When you write a file to a block device, use dd with oflag=direct. This uses O_DIRECT writes, which avoids using your RAM as a writeback cache. Note that to get good performance, oflag=direct usually needs a large block size.

This will avoid seeing impossibly fast progress, unless you have a weird device which itself has a very large RAM cache.

A lot of devices have a small amount of cache. In this case oflag=direct will show a realistic rate of progress. This is more meaningful, but it does not tell you everything you want to know :-). It does not guarantee the last writes are finished when dd says it is finished. You can make sure all writes are finished - and also check for any write errors - by using the option conv=fsync. This calls fsync() at the end to make sure the cache is flushed. Here is an example:

dd if=my.iso of=/dev/sdc oflag=direct bs=4M status=progress conv=fsync

Some people run the sync command afterwards, and understandably do not bother to remember conv=fsync. This is not quite as good. sync will not report whether one of the writes failed.

If the device has a very large RAM cache, you would have to use oflag=direct,sync. But normally I think of oflag=sync as a potential barrier to performance. You might want to increase the block size even further, to decrease the frequency of cache flushes. When doing very synchronous IO and reading multiple hardware blocks at a time like this, you might want to use double-buffering to maintain good performance, i.e. using a second dd command as in the link below.

Sometimes you might also want to pipe a disk image from another program, such as gunzip. In this case, good performance also depends on iflag=fullblock and piping through another dd command. There is a full example in the answer here: Why does a gunzip to dd pipeline slow down at the end?

Tags:

Dd