Why cached reads are slower than disk reads in hdparm --direct?

Per hdparm man page:

--direct

Use the kernel´s "O_DIRECT" flag when  performing  a  -t  timing
test.   This  bypasses  the  page cache, causing the reads to go
directly from the drive into hdparm's buffers,  using  so-called
"raw"  I/O.  In many cases, this can produce results that appear
much faster than the usual page cache method,  giving  a  better
indication of raw device and driver performance.

It so explains why hdparm -t --direct may be faster than hdparm -t. It also says that --direct only applies to the -t test, not to the -T test which is not supposed to involve the disk (see below).

-T 

Perform timings of cache reads for benchmark and comparison pur‐
poses.   For  meaningful  results,  this  operation  should   be  
repeated  2-3  times  on  an otherwise inactive system (no other
active processes) with at least a couple of  megabytes  of  free
memory.   This  displays  the speed of reading directly from the 
Linux buffer cache without disk  access.   This  measurement  is  
essentially  an  indication  of the throughput of the processor,
cache, and memory of the system under test.

I guess -T works by reading the same cached part of the disk. But your --direct prevents this. So, logically, you should have the same results with -t --direct as with -T --direct.

Based on my research, --direct sends any data acquired through the hard drive at least once. That means if the system cache procures data at 10GB/sec but the poor hard drive controller bogs down at 400MB/sec, 400MB/sec is what you will get.

As a refresher

• -Tt --direct Does two runs:

  • 1. reads the system cache, then sends it to the hard drive controller and back. This can be useful perhaps for testing SATA connections and max buffer speeds.
  • 2. Reads the disk without it's buffer.

• -T reads the system cache directly, tests processor and raid driver overhead

• -t reads the disk with the buffer enabled