disk write performance in golang
I think I get the answer to this question. Because my disk I/O speed is about 194M/s, and disk buffer I/O is about 9093M/s. But disk buffer size is about 18166M. So the writing speed is faster when file size is smaller than 20G. I used this command to test my disk I/O speed:
hdparm -Tt /dev/sde1
/dev/sde1:
Timing cached reads: 18166 MB in 2.00 seconds = 9093.93 MB/sec
Timing buffered disk reads: 584 MB in 3.01 seconds = 194.18 MB/sec
Your issue is not reproducible. Your code has bugs.
We would expect that disk write time would be affected by a number of factors: the program, other programs, the operating system, hardware, and so on.
Boot a standalone, dedicated machine, and run the diskio.go
program. What results do you get? For example, on my machine:
$ go build diskio.go
$ time ./diskio -size=32
written: 34359738368B 154333936544ns 34.36GB 154.33s 222.63MB/s
real 2m35.323s
user 0m6.418s
sys 0m41.994s
$ time ./diskio -size=16
written: 17179869184B 77901269159ns 17.18GB 77.90s 220.53MB/s
real 1m18.746s
user 0m2.849s
sys 0m21.721s
$ time ./diskio -size=8
written: 8589934592B 38940248134ns 8.59GB 38.94s 220.59MB/s
real 0m39.625s
user 0m1.719s
sys 0m12.493s
$ time ./diskio -size=1
written: 1073741824B 4738082404ns 1.07GB 4.74s 226.62MB/s
real 0m4.851s
user 0m0.069s
sys 0m0.755s
$
As expected, little time is spent in the program, more time is spent in the operating system, a lot of time is spent waiting on the disk. There is no precipitous change in write speed.
diskio.go
:
package main
import (
"bufio"
"flag"
"fmt"
"os"
"time"
)
func writeFile(fSize int64) error {
fName := `/home/peter/diskio` // test file
defer os.Remove(fName)
f, err := os.Create(fName)
if err != nil {
return err
}
const defaultBufSize = 4096
buf := make([]byte, defaultBufSize)
buf[len(buf)-1] = '\n'
w := bufio.NewWriterSize(f, len(buf))
start := time.Now()
written := int64(0)
for i := int64(0); i < fSize; i += int64(len(buf)) {
nn, err := w.Write(buf)
written += int64(nn)
if err != nil {
return err
}
}
err = w.Flush()
if err != nil {
return err
}
err = f.Sync()
if err != nil {
return err
}
since := time.Since(start)
err = f.Close()
if err != nil {
return err
}
fmt.Printf("written: %dB %dns %.2fGB %.2fs %.2fMB/s\n",
written, since,
float64(written)/1000000000, float64(since)/float64(time.Second),
(float64(written)/1000000)/(float64(since)/float64(time.Second)),
)
return nil
}
var size = flag.Int("size", 8, "file size in GiB")
func main() {
flag.Parse()
fSize := int64(*size) * (1024 * 1024 * 1024)
err := writeFile(fSize)
if err != nil {
fmt.Fprintln(os.Stderr, fSize, err)
}
}
Playground: https://play.golang.org/p/vnjnpgMzsV