How to find the largest open files?
You can use the -F option of lsof to get almost unambiguous output which is machine-parseable with only moderate pain. The output is ambiguous because lsof rewrites newlines in file names to \n.
The lsof output consists of one field per line. The first character of each name indicates the field type and the rest of the line is the field value. The fields are: p=PID (only for the first descriptor in a given process), f=descriptor, t=type (REG for regular files, the only type that has a size), s=size (only if available), n=name. The awk code below collects entries that have a size and prints the size and the file name. The rest of the pipelines sorts the output and retains the entry with the largest size.
lsof -Fnst | awk '
{ field = substr($0,1,1); sub(/^./,""); }
field == "p" { pid = $0; }
field == "t" { if ($0 == "REG") size = 0; else next; }
field == "s" { size = $0; }
field == "n" && size != 0 { print size, $0; }
' | sort -k1n -u | tail -n42 | sed 's/^[0-9]* //'
One convoluted way looks like this:
lsof \
| grep REG \
| grep -v "stat: No such file or directory" \
| grep -v DEL \
| awk '{if ($NF=="(deleted)") {x=3;y=1} else {x=2;y=0}; {print $(NF-x) " " $(NF-y) } }' \
| sort -n -u \
| numfmt --field=1 --to=iec
Output
...
....
129M /var/log/maillog
166M /var/log/nginx/access_log
172M /var/log/metrics/kubernetes/kubelet.log
185M /var/log/metrics/kubernetes/etcd.log
257M /var/log/metrics/kubernetes/etcd.log.1
335M /var/log/metrics/kubernetes/kubelet.log.1
I know this is not perfect. For example if the file name contains "DEL" this would purge that file from the output list.
lsof also has a -F option described in the OUTPUT FOR OTHER PROGRAMS section. Using that may be simpler.
Details
lsof prints something like this:
COMMAND PID TID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root cwd DIR 253,0 4096 128 /
tuned 2975 root 7u REG 253,0 4096 805307770 /tmp/ffiKkVeXD (deleted)
python2 49888 49890 root DEL REG 0,18 196039884 /dev/shm/sem.NXPFow
systemd 1 root mem REG 253,0 90664 10063 /usr/lib64/libz.so.1.2.7
java 149435 175229 box 69r REG 253,0 350872273 808108999 /box/var/log/metrics/kubernetes/kubelet.log.1
java 149435 149580 box 107w FIFO 0,8 0t0 272526226 pipe
prometheu 147867 148211 root mem REG 253,6 31457463 /lib64/ld-2.12.so (stat: No such file or directory)
grep REG keep regular files
grep -v "stat: No such file or directory" Remove the files that have stat error. (I do not know why this happens)
grep -v DEL Discard the Linux map files that have been deleted;
From lsof documentation:
''DEL'' for a Linux map file that have been deleted;
After this processing we are left with something like this:
tuned 2975 root 7u REG 253,0 4096 805307770 /tmp/ffiKkVeXD (deleted)
systemd 1 root mem REG 253,0 90664 10063 /usr/lib64/libz.so.1.2.7
java 149435 175229 box 69r REG 253,0 350872273 808108999 /box/var/log/metrics/kubernetes/kubelet.log.1
The size is either the 3rd or the 2nd column from last depending on the value of the last column. If the last column is (deleted) pick the 3rd from last otherwise 2nd.
awk '{if ($NF=="(deleted)") {x=3;y=1} else {x=2;y=0}; {print $(NF-x) " " $(NF-y) } }'
sort -n -u | numfmt --field=1 --to=iec
Sort, uniquify and make the bytes count human readable
You can do the following
lsof | grep REG | awk '{ print $1,$7,$9 }' | sort -t ' ' -k 2 -V
Using the awk you filter the output to include command, size and filename and sort it based on the 2nd column, which is size. -t specifies the delimiter, -V sort 'naturally' - so 1, 2, 10 will be sorted this way instead of 1, 10, 2. -k is the key for the sort (the column you want to sort by)