Best way to free disk space from deleted files that are held open

On unices, filenames are just pointers (inodes) that point to the memory where the file resides (which can be a hard drive or even a RAM-backed filesystem). Each file records the number of links to it: the links can be either the filename (plural, if there are multiple hard links to the same file), and also every time a file is opened, the process actually holds the "link" to the same space.

The space is physically freed only if there are no links left (therefore, it's impossible to get to it). That's the only sensible choice: while the file is being used, it's not important if someone else can no longer access it: you are using it and until you close it, you still have control over it - you won't even notice the filename is gone or moved or whatever. That's even used for tempfiles: some implementations create a file and immediately unlink it, so it's not visible in the filesystem, but the process that created it is using it normally. Flash plugin is especially fond of this method: all the downloaded video files are held open, but the filesystem doesn't show them.

So, the answer is, while the processes have the files still opened, you shouldn't expect to get the space back. It's not freed, it's being actively used. This is also one of the reasons that applications should really close the files when they finish using them. In normal usage, you shouldn't think of that space as free, and this also shouldn't be very common at all - with the exception of temporary files that are unlinked on purpose, there shouldn't really be any files that you would consider being unused, but still open. Try to review if there is a process that does this a lot and consider how you use it, or just find more space.

Files are deleted from filesystem where is deleted any reference to this inode. Reference can be on disk (link in any directory), and.. from open applications. If you remove file - you only delete reference from disk, but - there is still reference from application.

You can "free" space in two ways then:

1. as mentioned above - you can kill application, which open file.
2. you can... truncate file. Even if it's deleted:

If you know pid - look what files are open by this pid: ls -l /proc/PID/fd you see here links like:

undefine@uml:~$ ls -l /proc/18596/fd
razem 0
lrwx------ 1 undefine undefine 64 lut  1 00:06 0 -> /dev/pts/30
lrwx------ 1 undefine undefine 64 lut  1 00:06 1 -> /dev/pts/30
lrwx------ 1 undefine undefine 64 lut  1 00:05 2 -> /dev/pts/30
lr-x------ 1 undefine undefine 64 lut  1 00:06 3 -> /home/undefine/x (deleted)
lr-x------ 1 undefine undefine 64 lut  1 00:06 4 -> anon_inode:inotify

As you see - 3 fd is deleted. you can truncate it by command (for example):

undefine@uml:~$ :>/proc/18596/fd/3
undefine@uml:~$ 

Remember that if application read from this file - it can be dangerous to them. But - if it's only a log file - you can truncate it safetly.

As others have said lsof can be used to list all the deleted files which are still on disk due to open file descriptors. However, this may be a very long list. Here is a command which lists these files sorted by ascending size in bytes:

sudo lsof -F sn0 | tr -d '\000' | grep deleted | sed 's/^[a-z]*\([0-9]*\)n/\1 /' | sort -n

There may be a more succinct way of doing this but the above command worked for me.