Do LTO tapes have spare/unused capacity?

Solution 1:

If your drive is new and the tape is of good quality you can expect to be able to write more bytes to the tape than the official capacity. In some sense you can call that spare capacity, but it isn't unused.

As your drive heads wear down capacity will be decreasing. If you combine that with tapes of not as great quality the capacity can decrease even further.

Because capacity varies like that there need to be some way to signal to your backup application that you are out of capacity. It can be problematic for a backup application if it reaches the end of the tape and it wasn't prepared. It is better for the application with some advance warning such that it can use the remaining space to wrap up what it is doing.

If your OS happens to be Linux the API is such that every other write system call will fail with ENOSPC once you reach this last part of the tape. If your backup application doesn't know about this feature it would treat the first ENOSPC as the end, and there would be some unused space left on the tape.

I can imagine something similar can happen on other OS as well.

Solution 2:

Thanks to @kasperd I did some further investigation and this was indeed the problem. Turns out this feature is called the EWEOM (Early Warning End Of Media) and it refers to a marker placed on the tape by the tape manufacturer, so it is not the drive keeping track of how much tape has been spooled.

I wrote a patch for the mbuffer program I am using to write to the tape, and sure enough, at the point where I was reaching the end of the tape I get ENOSPC errors on alternating write() calls, but I can continue to write more data. In my case, quite a lot more data - between 8 and 19 GiB, depending on compression of my not-very-compressible data.

Interestingly after the EWEOM marker is reached, the tape write speed drops dramatically. It almost halves from 80MB/sec down to about 47MB/sec. This doesn't appear to be a data issue as the drive has maintained 80MB/sec for hours prior to this point. You can hear the drive motor running at a slower speed, and rewriting over a whole tape so this section gets rewritten doesn't increase the speed (so it's not a case of the first write being slower like it can be at the beginning of a brand new tape.)

I can't find any documentation about when the EWEOM marker should appear on the tape, so I'm not sure if it's standardised. All I could find is a vague reference to LTO-6/7 drives having this increased to be 5% of tape space, which seems like a lot. Perhaps this is to allow for large buffers to be flushed due to the tape's high write speed.

As far as the Linux API goes, the relevant line is in the st.c SCSI tape driver source code and the explanation of this behaviour is in the st driver documentation.