Safe to remove /usr/src/linux-headers after purging older linux images?

Those are header files are contained in the linux-headers-* and linux-headers-*-generic packages. It should be safe to remove them through apt-get. Maybe apt-get autoremove will already suggest that to you. Please do not remove them manually!

I just had this issue, and the suggestion to use apt-get autoremove did not work for me.

I resolved it by doing:

sudo apt-get purge linux-headers-3.2.0-23

on such linux headers that lived in /usr/src. I did not remove the headers that correspond to the kernel currently being used.

Restarted the server successfully.