Keeping track of programs

Install each program in a dedicated directory tree, and use Stow or XStow to make all the programs appear in a common hierarchy. Stow creates symbolic links from the program-specific directory to a common tree.

In more detail, pick a toplevel directory, for example /usr/local/stow. Install each program under /usr/local/stow/PROGRAM_NAME. For example, arrange for its executables to be installed in /usr/local/stow/PROGRAM_NAME/bin, its man pages in /usr/local/stow/man/man1 and so on. If the program uses autoconf, then run ./configure --prefix /usr/local/stow/PROGRAM_NAME. After you've run make install, run stow:

./configure --prefix /usr/local/stow/PROGRAM_NAME
make
sudo make install
cd /usr/local/stow
sudo stow PROGRAM_NAME

And now you'll have symbolic links like these:

/usr/local/bin/foo -> ../stow/PROGRAM_NAME/bin/foo
/usr/local/man/man1/foo.1 -> ../../stow/PROGRAM_NAME/man/man1/foo.1
/usr/local/lib/foo -> ../stow/PROGRAM_NAME/lib/foo

You can easily keep track of what programs you have installed by listing the contents of the stow directory, and you always know what program a file belongs to because it's a symbolic link to a location under that program's directory. Uninstall a program by running stow -D PROGRAM_NAME then deleting the program's directory. You can make a program temporarily unavailable by running stow -D PROGRAM_NAME (run stow PROGRAM_NAME to make it available again).

If you want to be able to quickly switch between different versions of the same program, use /usr/local/stow/PROGRAM_NAME-VERSION as the program directory. To upgrade from version 3 to version 4, install version 4, then run stow -D PROGRAM_NAME-3; stow PROGRAM_NAME-4.

Older versions of Stow doesn't go very far beyond the basics I've described in this answer. Newer versions, as well as XStow (which hasn't been maintained lately) have more advanced features, like the ability to ignore certain files, better cope with existing symlinks outside the stow directory (such as man -> share/man), handle some conflicts automatically (when two programs provide the same file), etc.

If you don't have or don't want to use root access, you can pick a directory under your home directory, e.g. ~/software/stow. In this case, add ~/software/bin to your PATH. If man doesn't automatically find man pages, add ~/software/man to your MANPATH. Add ~/software/info to your INFOPATH, ~/software/lib/python to your PYTHONPATH, and so on as applicable.

You can use checkinstall to create a package (RPM, Deb, or Slackware compatible packages) That way, you can use your distros package manager to add/remove the application (but not update)

You use checkinstall in place of the make install command (using the -D parameter for Deb; -R is RPM and -S is Slackware):

root@nowhere# ./configure
root@nowhere# make
root@nowhere# checkinstall -D

checkinstall will build and install the package by default, or you can have it only build the package without installing.

checkinstall is available in most distros repositories.

For the most part this was the reason behind packages, ports, and other types of managers to prevent this type of thing from happening.

I would say that manual deletion is the only way for a manual install, unless someone else has a better answer to that point I may not be aware of.