Have a file named ~ (tilde) in my home-directory

The pretty much ultimate solution when it comes to files that can't be deleted by normal means:

ls -il 

The first column will show the inode number of the files.

find . -inum [inode-number] -exec rm -i {} \;

This will delete the file with the specified inode-number after verification.

You should be able to refer to that file as ~/~ (without quotes) because tilde-expansion only applies the the tilde (~) at the very beginning of the word.

Quote it (rm '~') or escape it (rm \~).

---

It's always either of those (also for e.g. $), or add -- to prevent the file name from being interpreted as argument: rm -- -i removes the file named -i; also useful for rm -- * when you want to delete all files in the current directory: No accidental rm -f * just because a file is named like that.