rsync not deleting Folders in destination

The directory won't be deleted if permissions forbid it, or if the directory is not empty. Often there are files that can't be deleted because they are immutable or otherwise protected. What does ls -a /usr show on the destination?

You might want to try the --force-delete flag, but first run it with -n (dry run mode) to see what it would delete.