What's the reason that "rm -rf a/b" says "a/b is not empty"?

There are other processes that are generating files to a/b, is that related?

Very possibly. It might be that rm -rf first removes all files, then removes all directories. Under the hood, the rmdir system call for removing a directory will fail if the directory is not empty. You might have a race condition going on, where rm -rf checks that the directory is empty, and it is; another process then creates a new file; finally rm -rf calls rmdir, but another process creates a file beforehand.

Faheem's advice is good: put an ls into your error condition

ssh -T user@host <<EOF
  cd somewhere
  rm -rf a/b || ( ls -a a/b && exit 1 )
EOF

rmdir(2) will fail if the directory is not empty. If another process is creating files while rm(1) is removing them, it will not know to delete them and consequently when it comes time for rm(1) to try to delete what it believes should be an empty directory, it will fail with the error you've posted.

One way to delete the directory in the face of concurrent file creations in the directory is to rename it:

mv a a~
rm -rf a~

It's possible that this may not work if the processes creating the files in a/b are not doing so by path (open(2) vs. openat(2)).

I am assuming that the process(es) that creates files in a/b will recreate that directory if it does not exist, or will handle failure gracefully if it does not exist. Since you are already trying to delete the directory from under other processes, that seems like a safe assumption.