Why shouldn't `'` be used to escape single quotes?

" is on the official list of valid HTML 4 entities, but ' is not.

From C.16. The Named Character Reference ':

The named character reference ' (the apostrophe, U+0027) was introduced in XML 1.0 but does not appear in HTML. Authors should therefore use ' instead of ' to work as expected in HTML 4 user agents.

' is not part of the HTML 4 standard.

" is, though, so is fine to use.

" is valid in both HTML5 and HTML4.

' is valid in HTML5, but not HTML4. However, most browsers support ' for HTML4 anyway.