Not all self-adjoint operators are observables?
The appendix of Mackey talks about superselection rules, and indeed superselection is the phenomenon where there are self-adjoint operators that are not observables. Whether this is obvious or not depends on how one defines "superselection".
The standard definition would be that the Hilbert space $H$ splits into the direct sum $H_1\oplus H_2$ such that for all $\lvert\psi\rangle\in H_1, \lvert \phi\rangle\in H_2$ and all observables $A$ we have that $\langle \psi \vert A\vert \phi \rangle = 0$, which implies that $AH_1 \subset H_1,AH_2\subset H_2$ for all observables, but which is clearly not true for all self-adjoint operators on $H_1\oplus H_2$.
An easy (albeit somewhat artificial) example of a superselected system is when we take $H_1$ to be the state space of a boson and $H_2$ the state space of a fermion, see also this answer of mine. Other examples can arise from theories with spontaneous symmetry breaking where states belonging to different VEVs cannot interact with each other and form superselection sectors.
In general, for physical reasons not all self-adjoint operators are observables. I will discuss the problem at the level of von Neumann algebras that are (perhaps unawares) more familiar to physicists. There are two further possibilities leading to a bit differently articulated answers: $C^*$-algebras and lattice of elementary propositions theory.
Let us start from observing that self-adjoint operators of a quantum system $S$ are not enough to exhaust the set of operators useful in a description of $S$ itself. For instance, if $A$ is an observable $U_a = e^{-iaA}$, where $a\in \mathbb R$, defines a group of continuous symmetries associated to that observable and this operator (for fixed $a \in \mathbb R$) is not self-adjoint.
The class of useful operators is constructed this way. First notice that the observables can always be reduced to a larger class of bounded operators: if $A$ is unbounded (i.e. the set of attained values defining its spectrum $\sigma(A)$ is an unbounded subset of $\mathbb R$) the class $\{A_n\}_{n \in \mathbb N}$ of bounded self-adjoint operators $$A_n = \int_{(-n,n] \cap \sigma(A)} a dP^{(A)}(a)\tag{1}$$ where $P^{(A)}$ is the spectral measure of $A$, encompasses the whole information of $A$ itself. In particular, if $\psi \in D(A)$ then $$A\psi = \lim_{n\to +\infty}A_n \psi\tag{2}$$ and $$\cup_{n \in \mathbb R} \sigma(A_n) = \sigma(A) \:\:\: \mbox{up to the possible element $0 \in \sigma(A)$}$$ (2) says that the class of bounded operators $A_n$ constructs $A$ in the strong operator topology. This way every observable is de-constructed into a set of bounded observables. Next one considers products of all possible complex combinations of bounded observables including infinite combinations in the strong-operator topology. The obtained set of operators $R_S$ includes all conceivable operators useful for a quantum system extracted from observables (including the bounded observables themselves coinciding with the self adjoint elements of $R_S$, their spectral measures and the continuous symmetries generated from them).
This class of operators $R_S$ is known as the von Neumann algebra (or $W^*$ algebra) of a given quantum system $S$.
The natural question, which in complex Hilbert spaces is equivalent to your initial question reads as follows.
For a quantum system described in the Hilbert space $H$, does $R_S = B(H)$?
where $B(H)$ is the largest von Neumann algebra in $H$ consisting of all bounded operators $A : H \to H$.
(Indeed, if all self-adjoint operators are observables, then $R_S$ includes all bounded self-adjoint operators and their complex combinations and thus it includes the full $B(H)$ because every bounded operator is a complex linear combination of a couple of bounded self-adjoint operators. If, vice versa, $R_S = B(H)$, then every self adjoint operator is an observable since, by definition, all self-adjoint operators in $R_S$ are the observables of $S$.)
Physics decides actually.
There are two independent possibilities when $R_S \subsetneq B(H)$.
$\:\:\:\:$(A) Presence of Abelian superselection rules.
This means that there are orthogonal projectors $P_k$ in $R_S$ such that
(i) $P_k$ commutes with every element of $R_S$,
(ii) $P_k \perp P_h$ if $k \neq h$,
(iii) $\oplus_k P_k =I$.
The projection spaces $H_k = P_k(H)$ are called superselection sectors and $H$ is the orthogonal sum of them $H= \oplus_k H_k$ due to (ii) and (iii).
In this case, evidently $R_S \subsetneq B(H)$ because, barring trivial cases, $B(H)$ includes some operators which do not commute with some $P_k$.
On the physical side this means that for instance there is no way to distinguish a vector state $$\psi = \sum_k c_k\psi_k\tag{1}$$ where $\psi_k \in H_k$ are unit vectors, and the mixture $$\rho_\psi = \sum_k |c_k|^2|\psi_k\rangle \langle \psi_k|$$ Using the fact that $A^\dagger=A \in R_S$ commutes with every $P_k$ it is easily proved, for instance, that $$tr(\rho_\psi A) = \langle \psi|A\psi \rangle$$ but the result extends to probabilities of outcomes and so on. Another way to physically illustrate this phenomenon is saying that
no coherent superposition of pure states of different superselection sectors $H_k$ is possible
As a typical example, consider the observable electric charge $Q$ for an electrically charged quantum system (also a quantum field). $Q$ has a discrete, in general unbounded, spectrum and the superselection rule of electric charge states that all observables commute with it. It immediately implies that the eigenspaces $H_q$ of $Q$ are superselection sectors and that an Abelian superselection rule takes place. All that is equivalent to stating that no coherent superposition of states with different charge is allowed.
Regarding the initial question, consider for instance two different values of the charge $q$ and $q'$ and associated eigenvectors $|q\rangle$ and $|q'\rangle$. The self-adjoint operator $$A = |q\rangle\langle q'| + |q'\rangle\langle q|$$ does not define an observable because it does not commute with the projector $P_q$ onto $H_q$.
Similar well-known superselection rules in quantum physics are the superselection rule of mass (Bargmann's superselection rule) for non-relativistic quantum systems and the superselection rule of angular momentum (integer values vs semi-integer values).
$\:\:\:\:$(B) The theory admits a (non-Abelian) gauge group.
This means that there is a class of observables $R'_S$, generally called the commutant of $R_S$, whose elements commute with every element of $R_S$ but some elements of the commutant are not included in $R_S$ itself, as it happens for (A) instead. This is in particular the case if the commutant $R_S'$ includes a pair of non-commuting operators at least. Let us stick to this case (which is the only possible actually due to the "double commutant theorem"). It is easy to prove that the unitary elements of $R_S'$ give rise to a non-Abelian unitary group. In other words all observables of $R_S$ must commute with unitary operators forming a non-abelian group, the (non-Abelian) gauge group of the theory.
An example consists of the description of quarks: all their observables must commute with a unitary representation of $SU(3)$ (color).
As an example of self-adjoint operators which are not observables in this case, just consider the self-adjoint generators of the relevant $SU(3)$ representation: they cannot commute with the representation (unless trivial) so they cannot be interpreted as quark observables because all observables commute with the representation.
(I discussed these topics in my 2013 Springer book on "Spectral theory and QM" and in the next very enlarged 2018 edition in print which includes an even wider discussion.)
ADDENDUM. It should be clear that in both cases where not all self-adjoint operators are observables the one-to-one correspondence
pure states $\leftrightarrow$ unit vectors up to phases
dramatically fails.
An example was provided in the case of Abelian superselection rules. Notice that, referring to that example $\psi$, $\rho_\psi$ but also (comparing with (1)) $$\psi' = \sum_k e^{i\theta_k}c_k \psi_k$$ are physically indistinguishable. In the case of presence of a gauge group $\psi$ and $U\psi$, where $U$ is a unitary operator of the gauge group, determine the same state.
Another physically important consequence of the presence of a gauge group is that no maximal set of commuting observables may exist. The proof is a bit technical and I omit it. In practice, we cannot prepare the quantum system in a preferred state by a selective sequence of measurements of compatible observables (with pure point spectrum).