Does the Cayley–Hamilton theorem work in the opposite direction?

Even without counterexamples it is obvious that your statement can't be true because if $A$ is a root of the polynomial $p(x)$ then it must be the root of $p(x)q(x)$ for any polynomial $q$. So that way we would get the matrix $A$ has infinitely many characteristic polynomials.

No. For example, $I-1=0$, but the characteristic polynomial of $I$ is $(x-1)^n$.

For any $n \times n$ matrix $A$, the $(2n) \times (2n)$ matrix $\pmatrix{A & 0\cr 0 & A\cr}$ satisfies the characteristic polynomial of $A$, but its own characteristic polynomial is the square of that of $A$.