How many $3 \times 3$ non-symmetric and non-singular matrices $A$ are there such that $A^{T}=A^2-I$?

We have $$(A^T - A)(A + I)=0$$

Since $A$ is nonsingular, $\det A \neq 0 \implies \det A^T \neq 0 \implies \det (A^2 - I) \neq 0 \implies \det(A+I)\cdot\det(A-I)\neq 0$ $$\implies \det(A+I) \neq 0 \neq \det(A-I)$$

Therefore, $$(A^T - A)(A + I)=0 \implies A^T - A = 0$$ And we're done

---

Here's something else that I noticed. You arrive at the following equation: $$A^3 - 2A - I=0 \implies A^3 - A = A+I \implies A(A^2 - I) = A+I \implies AA^T = A +I \\ \implies A = AA^T - I$$ Therefore, $A$ is symmetric

Your idea is very good: $A=(A^T)^2-I$, so $$ A=(A^2-I)^2-I $$ which becomes $A=A^4-2A^2+I-I$ and, owing to $A$ nonsingular, $A^3-2A-I=0$. This can be rewritten as $A(A^2-I)-(A+I)=0$ or $(A+I)(A^2-A-I)=0$, hence $(A+I)(A^T-A)=0$.

You have to exclude that $-1$ is an eigenvalue of $A$, so ensuring $A+I$ is invertible, which would lead to $A^T=A$.

If $Av=-v$, with $v\ne0$, then $A^Tv=A^2v-v=0$. This is not possible because $A^T$ is nonsingular. The assumption the matrices are $3\times3$ is not required.

I would like to say why the above dan_fulea's comment is good and Ian's comment (upvoted!) is inappropriate and also why it is assumed that $A$ is invertible. Note also that the hypothesis $n=3$ has nothing to do here. Assume that $A\in M_n(K)$.

Case 1. $K=\mathbb{R}$. Since $AA^T=A^TA$, $A$ is normal and therefore unitarily diagonalizable; then we may assume that $A=diag(\lambda_i),A^T=diag(\overline{\lambda_i})$ where $\overline{\lambda_i}=\lambda_i^2-1$. The previous equation has only real solutions and, consequently, $A=A^T$. Note that the hypothesis "$A$ invertible" is useless.

Case 2. $K=\mathbb{C}$. Note that $A(A+I)(A^2-A-I)=0$ and $A$ is diagonalizable over $\mathbb{R}$.

If $\det(A)\not= 0$, then $A(A^2-I)$ is invertible and consequently $A(A+I)$ too; we deduce easily that $A^2-A-I=0$ and $A=A^T$.

Assume that $\det(A)=0$ and $A$ non-symmetric; then $0,0^2-1=-1\in spectrum(A)$ and $A^2-A-I\not= 0$.

For $n=2$ a solution is $A_2=1/2\begin{pmatrix}-1&-i\\i&-1\end{pmatrix}$ where $spectrum(A_2)=\{0,-1\}$.

For $n=3$, a solution is $A_3=diag(A_2,(1+\sqrt{5})/2)$ where $spectrum(A_3)=\{0,-1,(1+\sqrt{5})/2\}$.