How to compute the similarity transformation matrix

In order to find your $P$, you can do as follows:

• First, you find a diagonal matrix $D$ to which both $A$ and $B$ are equivalent. For this, you need to find the eigenvalues of both matrices and if they coincide, they are equivalent. (The general case would be more involved: but in yours, both matrices diagonalize easily.)
• Then you have to find bases of eigenvectors for both matrices and form with them change of bases matrices $S$ and $T$ such that

$$ D = S^{-1}A S \qquad \text{and} \qquad D = T^{-1}BT \ . $$

• Now you'll have

$$ S^{-1}A S = T^{-1}BT \qquad \text{and hence} \qquad AST^{-1} = ST^{-1}B \ . $$

So $ST^{-1}$ will be your matrix $P$.