Finding the similarity transformation between two matrices

Your specific example can be solve with a general $P$, see code below.

(*Data*)
A = {{-2, -2, 1}, {2, x, -2}, {0, 0, -2}};
B = {{2, 1, 0}, {0, -1, 0}, {0, 0, y}};

(*Search for x and y based on characteristic polynomial*)
n = Length@A;
Id = IdentityMatrix@n;
solxy = SolveAlways[Det[A - l*Id] == Det[B - l*Id], l]

(*Update data*)
A = A /. solxy[[1]];
B = B /. solxy[[1]];

(*Solve for general P*)
P = Array[p, {n, n}];
solP = Solve[P.B == A.P, Flatten@P];
P = P /. solP[[1]]

(*Check*)
B == [email protected] // Simplify

You can then enter some values for the free components of $P$.

The purely linear algebraic way to do this is to reduce both matrices to Jordan form:

{sa, ja} = JordanDecomposition[{{-2, -2, 1}, {2, x, -2}, {0, 0, -2}}];
{sb, jb} = JordanDecomposition[{{2, 1, 0}, {0, -1, 0}, {0, 0, y}}];

Inspecting both ja and jb shows that they are both diagonal, so we can proceed:

Diagonal[ja]
   {-2, 1/2 (-2 + x - Sqrt[-12 + 4 x + x^2]), 1/2 (-2 + x + Sqrt[-12 + 4 x + x^2])}

Diagonal[jb]
   {-1, 2, y}

A moment's consideration leads us to letting y == -2. To try finding x, we try equating one of ja's unknown eigenvalues to one of jb's:

Solve[1/2 (-2 + x - Sqrt[-12 + 4 x + x^2]) == -1, x]
   {{x -> 3}}

(Exercise: look at what happens if you take all corresponding pairs of eigenvalues between ja and jb, and equate them.)

Thus, we can assemble the similarity transformation as:

pa = (sa /. x -> 3).Inverse[sb[[All, {3, 1, 2}]]]
   {{-1/2, -13/6, -1/4}, {1, 4/3, 1/2}, {0, 0, 1}}

(Exercise: derive the expression I used for the similarity transformation)

Check:

Inverse[pa].({{-2, -2, 1}, {2, x, -2}, {0, 0, -2}} /. x -> 3).pa -
({{2, 1, 0}, {0, -1, 0}, {0, 0, y}} /. y -> -2)
   {{0, 0, 0}, {0, 0, 0}, {0, 0, 0}}