Isogonal operator is the product of an orthogonal map with a homothety

Let me suggest an alternative derivation which, at the end, boils down to what has been suggested. If $A$ is a product of a homothety by a factor $\lambda$ and an orthogonal operator $O$, we can write $A = \lambda O$. Then

$$ A^TA = (\lambda O)^T(\lambda O) = \lambda^2 O^T O = \lambda^2 I. $$

This suggests that given an isogonal operator $A$, by looking at $A^TA$, we should be able to extract the homothety factor $\lambda$ and then the orthogonal matrix $O$ so let us try to do that. Define $B = A^TA$ and let $x, y \in \mathbb{R}^n$ with $x \perp y$. Then $\left< x, y \right> = 0$ and so

$$ \left< Bx, y \right> = \left< A^TAx, y \right> = \left< Ax, Ay \right> = 0 $$

which shows that $Bx \in \left( \operatorname{span} \{ x \}^{\perp} \right)^{\perp} = \operatorname{span} \{ x \}$. This means that each vector $x \in \mathbb{R}^n$ is an eigenvector of $B$ (right now, possibly with a different eigenvalue). Now, if $x_1, x_2$ are linearly independent, write $Bx_i = \lambda_i x_i$. Then

$$ B(x_1 + x_2) = \lambda_1 x_1 + \lambda_2 x_2 = \mu (x_1 + x_2) $$

and since the $x_i$ are linearly independent, we see that we must have $\lambda_1 = \lambda_2 = \mu$ which shows that in fact, $B = \mu I$ for some $\mu \in \mathbb{R}$.

Now, note that $B = A^T A$ is a positive semi-definite operator and so $\mu \geq 0$. Set $\lambda = \sqrt{\mu}$. If $\lambda = 0$ then $A^T A = 0$ and so $A = 0$. If $\lambda > 0$, we can define $O = \frac{A}{\lambda}$ and then

$$ O^T O = \frac{1}{\lambda^2} A^T A = \frac{1}{\mu} \mu I = I$$

showing that $O$ is orthogonal and $A = \lambda O$.

---

Regarding the more general question, let $g \colon \mathbb{R}^n \times \mathbb{R}^n \rightarrow \mathbb{R}$ be a symmetric bilinear form and let $A \colon \mathbb{R}^n \rightarrow \mathbb{R}^n$ be an operator such that if $g(x,y) = 0$ then $g(Ax, Ay) = 0$. Note that if $g$ is indefinite (and in particular $n \geq 2$), we can always find a one-dimensional subspace $V \subseteq \mathbb{R}^n$ such that $g(x,x) = 0$ for all $x \in V$. But then any linear map $A$ that maps $\mathbb{R}^n$ to $V$ will satisfy the property above trivially but it won't be of the form $\lambda O$ for a $g$- isometry $O$ because $\lambda O$ is either $0$ or has full rank.