Is a bijective, norm-preserving and rotation-invariant mapping linear?
Here is an initial step: Every such map must be a conformal diffeomorphism.
To see it consider a point $z\in S^N$ and let $R_z$ be a rotation taking $h(z)$ to $z$. We look at $\tilde h:= R_z\circ h$ to note that $$(\tilde h(Rx) , \tilde h(Ry) ) =(R_z h(Rx), R_z h(Ry)) = (h(x), h(y)) = (R_z h(x), R_z h(y))$$ and see that this enjoys the same property as $h$ with the added benefit that $z$ is fixed point of $\tilde h$. Now let $x(t)$ and $y(s)$ be two differentiable paths with $x(0)=y(0)=z$. Then for an arbitrary rotation $R$ with $R(z)=z$ one has: $$( D_z\tilde h ( R x'(0)) , D_z\tilde h(R y'(0) ) )= \frac{d}{dt}\frac d{ds}(\tilde h(R x(t)), \tilde h(Ry(t) )\lvert_{s,t=0}= \frac{d}{dt}\frac d{ds}(\tilde h( x(t)), \tilde h(y(t) )\lvert_{s,t=0}\\ =( D_z\tilde h ( x'(0)) , D_z\tilde h( y'(0) ) ).$$
Since this then holds for all tangent vectors $x'(0), y'(0)$ one has that $R^T(D_z\tilde h)^T D_z\tilde h R = (D_z\tilde h)^T D_z \tilde h$ for any rotation preserving the point $z$. This implies that $(D_z\tilde h)^T D_z\tilde h$ is a scalar multiple of the identity on $T_zS^N$, ie that $D_z\tilde h = \lambda S$ for some $\lambda\in \Bbb R$ and a rotation $S$ of $T_zS^N$.
This means that $D_z\tilde h$ is a conformal mapping $T_zS^N\to T_zS^N$. Since $R_z$ is an isometry one has that $D_zh = D_z(R_z^{-1}\tilde h)$ is then also a conformal mapping. Since $z$ was arbitrary $h$ is a conformal map.
However it is not true that every conformal diffeomorphism of the sphere satisfies the desired equation, it is easy to construct counter examples on $S^2$ via Möbius transformations. This leads me to believe that only the isometries, ie the rotations, have this property.
Here is an attempt for a formal proof using Theorem 1 in [1] which basically states that $h$ is linear if $h(x)^\top h(y) = x^\top y$. We proof by contradiction that $h$ has this property. Let $\alpha(x,y) = \cos^{-1}(x^\top y)$ be the angular distance between $x$ and $y$. The proof has four main steps:
If $\exists x, y\,\,\alpha(h(x), h(y)) > \alpha(x, y)$, then there exists a $z$ on the geodesic between $x$ and $y$ such that $\alpha(h(x), h(z)) > \alpha(x, z) < \epsilon$ for a given $\epsilon > 0$.
If $\exists x, y\,\,\alpha(h(x), h(y)) > \alpha(x, y)$ then there exists an $h'$ fulfilling all desired properties (1-3) such that $\alpha(h'(x), h'(y)) < \alpha(x, y)$ (and vice versa).
If $\exists x, y\,\,\alpha(h(x), h(y)) < \alpha(x, y)$ then $\alpha(h(x), h(z)) < \alpha(x, z)$ for all $z$ with $\alpha(x, y) = \alpha(x, z)$.
If $\alpha(h(x), h(y)) < \alpha(x, y) < \epsilon$ for any $\epsilon > 0$ then $h$ cannot be bijective, contradicting the first assumption on $h$.
This concludes the proof. I now prove each part individually:
Proof of part 1
Assume there exists $x, y$ s.t. $\alpha(h(x), h(y)) > \alpha(x, y)$. Now consider a point $z$ and a rotation $R$ such that $z = Rx$ and $y = Rz$ (i.e. $z$ is a midpoint of $x$ and $y$ on the hypersphere). Then
\begin{align} 2\alpha(x, z) &= \alpha(x, y), \\ &< \alpha(h(x), h(y)), \\ &\leq \alpha(h(x), h(z)) + \alpha(h(z), h(y)) , \\ &= \alpha(h(x), h(z)) + \alpha(h(Rx), h(Rz)), \\ &= 2\alpha(h(x), h(z)) \end{align}
where I used that the angular distance is a metric (and thus I can use the triangle inequality). Hence, $\alpha(x, z) < \alpha(h(x), h(z))$. We now just repeat this argument several times (each time halving the angle between $x$ and $z$ until $\alpha(x, z) < \epsilon$).
Proof of part 2
For any given $h$ fulfilling assumptions (1-3), its inverse $h^{-1}$ also fulfills (1-3). If there exists $x, y$ such that $\alpha(h(x), h(y)) > \alpha(x, y)$, then $\alpha(h^{-1}(x'), h^{-1}(y')) < \alpha(x', y')$ for $x' = h(x)$ and $y' = h(y)$.
Proof of part 3
For any $x^\top y = x^\top z$ we can find a rotation $R$ such that $x = Rx$ (i.e. $x$ is a fixed point of $R$) and $y = Rz$. Then if $\alpha(h(x), h(y)) < \alpha(x, y)$ we find that
$$\alpha(h(x), h(z)) = \alpha(h(Rx), h(Rz)) = \alpha(h(x), h(y)) < \alpha(x, y).$$
Proof of part 4
Given part 1., 2. and 3. we know that if there exists any $h$ for which there exists $x, y$ such that $h(x)^\top h(y) \ne x^\top y$, there must be a $\tilde h$ s.t. $\alpha(\tilde h(x), \tilde h(y)) < \alpha(x, y) < \epsilon$ for all $z$ with $x^\top y = x^\top z$. Choose any $x', y'$ such that $\tilde h(x') = -\tilde h(y')$ (which must exist because $\tilde h$ is bijective). Let further $z'$ be on the geodesic between $x'$ and $y'$ such that $\alpha(\tilde h(x'), \tilde h(z')) < \alpha(x', z') < \epsilon$ and let $N$ be such that $N\alpha(x', z') = \alpha(x', y')$ (this can be made precise in the limit $\epsilon\to 0$ and $N\to\infty$). Then,
\begin{align} \pi &= \alpha(\tilde h(x'), -\tilde h(x')), \\ &= \alpha(\tilde h(x'), \tilde h(y')), \\ &\leq N \alpha(\tilde h(x'), \tilde h(z')), \\ &< N\alpha(x', z'), \\ &= \alpha(x', y'), \\ &\leq \pi. \end{align}
This contradiction completes the proof.
[1] Linear mappings approximately preserving orthogonality, https://reader.elsevier.com/reader/sd/pii/S0022247X04007474?token=8EDC87750F4B5D379A98264CBF42E0CC2F4B73DFF6217BE5CB03B27A4ABD495E189DF508816E4E3DCC3E9ACEC5DA5B73