How to define the magnitude of a rotation in $\mathbb{R}^n$?
Every $n\times n$ rotation matrix is diagonalizable over $\mathbb C$, yielding $n$ complex eigenvalues.
In $\mathbb R^2$ the two eigenvalues are $e^{i\theta}$ and $e^{-i\theta}$. Take logs and you obtain $\pm i\theta$, which corresponds to the magnitude of the rotation being $\theta$.
In $\mathbb R^3$ the three eigenvalues are $e^{i\theta}$, $e^{-i\theta}$, and $1$. Take logs and you obtain $\pm i\theta$ and $0$, so again it makes sense to define the magnitude of the rotation as $\theta$.
In $\mathbb R^n$ there are $\lfloor n/2\rfloor$ pairs of conjugate eigenvalues, and possibly a $1$ if $n$ is odd. So you have $k=\lfloor n/2\rfloor$ different rotation angles, and it is up to you to combine them. You could take $\sqrt{\theta_1^2+\cdots+\theta_k^2}$ or $|\theta_1|+\cdots+|\theta_k|$ or $\max(\theta_1,\ldots,\theta_n)$ or something similar. Any of these would be consistent with the well-known low-dimensional cases, as long as it is symmetric in the $\theta_i$ and reduces to $\theta_1$ when $\theta_2=\cdots=\theta_k=0$.
That said, the $\|R-I\|$ idea you had can also be a reasonable choice. In the low-dimensional cases, it has a simple relationship with rotation angle.
In $\mathbb R^2$, if $R$ is a rotation by $\theta$, then $\|R-I\|$ in the Frobenius norm is $2\sqrt2|\sin(\theta/2)|$.
In $\mathbb R^3$, we can pick a basis so that the $z$-axis lies along the axis of rotation, so that $R$ becomes a block diagonal matrix consisting of a $2\times2$ rotation by $\theta$ and a $1\times1$ identity matrix. Since the Frobenius norm is invariant to orthogonal changes of basis, again $\|R-I\|_F = 2\sqrt2|\sin(\theta/2)|$.
In $\mathbb R^n$, if $R$ rotates a single $2$-dimensional subspace by $\theta$ and leaves the orthogonal subspace unchanged, we still have $\|R-I\|_F = 2\sqrt2|\sin(\theta/2)|$. In all the cases above, we can recover the angle of rotation as $\theta = 2\arcsin\big(\|R-I\|_F/2\sqrt2\big)$. Unfortunately, for a general rotation matrix, $\|R-I\|_F$ can be greater than $2\sqrt2$, so we can't recover a generalized rotation magnitude consistent with the low-dimensional cases by the above formula. Nevertheless, we can just work with $\|R-I\|_F$.
P.S. I think in general we have $\|R-I\|_F=2\sqrt2\sqrt{\sin^2(\theta_1/2) + \cdots + \sin^2(\theta_k/2)}$, but I'm not positive.
P.P.S. The overall idea is analogous to the fact that the geodesic distance between two points $x$ and $y$ on a unit sphere in $\mathbb R^n$ can be easily computed as $2\arcsin(\|x-y\|/2)$.
Here's a complete answer to your question for the case of $\mathbb{R}^3$.
In $\mathbb{R}^3$ there are actually infinitely many lines to rotate around. You don't have to restrict yourself to rotations around the coordinate axes. You can rotate around the line $\{(t,t,t) \,\, | \,\, t \in \mathbb{R}\}$, for example. The rotation axes of all the planets in the solar system do not line up nicely with the coordinate axes of any particular coordinate system, for another example.
One can prove that for any rigid motion of $\mathbb{R}^3$ which preserves orientation, that rigid motion is either the identity map or is a rotation around a well-defined axis by an angle with a well-defined absolute value in $[0,\pi]$. So in this situation, the absolute value of the rotation angle serves as a good way to quantify the rotation.
(This is just an idea.)
It seems like the natural way to define the magnitude of a rotation in $\mathbb{R}^n$ would be by measuring the angle at which it displaces a given unit vector. If $A$ is the rotation and $e$ is a unit vector, then the angle $\theta$ should satisfy $$ \cos \theta = (Ae) \cdot (e) $$ As pointed out in the comments, the actual angle $\theta$ depends on the unit vector we pick. Therefore, perhaps a good definition of magnitude would be: $$ \theta_{\text{max}} := \sup_{e \in \mathbb{R}^n \\ \| e\| = 1} \Big[ \arccos \big( (Ae) \cdot e \big) \Big]. $$