Proving that the angular velocity vector is equal to a limit involving the rotation vector

If you take $t_0 = t_1$ in the EDIT part of your question, with $\vec{\theta}(t_0,t_1) = \vec{\theta}(t_1,t_1) = 0$, you are in the special case $\omega = \dot{\alpha}$ from Asher Peres's paper you have mentioned, which then proves your statement according to your observation from the EDIT part, because in that case, you have $$\frac{\vec{\theta}(t_0,t_2)-\vec{\theta}(t_0,t_1)}{t_2-t_1} = \frac{\vec{\theta}(t_1,t_2)}{t_2-t_1},$$ whose limit when $t_2 \to t_1$ is equal to $\dot{\alpha}(t_1) = \omega (t_1)$.

Indeed, according to Asher Peres, we have: $$ w = \dot{\alpha} + \frac{1 - \cos \alpha}{\alpha^2} (\alpha \times \dot{\alpha}) + \frac{\alpha - \sin \alpha}{\alpha^3} (\alpha \times (\alpha \times \dot{\alpha})),$$ which, for $\alpha(t) = \vec{\theta}(t_1,t)$ and $t = t_1$, and using $\alpha(t_1) = \vec{\theta}(t_1,t_1) = 0$ (see above), reduces to $$ w(t_1) = \dot{\alpha}(t_1) + 0.$$ Note that luckily $1 - \cos \alpha = \mathrm{O}(\alpha^2)$ and $\alpha - \sin \alpha = \mathrm{O}(\alpha^3)$, hence there is not problem when passing to the limit when $\alpha \to 0.$


Consider a fixed point with location $\vec{a}$ or a rigid body.

To prove the rotation first establish that $$ {\rm d}\vec{a} = {\rm d} \vec{\theta} \times \vec{a} \tag{1}$$

This can be done with just geometry given that small angle approximations. For example the change in the x-direction is ${\rm d}a_x =a_z {\rm d} \theta_y - a_y {\rm d}\theta_z $.

The expression can be written as $$\vec{v} =\frac{{\rm d}\vec{a}}{{\rm d}t} = \vec{\omega} \times \vec{a} \tag{2} $$

The last part is to calculate $$\vec{a} \times \vec{v} = \vec{a} \times (\vec{\omega} \times \vec{a}) = \vec{\omega} ( \vec{a} \cdot \vec{a} ) - \vec{a} (\vec{r} \cdot \vec{\omega}) \tag{3}$$

Take the projection of the location perpendicular to the rotation $\vec{r}$ with $\vec{r} \cdot \vec{\omega}=0$ then

$$\require{cancel} \vec{r} \times \vec{v} = \vec{\omega} \| \vec{r} \|^2 - \cancel{\vec{r} \cdot \vec{\omega}} $$ $$ \boxed{\vec{\omega} = \frac{\vec{r} \times \vec{v}}{\| \vec{r} \|^2} } \tag{4}$$


Edit 1

A more vigorous treatment involves creating a 3×3 rotation matrix, and applying small angle approximation to it. Use $\vec{\theta} = (\theta_x,\theta_y,\theta_z)$ as successive rotations

$$\mathtt{R}=\mathtt{R}_x(\theta_x)\mathtt{R}_y(\theta_y)\mathtt{R}_z(\theta_z) =\\ \begin{vmatrix} \cos\theta_y \cos\theta_z & -\cos\theta_y \sin\theta_z & \sin\theta_y\\ \cos \theta_x \sin\theta_z + \sin\theta_x \sin\theta_y \cos\theta_z & \cos\theta_x \cos\theta_z - \sin\theta_x\sin\theta_y\sin\theta_z &-\sin\theta_x \cos\theta_y \\ \sin \theta_x \sin\theta_z - \cos\theta_x \sin\theta_y \cos\theta_z & \sin\theta_x \cos\theta_z + \cos\theta_x\sin\theta_y\sin\theta_z &\cos\theta_x \cos\theta_y \end{vmatrix} $$

All this now applied to a small angle to make ${\rm d}\vec{a} =( {\rm d}\mathtt{R})\vec{a} -\vec{a}$ such that $\sin(\square)=\square$ and $\cos(\square)=1$

$${\rm d}\mathtt{R}= \mathtt{R}_x({\rm d}\theta_x)\mathtt{R}_y({\rm d}\theta_y)\mathtt{R}_z({\rm d}\theta_z) =\\ \begin{vmatrix} 1 &-{\rm d}\theta_z & {\rm d}\theta_y \\ {\rm d}\theta_z + {\rm d}\theta_x {\rm d}\theta_y & 1 - {\rm d}\theta_x {\rm d} \theta_y {\rm d} \theta_z & -{\rm d}\theta_x \\ -{\rm d}\theta_y+{\rm d}\theta_x {\rm d}\theta_z & {\rm d}\theta_x + {\rm d}\theta_y {\rm d}\theta_z & 1\end{vmatrix} \\ =\begin{vmatrix} 1 &-{\rm d}\theta_z & {\rm d}\theta_y \\ {\rm d}\theta_z & 1 & -{\rm d}\theta_x \\ -{\rm d}\theta_y & {\rm d}\theta_x & 1\end{vmatrix} \tag{6} $$

So with small angle approximation $${\rm d}\vec{a} = ( {\rm d} \mathtt{R})\vec{a} -\vec{a} = \left({\rm d}\mathtt{R} - \mathtt{1}\right) \vec{a} ={\rm d} \vec{\theta} \times \vec{a}$$

$$ [{\rm d} \vec{\theta} \times] = \begin{vmatrix} 0 &-{\rm d}\theta_z & {\rm d}\theta_y \\ {\rm d}\theta_z & 0 & -{\rm d}\theta_x \\ -{\rm d}\theta_y & {\rm d}\theta_x & 0\end{vmatrix}$$

$$ \frac{{\rm d} \vec{\theta} }{{\rm d}t} \times = \begin{vmatrix} 0 & -\omega_z & \omega_y \\\omega_z &0&-\omega_x\\ -\omega_y&\omega_x&0\end{vmatrix}$$

The last 3×3 matrix is called the vector cross product operator matrix. It is skew symmetric and it is used widely in computer graphics and in dynamics.