Why do we use cross products in physics?

This is a great question. The dot and cross products seem very mysterious when they are first introduced to a new student. For example, why does the scalar (dot) product have a cosine in it and the vector (cross) product have a sine, rather than vice versa? And why do these same two very non-obvious ways of "multiplying" vectors together arise in so many different contexts?

The fundamental answer (which unfortunately may not be very accessible if you're a new student) is that there are only two algebraically independent tensors that are invariant under arbitrary rotations in $n$ dimensions (we say that they are "$\mathrm{SO}(n)$ invariant"). These are the Kronecker delta $\delta_{ij}$ and the Levi-Civita symbol $\epsilon_{ijk \cdots}$. Contracting two vectors with these symbols yields the dot and cross products, respectively (the latter only works in three dimensions). Since the laws of physics appear to be isotropic (i.e. rotationally invariant), it makes sense that any physically useful method for combining physical quantities like vectors together should be isotropic as well. The dot and cross products turn out to be the only two possible multilinear options.

(Why multilinear maps are so useful in physics is an even deeper and more fundamental question, but which answers to that question are satisfying is probably inherently a matter of opinion.)

A cross product is highly related to another concept, the exterior product (or wedge product). An exterior product is a very natural product which occurs in algebra. The exterior product of two vectors is a bivector, whose directions are very natural (while torque as a vector is at right angles to the force and the lever arm, in exterior product it's simply a bivector defined by two directions -- the force and the leve arm).

Unfortunately, exterior products are difficult to teach early on. They take a lot of math. Cross products are much easier to explain. And, as it turns out, in 3 dimensions, cross products and exterior products are isometric. They transform in the same ways. If you do the math with cross products, you get the same answer as if you did them with exterior products. This doesn't work in all dimensions (cross products are a 3 dimensional thing, while exterior products can be done in any number of dimensions), but it does work in 3, and lots of physics is done in three dimensions!

I am focusing on the geometry of cross products

Cross products are used when we are interested in the moment arm of a quantity. That is the minimum distance of a point to a line in space.

  1. The Distance to a Ray from Origin. A ray along the unit vector $\boldsymbol{e}$ passes through a point $\boldsymbol{r}$ in space.

    $$ d = \| \boldsymbol{r} \times \boldsymbol{e} || \tag{1}$$

    $d$ is the perpendicular distance to the ray (also known as the moment arm of the line).

  2. The moment arm of Force (Torque Vector). A force $\boldsymbol{F}$ along $\boldsymbol{e}$ causes the following torque about the origin

    $$ \boldsymbol{\tau} = \boldsymbol{r} \times \boldsymbol{F}\;\; \rightarrow \| \boldsymbol{\tau} \| = d\, \| \boldsymbol{F} \| \tag{2}$$

  3. The moment arm of Rotation (Velocity Vector). A rotation $\boldsymbol{\omega}$ about the axis $\boldsymbol{e}$ causes the a body to move at the origin location by

    $$ \boldsymbol{v} = \boldsymbol{r} \times \boldsymbol{\omega}\;\; \rightarrow \| \boldsymbol{v} \| = d\, \| \boldsymbol{\omega} || \tag{3}$$

  4. The moment arm of Momentum (Angular Momentum). A classical particle with momentum $\boldsymbol{p}$ along $\boldsymbol{e}$ has angular momentum about the origin

    $$ \boldsymbol{L} = \boldsymbol{r} \times \boldsymbol{p} \;\; \rightarrow \| \boldsymbol{L} \| = d\, \| \boldsymbol{p} \| \tag{4}$$