Formal Definition of Dot Product

Your top-line question can be answered at many levels. Setting aside issues of forms and covariant/contravariant, the answer is:

The dot product is the product of the magnitudes of the two vectors, times the cosine of the angle between them.

No matter what basis you compute that in, you have to get the same answer because it's a physical quantity.

The usual "sum of products of orthonormal components" is then a convenient computational approach, but as you've seen it's not the only way to compute them.

The dot product's properties includes linear, commutative, distributive, etc. So when you expand the dot product

$$(a_x \hat{x}+a_y \hat{y} + a_z \hat{z}) \cdot (b_x \hat{X}+b_y \hat{Y} + b_z \hat{Z})$$

you get nine terms like $( a_x b_x \hat{x}\cdot\hat{X}) + (a_x b_y \hat{x}\cdot\hat{Y})+$ etc. In the usual orthonormal basis, the same-axis $\hat{x}\cdot\hat{X}$ factors just become 1, while the different-axis $\hat{x}\cdot\hat{Y}$ et al factors are zero. That reduces to the formula you know.

In a non-orthonormal basis, you have to figure out what those basis products are. To do that, you refer back to the definition: The product of the size of each, times the cosine of the angle between. Once you have all of those, you're again all set to compute. It just looks a bit more complicated...

Dot products, or inner products are defined axiomatically, or abstractly. An inner product on a vector space $V$ over $R$ is a pairing $V\times V\to R$, denoted by $ \langle u,v\rangle$, with properties $\langle u,v\rangle=\langle v,u\rangle$, $\langle u+cw,v\rangle= \langle u,v\rangle+c\langle w,v\rangle$, and $ \langle u,u\rangle\gt0$ if $u\ne0$. In general, a vector space can be endowed with an inner product in many ways. Notice here there is no reference to a basis/coordinate system.

Using what is called the Gram-Schmidt process, one can then construct a basis $\{e_1,\cdots e_n\}$ for $V$ in which the inner product takes the computational form which you stated in your question.

In your question, you are actually starting with what is called an orthonormal basis for an inner product. The coordinate-free approach is to state the postulates that an inner product should obey, then after being given an explicit inner product, construct an orthonormal basis in which to do computations.

In general, an orthonormal basis $\{e_1,e_2,e_3\}$ for one inner product on $V$ will not be an orthonormal basis for another inner product on $V$.

The dot product can be defined in a coordinate-independent way as

$$\vec{a}\cdot\vec{b}=|\vec{a}||\vec{b}|\cos\theta$$

where $\theta$ is the angle between the two vectors. This involves only lengths and angles, not coordinates.

To use your first formula, the coordinates must be in the same basis.

You can convert between bases using a rotation matrix, and the fact that a rotation matrix preserves vector lengths is sufficient to show that it preserves the dot product. This is because

$$\vec{a}\cdot\vec{b}=\frac{1}{2}\left(|\vec{a}+\vec{b}|^2-|\vec{a}|^2-|\vec{b}|^2\right).$$

This formula is another purely-geometric, coordinate-free definition of the dot product.