Determinant of the derivative of a map between manifolds
Both definitions you suggest do not compile:
- The volume form $\omega$ on $M$ eats $m$ tangent vectors at each point. How do you feed it with only $n$ vectors?
- The matrix $D\Phi(\varphi^{-1}(x))$ is an $n \times m$ matrix. How do you define the determinant of a non-block matrix?
In order to clarify things let me discuss first the linear algebra relevant for your question. The main point is that there is no notion of determinant for a linear map between two vector spaces of different dimensions and there is no notion of determinant for a linear map between two different vector spaces of the same dimension without a choice of an extra data.
If $T \colon V \rightarrow V$ is a linear operator on a finite dimensional vector space (so the domain and codomain are the same), you can define $\det(T)$ by choosing a basis $\mathcal{B}$ for $V$ and defining $\det(T) := \det([T]_{\mathcal{B}})$ where $[T]_{\mathcal{B}} \in M_n(\mathbb{F})$ is the square matrix that represents the operator $T$ with respect to the basis $\mathcal{B}$. This definition uses a basis but is, in fact, independent of the basis we work with as $$\det([T]_{\mathcal{B}'}) = \det(P^{-1} [T]_{\mathcal{B}} P) = \det([T]_{\mathcal{B}}) $$ where $P$ is the change of basis matrix $P =[\operatorname{id}]_{\mathcal{B}}^{\mathcal{B'}}$.
If $T \colon V \rightarrow W$ is a linear map between two vector spaces of the same (finite) dimension, you can try and define $\det(T)$ by representing $T$ as a matrix. However, to represent $T$ as a matrix you need to pick two different bases $\mathcal{B}$ for $V$ and $\mathcal{C}$ for $W$ and if $\mathcal{B}', \mathcal{C}'$ are other bases, we have
$$ [T]_{\mathcal{C}}^{\mathcal{B}} = [\operatorname{id}]^{\mathcal{C}'}_{\mathcal{C}}[T]^{\mathcal{B}'}_{\mathcal{C}'} [\operatorname{id}]^{\mathcal{B}}_{\mathcal{B}'}$$
and so there's no reason that $\det([T]_{\mathcal{C}}^{\mathcal{B}}) = \det([T]^{\mathcal{B}'}_{\mathcal{C}'})$.
To save the situation, we should go back to the case $T \colon V \rightarrow V$ and reinterpret $\det(T)$ differently. If $V = \mathbb{R}^n,$ the scalar $\det(T)$ is the signed factor by which $T$ scales the volume of an $n$-dimensional parallelotope. In an abstract vector space $V$, we have no natural notion of (signed) volume of an $n$-dimensional parallelotope. Such notion is provided by a choice of a volume form $0 \neq \omega_V \in \Lambda^{\text{top}}(V^{*})$. Given such a volume form, we can define $\det(T)$ as the unique scalar such that $T^{*}(\omega_V) = \det(T) \omega_V$ and the nice thing is that this definition is actually independent of the volume form!
Finally, to define a notion of a determinant for a map $T \colon V \rightarrow W$, we equip $V$ and $W$ with volume forms $\omega_V \in \Lambda^{\text{top}}(V^{*})$ and $\omega_W \in \Lambda^{\text{top}}(W^{*})$ and define $\det(T)$ by the equation $T^{*}(\omega_W) = \det(T) \omega_V$. This definition depends on both $\omega_V$ and $\omega_W$.
Now assume $(V, g_V)$ is an inner product space. Even though you provided extra data, there is still no natural volume form that is defined on $V$. In order to define a natural form, you need to choose an orientation for $V$ (and equivalence class $\mathfrak{o}_V$ of elements of $\Lambda^{\operatorname{top}}(V)$ or $\Lambda^{\operatorname{top}}(V^{*})$ depending on your definitions). Once chosen, there is a unique volume form $\omega_{g_V,\mathfrak{o}_V}$ that behaves nicely with the metric and the orientation (this is precisely the Riemannian volume form). It is determined by the fact that if $(e_1, \dots, e_n)$ is a positive orthonormal basis of $V$ then $\omega_{g_V,\mathfrak{o}_V}(e_1 \wedge \dots \wedge e_n) = 1$.
In your situation you have a surjective linear map $T \colon (V, g_V, \mathfrak{o}_V,\omega_V) \rightarrow (W, g_W, \mathfrak{o}_V,\omega_W)$ with $\dim V = m, \dim W = n$. The map $T|_{(\ker T)^{\perp}} \colon (\ker T)^{\perp} \rightarrow W$ is a linear map between two vector spaces of the same dimension and so we can try and make sense of $\det \left( T|_{(\ker T)^{\perp}} \right)$. The right hand side has by assumption a volume form but the left hand side is only a subspace of a space that has a volume form. In general, a subspace of a space with a volume form doesn't get a volume form but $(\ker L)^{\perp}$ has an inner product (the restriction of $g_V$) and we can give it an orientation using the map $T$ and these two structures endow $(\ker T)^{\perp}$ with an orientation and allow us to talk of the determinant.
Finally, we can provide corrected versions of your definitions for $\det T$:
- If $v_1, \dots, v_n$ is a basis of $( \ker T)^{\perp}$ such that $Tv_1, \dots, Tv_n$ is a positive basis of $W$, complete it to a positive basis $v_1, \dots, v_n, u_1, \dots, u_{m-n}$ of $V$ and then $$ \det \left( T|_{(\ker T)^{\perp}} \right) = \frac{\omega_W(Tv_1, \dots, Tv_n)}{\omega_V(v_1, \dots, v_n, u_1, \dots, u_{m-n})}. $$
- If $\mathcal{B} = (v_1, \dots, v_n)$ is an orthonormal basis of $( \ker T)^{\perp}$ such that $Tv_1, \dots, Tv_n$ is a positive basis of $W$ and $\mathcal{C} = (w_1, \dots, w_m)$ is a positive orthonormal basis of $W$ then $\det \left( T|_{(\ker T)^{\perp}} \right) = \det\left(\left[ T|_{(\ker T)^{\perp}} \right]_{\mathcal{C}}^{\mathcal{B}}\right)$.
I'll leave it to you to verify that the definitions are equivalent and consistent with what I described before.