Let $n \geq 1$ be an odd integer. Show that $D_{2n}\cong \mathbb{Z}_2 \times D_n$.
Let's look at your map--let's call it $\phi$. I claim that it fails to be surjective. Suppose it is surjective, so in particular, $\phi(R^jM^k)=(1,e)$ for some natural $j,k$. Now, $$\phi(R^j)=\phi(R)^j=\bigl(0,r^{\frac{n+1}2}\bigr)^j=\bigl(j\cdot 0,r^{j\cdot\frac{n+1}2}\bigr)=\bigl(0,r^{j\cdot\frac{n+1}2}\bigr)$$ and $$\phi(M^k)=\phi(M)^k=(1,m)^k=(k\cdot 1,m^k)=(k,m^k),$$ so $$\begin{align}(1,e) &= \phi(R^jM^k)\\ &= \phi(R^j)\phi(M^k)\\ &= \bigl(0,r^{j\cdot\frac{n+1}2}\bigr)(k,m^k)\\ &= \bigl(0+k,r^{j\cdot\frac{n+1}2}m^k\bigr)\\ &= \bigl(k,r^{j\cdot\frac{n+1}2}m^k\bigr).\end{align}$$ Then $k$ must be odd (since $1=k$ modulo $2$), but if $k$ is odd, then we can't have $r^{j\cdot\frac{n+1}2}m^k=e$, so we have a contradiction.
Finding an isomorphism is (in many cases) quite a bit out of our way, but let's go ahead and do it here anyway. Whatever we map $R$ to is going to have to have order $2n$ for the map to be injective. (Why?) A convenient choice is $(1,r),$ because $$(0,e)=(1,r)^k=(k,r^k)$$ implies that $k$ is even (since $k=0$ modulo $2$) and that $n$ divides $k$ (since $r^k=e$ and $r$ has order $n$), whence $k$ is an integer multiple of $2n$ (since $n$ is odd). Whatever we map $M$ to can't be in the subgroup of $\Bbb Z_2\times D_n$ generated by $(1,r)$ (why not?). An easy choice then is $(1,m).$ Now, let's prove surjectivity by showing that every member of a generating set of $\Bbb Z_2\times D_n$ is mapped to by the homomorphism $\Phi:D_{2n}\to\Bbb Z_2\times D_n$ induced by $R\mapsto(1,r),M\mapsto(1,m).$ (Why is this enough?) A nice generating set of $\Bbb Z_2\times D_n$ is $$\bigl\{(1,e),(0,r),(0,m)\bigr\},$$ and it can be determined that $$\Phi(R^n)=(1,e),\;\Phi(R^{n+1})=(0,r),\text{ and }\Phi(R^nM)=(0,m).$$ Since we're dealing with finite groups of the same order, then surjectivity is the same as injectivity, so we're done.
The approach above was fairly tedious, and we'd generally like to avoid using it. Instead, consider the following
Theorem: Let $G,H,K$ be (multiplicative) groups. Then $G\cong H\times K$ if and only if $G$ has normal subgroups $H',K'$ such that $H'\cong H,K'\cong K,H'\cap K'=\{\text{id}_G\},$ and $$G=H'K':=\{hk:h\in H',k\in K'\}.$$
Proving the above result is a nice exercise (let me know if you want any hints on how to prove the backward direction). It then yields the following
Corollary: Let $G,H,K$ be finite (multiplicative) groups. Then $G\cong H\times K$ if and only if $G$ has normal subgroups $H',K'$ such that $H'\cong H,K'\cong K,\text{ and }H'\cap K'=\{\text{id}_G\}.$
By the Corollary, we need only find normal subgroups of $D_{2n}$ isomorphic to $\Bbb Z_2$ and $D_n$, whose intersection is precisely $\{E\}.$ The subgroup generated by $R^2$ and $M$ should do the trick as a subgroup isomorphic to $D_n$ (automatically normal, since it has index $2$), and since $R^n$ has order $2$ and commutes with everything, then the subgroup it generates is normal and isomorphic to $\Bbb Z_2$.
Edit: Sadly, the Corollary is false in the form shown above! (Credit to Geralt of Rivia for pointing out the issue.) Another condition is needed. If we add the hypothesis that $\|H\|\cdot\|K\|=\|G\|,$ that gets us the rest of the way.
As far as the specific example goes, since we readily have $\left\|D_{2n}\right\|=2\cdot\left\|D_n\right\|,$ then the adjusted Corollary gets us there.
Although I don't exactly understand your attempt, you seem to be mapping all rotations to elements of the form $(0,y)$, and all reflections to elements of the form $(1,y)$. This cannot work, since the subgroup of rotations in $D_{2n}$ is not isomorphic to $D_n$ (for instance because it is commutative). One indication of how to define the map is that $D_{2n}$ has a single non-identity element in its centre, namely the half-turn rotation, which is also minus the identity $-I$ (a central symmetry). This element has to map to $(1,0)$, the unique non-identity element in the centre of $\Bbb Z_2\times D_n$.
Here is a geometric way of defining a map $\Bbb Z_2\times D_n\to D_{2n}$. Draw a regular $n$-gon inside a regular $2n$-gon by alternatingly using and not using vertices as one goes around the $2n$-gon. Now every symmetry of the $n$-gon is also a symmetry of the $2n$-gon, and this defines an injective group morphism $f_1:D_n\to D_{2n}$. An injective group morphism $\Bbb Z_2\to D_{2n}$ is given by sending the non-identity element to $-I$ (so the map is $f_2:\overline m\mapsto (-I)^m$). The image of $f_2$ is central in $D_{2n}$, so it commutes with the image $D_n$ of $f_1$, and the two morphisms combine to a morphism $f:\Bbb Z_2\times D_n\cong D_{2n}$ (formally, $f(x,y)=f_1(x)f_2(y)$). Since $-I$ is not in $D_n$, the kernel of $f$ consists of just the identity and $f$ is injective. Both groups have the sane cardinality $4n$, so $f$ is in fact an isomorphism (surjectivity is also easy to check directly: if an element of $D_{2n}$ is not a symmetry of the $n$-gon, then it interchanges the chosen and non-chosen vertices, and composing it with $-I$ then gives an element of $D_n$).
The isomorphism $D_{2n}\cong\Bbb Z_2\times D_n$ you were looking for is of course $f^{-1}$. From the above, one can gather the following direct description of $f^{-1}$. Given an element $g\in D_{2n}$ the first component of $f^{-1}(g)$ tells whether ($\overline0$) or not ($\overline1$) $g$ preserves the $n$-gon inscribed in the $2n$-gon. The second component of $f^{-1}(g)$ gives a symmetry of the (chosen) inscribed $n$-gon, defined depending on the first component: when $\overline0$, the second component is just the restriction of the action of$~g$, and when $\overline1$ it is the restriction of the composition of $g$ with the central symmetry$~{-}I$ (which composition preserved the inscribed $n$-gon in this case).