Epitrochoids and adjacent loop touching
There will be two solutions of $a$ for a given $\alpha$, although one is simply the negative of the other. For now, I will assume that $a < 0$. Because the epitrochoid is symmetric about the $x$-axis, there will be a tangent point at $y=0$ with $x > 0$.
$$\sin(t) - a \sin(\alpha t) = 0$$ must be true there. In the interval $0 \le t < 2\pi$, we want exactly $6$ solutions of $t$ for $\alpha \ge 4$: $1$ is for $t = 0$, $2$ are for the first loop and the last loop. If $\alpha$ is odd, the last $3$ are for the two middle loops on the left side touching the $x$-axis and the "outwards left" loop crossing the axis. If $\alpha$ is even, the last $3$ are for the middle inner loop on the left side of the $y$-axis crossing the axis and then looping back around twice (one point counts for two since there are different values of $t$ that could produce this point).
What this means in terms of the math is that the first relative minimum of $\sin(t) - a \sin(\alpha t)$ for $t > 0$ must have a value of $0$.
In other words, this point in the picture must be on the $x$-axis. Using the derivative of $\sin(t) - a\sin(\alpha t)$, I find that $$\cos(t) = a \alpha \cos(\alpha t)$$ must be true at the extrema. Then, if I take the second solution of that for $t > 0$, and then plug that into $\sin(t) - a \sin(\alpha t)$, the result should be $0$. Since it is known that $\cos(t) = a \alpha \cos(\alpha t)$, I did a bunch of algebraic and trig manipulation to get that $$a\cos\left(\alpha\arccos\left(\alpha\sqrt{\frac{a^{2}-1}{1-\alpha^{2}}}\right)\right)-\sqrt{\frac{a^{2}-1}{1-\alpha^{2}}} = 0 \tag 1$$ must be true for the loops to be tangent with $-1 \le a < 0$. This will have multiple solutions in $a$ for higher values of $\alpha$, but seems that the right root is the solution that is second-closest to $0$ (although I don't have a proof for this). Since $\cos(\alpha \arccos(x))$ can be rewritten as a polynomial in $x$, $a$ will be the root of a polynomial in $\alpha$, and as such, $a$ will be an algebraic number.
Using this, I found $a$ for small values of $\alpha$. For $\alpha = 3, a = \pm 1$. For $\alpha = 4, a = \pm \frac{3\sqrt{\frac{3}{2}}}{4}$. For $\alpha = 5, a = \pm \frac{4}{5}$. For $\alpha = 6, a = \pm \frac{\sqrt{102-7\sqrt{21}}}{12}$. For $\alpha = 7, a = \pm \frac{-1+2\sqrt{7}}{7}$. For higher values of $\alpha$, the expression for $a$ becomes increasingly complex (or might not even exist in closed form).
Edit: As $\alpha$ goes to $\infty$, $a\alpha$ approaches the root of $1 + x\cos(\sqrt{-1+x^2})$ around $x = 4.6$. This can be seen by plugging in $a = \frac{c}{\alpha}$, and then finding for what value of $c$ the LHS side of $(1)$ multiplied by $\alpha$ will go to $0$ as $\alpha \to \infty$. More specifically, I found the value of $c$ such that $$\lim_{\alpha \to \infty} \left( -c \cos\left( \alpha \arccos\left( \sqrt{\frac{c^2 - \alpha^2}{1-\alpha^2}} \right) \right) - \sqrt{\frac{c^2 - \alpha^2}{1-\alpha^2}} \right) = -1-c \cos(\sqrt{-1+c^2}) = 0$$
I'll replace OP's $\alpha$ with $n$ (to avoid visual confusion with $a$). Also, I'll use a version of the curve such that each "half-leaf" is traced by $t$ in the range of consecutive integer multiples of $\pi/(n-1)$.
$$P = (\;\cos t + a \cos n t \;,\; \sin t + a \sin n t\;) \tag{1}$$
With this version of the curve, we observe that, for $a>0$, when adjacent loops are tangent, the point of tangency lies on the $x$-axis, and the tangent vector at that point is horizontal. (When $a<0$, the curve ---and its corresponding tangent vectors--- are rotated by an angle of $\pi/(n-1)$. In the analysis below, $a$ will ultimately appear to even powers in the key equations, so we'll get negative $a$-values "for free" if we just focus on positive ones. (That said, there's a way to re-frame the argument to incorporate non-horizontal tangents from the get-go. See the Addendum.)) Thus, a necessary condition for tangency is that the $y$-components of $P$ and $\frac{d}{dt}P$ simultaneously vanish: $$\sin t + a \sin n t = 0 = \cos t + na \cos n t \tag{2}$$ for some $\dfrac{\pi}{n-1}\leq t < \dfrac{2\pi}{n-1}$. Also, we observe that $|a|>1$ causes loops to surround the origin, precluding tangency; so we may assume $|a|\leq 1$.
Our goal is to eliminate $t$ from the system in $(2)$. One route ventures through the complex realm. Recall the expressions for cosine and sine in terms of the complex exponential: $$\cos\theta = \frac12\left(e^{i\theta}+e^{-i\theta}\right)\qquad \sin\theta = \frac1{2i}\left(e^{i\theta}-e^{-i\theta}\right) \tag{3}$$ Defining $\omega = \exp(it)$ and $\omega_n = \exp(int)$, our system becomes $$\begin{align} \omega_n(\omega^2 - 1) + \phantom{n}a \omega(\omega_n^2-1) &= 0 \\ \omega_n(\omega^2+1)+ na\omega(\omega_n^2+1) &= 0 \end{align} \tag{4}$$ which has four solutions of the form $$\omega = \frac{p+q}{\sqrt{n^2-1}} \qquad \omega_n = -\frac{p+nq}{na\sqrt{n^2-1}} \qquad\qquad \begin{array}{l} p^2 := n^2 (1-a^2) \geq 0 \\ q^2 := 1-n^2a^2\;\; \text{(possibly $<0$)} \end{array} \tag{5}$$ (taking all combinations of signs on $p$ and $q$). Since $\omega^n=\omega_n$, we can write
$$na\left(p+q\right)^n = -(p+nq)(n^2-1)^{(n-1)/2} \tag{$\star$}$$
So, for a given $n$, candidate values of $a$ that cause tangent loops satisfy the four sign variants of $(\star)$.
Interestingly, $p^2-q^2=n^2-1$ and $p^2-n^2q^2 = n^2a^2(n^2-1)$, so that can square the equation (potentially introducing extraneous $a$ candidates) and rewrite it a bit more symmetrically as
$$\left(\frac{p+q}{p-q}\right)^{n} = -\frac{p+nq}{p-nq} \tag{6}$$
In this form, we see that the simultaneous substitutions $p\to-p$ and $q\to-q$ preserves the relation, so there are really only two sign combinations. Moreover, the substitution $p\to-p$ effectively reciprocates and swaps the signs of the fractions on each side of the equation.
Consequently, we can handle the sole sign ambiguity by squaring:
$$\left(\frac{p+q}{p-q}\right)^{2n} = \left(\frac{p+nq}{p-nq} \right)^2 \tag{$\star\star$}$$
We see immediately that $a=\pm1$ and $a=\pm 1/n$ satisfy $(\star\star)$ by making either $p$ or $q$ vanish. Note that the latter case corresponds to epitrochoids with tangencies, but not quite the tangencies we seek:
So, perhaps $a=\pm1/n$ (ie, $q=0$) should be considered extraneous. Moreover, $a=\pm1$ only "works" for $n=3$.
Cross-multiplying $(\star\star)$ and expanding via the Binomial Theorem, we get some even-odd term cancellation. The result is something we can express in a convenient, radical-free (except for the factor $pq$) form:
$$p q \;\sum_{h=0}^{n} \left(\binom{2n}{2h}2n-\binom{2n}{2h+1}(r^2+n^2)\right)r^{2h} = 0 \qquad\qquad r^2 := \frac{p^2}{q^2} \tag{$\star\star\star$}$$
Therefore, for a given $n$, the the values of $a$ corresponding to self-tangent epitrochoids must satisfy $(\star\star\star)$. Examples are provided below.
The reader will observe that $a$ occurs to even powers, so both positive and negative values satisfy $(\star\star\star)$. One of these values yields a curve with a horizontal tangency on the $x$-axis; the other corresponds to a "twin" curve, identical to the first but rotated by $\pi/(n-1)$, so the same loop tangencies are present.
Also, the algebra does recognize "non/adjacent" leaves in the curve. For example, $n=6$ case shows two self-tangent epitrochoids, but the latter's tangencies skip a leaf.
Interestingly, while extraneous (real) values of $a$ don't correspond to curves with self-tangencies, the examples suggest that "twin pairs" are tangent (with possible leaf-skipping). (See the Addendum.)
$n=3: pq( 27 a^4 - 18 a^2 - 1 ) = 0$
$a = \pm 1$ (same curve):
Twins:
$n=4: pq( 32 a^2 - 27 ) (2048 a^4 - 896 a^2 -27 ) = 0$
$a = \pm \sqrt{27/32}$:
Twins:
$n=5: pq( 25 a^2 - 16 ) (3125 a^6 - 3750 a^4+ 825 a^2 + 16)= 0$
$a=\pm 4/5$:
Twins:
$n=6: pq (6912 a^4 - 9792 a^2 + 3125) (1492992 a^6 - 1430784 a^4 + 234792 a^2 + 3125) = 0$
$a = \pm 0.6968\ldots$, and $a = \pm 0.9649\ldots$ (self-tangent, but non-adjacent leaves):
Twins:
Addendum. Why does $(\star\star\star)$ cover more tangent scenarios than originally intended? Symmetry!
For a given $n$, the epitrochoid has $(n-1)$-fold rotational symmetry. If the curve is to exhibit leaf-to-leaf tangency (with itself or its rotated "twin"), then the tangent vectors at the points-of-tangency will be symmetrically arranged. Consequently, those vectors must each make an angle of, say, $\phi_k := k\pi/(n-1)$ with the horizontal. (For self-tangent epitrochoids, we'd use only-even $k$ or only-odd $k$; for tangent twins, we'd use all $k$. In any case, the $\phi_k$ cover all angles we could possibly need.) Moreover, the tangent vectors are direction vectors for the points-of-tangency.
So, we needn't restrict attention to a horizontal tangent vector at a point on the $x$-axis. Rather, we can consider an angle-$\phi_k$ tangent vector at a point with direction angle $\phi_k$. That is, we replace $(2)$ with $$\frac{\sin t+a \sin nt}{\cos t+a \cos n t} = \frac{\sin \phi_k}{\cos\phi_k}= \frac{\phantom{-}\cos t+na \cos nt}{-\sin t-na \sin n t} \tag{A.1}$$ Cross-multiplying across the left inequality, and then also across the right one, we find $$\begin{align} \sin(t-\phi_k) + \phantom{n}a \sin(nt-\phi_k) &= 0 \\ \cos(t-\phi_k) + na\cos(nt-\phi_k) &= 0 \end{align} \tag{A.2}$$ Now we note the following about $\phi_k$: $$n\phi_k = \frac{nk\pi}{n-1} = \frac{(n-1+1)k\pi}{n-1}=k\pi+\phi_k\quad\to\quad nt-\phi_k=n(t-\phi_k)+k\pi \tag{A.3}$$ Since $\sin(\theta+k\pi)=\pm\sin\theta$ and $\cos(\theta+k\pi)=\pm\cos\theta$, we can re-write $(A.2)$ as $$\sin(t-\phi_k)\pm a\sin(n(t-\phi_k)) = 0 = \cos(t-\phi_k)\pm na\sin(n(t-\phi_k)) \tag{A.4}$$ This is just $(2)$ with $t\to(t-\phi_k)$ and a now-ambiguously-signed $a$. The former requires adjusting our earlier analysis by defining $\omega:=\exp(i(t-\phi_k))$ and $\omega_n:=\exp(in(t-\phi_k))$, but how we use these quantities is unchanged. The ambiguous sign on $a$ becomes a non-issue once all occurrences of $a$ appear to even powers. Consequently, even this more-general framing of the problem ultimately leads to the same solution, $(\star\star\star)$. $\square$