What makes the 'special groups' ($\det A = 1$) special?

This is a very broad topic, so I can only mention a few starting points, and I'll mostly just discuss the special linear groups, $$SL(n, \Bbb F) := \{A \in GL(n, \Bbb F) : \det A = 1\} .$$

Other special groups, e.g., $SO(n, \Bbb R)$ and $SU(n, \Bbb R)$, are intersections of a suitable parent group and the special linear group over the appropriate field, for example, $$SO(n, \Bbb R) = O(n, \Bbb R) \cap SL(n, \Bbb R) ,$$ and so inherit some features from both groups. More can often be said in particular cases.

For any field $\Bbb F$ the group $SL(1, \Bbb F)$ is trivial, by the way, and so its behavior is exceptional among special linear groups. For ease of exposition henceforth we'll take $n > 1$. I'm happy, by the way, to give suggestions for further reading about any of these topics if you're interested.

Geometry First, the group $SL(n, \Bbb R)$ acts transitively on $\Bbb R^n - \{ 0 \}$, so the only property of a vector in $\Bbb R^n$ preserved under the action of $SL(n, \Bbb R)$ is whether the vector is zero or not. (Throughout I mostly refer to the real special linear group, but most of what's said here applies just as well to special linear groups over some/all other base fields).

On the other hand, the action of $GL(n, \Bbb R)$ on $\Bbb R^n$ induces an action of the space of volume elements $\bigwedge^n \Bbb R^n$---this vector space is $1$-dimensional and so linearity implies that this action is given by $$A \cdot \omega = (\det A) \omega .$$ So, $SL(n, \Bbb R)$ consists precisely of the linear transformations that preserve a volume form, or, equivalently, both (unsigned) volume and orientation on $\Bbb R^n$. (Thus, for example, the special orthogonal group $SO(n, \Bbb R) = O(n, \Bbb R) \cap SL(n, \Bbb R)$ consists of linear transformations that preserve a inner product on $\Bbb R^n$ and a choice of orientation; these two objects determine a volume form which is thus also preserved.)

Just as the definition of the orthogonal group characterizes Euclidean geometry---namely, the geometry on $\Bbb R$ under transformations of preserving lengths (and therefore angles)---the special linear group characterizes the geometry on $\Bbb R$ under transformations preserving a volume form. This is called equi-affine geometry, and since $O(n, \Bbb R) \lneq SL(n, \Bbb R) \lneq GL(n, \Bbb R)$, an equi-affine structure contains strictly more information than an affine structure (in which only incidence is preserved---this corresponds to $GL(n, \Bbb R)$) but strictly less information than an orthogonal (Euclidean) structure.

Alternatively, since $SL(n + 1, \Bbb R)$ acts linearly and transitively on $\Bbb R^{n + 1} - \{ 0 \}$, it descends to an action on the real projective space $\Bbb R P^n := \Bbb P(\Bbb R^{n + 1} - \{ 0 \})$. Since $SL(n + 1, \Bbb R)$ maps $k$-planes to $k$-planes (in particular for $k = 2$, this action preserves projective lines, which are precisely the unparameterized geodesics of the canonical ("flat") projective structure on $\Bbb R P^n$. So viewed, projective space, equipped with the projective lines, serves as a model for projective (differential) geometry the same way that Euclidean space serves as a model for Riemannian geometry. Similar constructions give rise models of other, more exotic, but still interesting, geometric structures, realized as $SL(n + 1, \Bbb R)$-homogeneous spaces.

Algebra It turns out that $SL(n, \Bbb R)$ is exactly the commutator subgroup $\langle \{ A B A^{-1} B^{-1} : A, B \in GL(n, \Bbb R) \} \rangle$ of $GL(n, \Bbb R)$. This is closely related to the fact that the Lie algebra $\mathfrak{sl}(n, \Bbb R)$ of $SL(n, \Bbb R)$ is simple: Its only ideals are the Lie algebra itself and $\{0\}$. (In fact, $SL(n, \Bbb F)$ is very nearly simple: Its center is $Z := Z(SL(n, \Bbb F)) = \{\zeta I_n : \zeta \in \Bbb F, \zeta^n = 1\}$, and the quotient $PSL(n, \Bbb F) = SL(n, \Bbb F) / Z$ is simple, except for a few cases over small finite fields.) Simplicity has many powerful consequences (see the next section).

Since the determinant of $A$ is the product of the eigenvalues of $A$, by the way, the constraint on the spectrum $(\lambda_a)$ of $A \in SL(n, \Bbb F)$ is precisely that $\lambda_1 \cdots \lambda_n = 1$, but this places no constraint on any single eigenvalue $\lambda_a$ beyond $\lambda_a \neq 0$.

Representation theory The fact that $\mathfrak{sl}(n, \Bbb R)$ is simple makes available the well-developed theory of representations of (semi)simple Lie algebras and Lie groups, which applies just as well to the other groups you mention, $O(n, \Bbb R), SO(n, \Bbb R), SU(n)$, as well as others. In some ways, $SL(n, \Bbb C)$ is the easiest family of semisimple Lie groups to understand and so it makes for a natural first example when learning new concepts in the topic. This gives, among many other results, a complete description of all of the representations (in both the group and algebra settings), and complete reducibility of all representations, i.e., every $\mathfrak{sl}(n, \Bbb C)$-representation can be decomposed essentially uniquely as a direct sum of irreducible representations. (This theory can be bootstrapped to recover the representation theory of $GL(n, \Bbb C)$, which is similar to the representation of the special linear group, but there are some key differences: For example, since the special linear group preserves a volume form on $\Bbb C^n$, contraction with that volume form defines a natural isomorphism $\Bbb C^n \cong \bigwedge^{n - 1} (\Bbb C^n)^*$, but the general linear group preserves no such volume form, so $\Bbb C^n$ and $\bigwedge^{n - 1} (\Bbb C^n)^*$ are inequivalent as $GL(n, \Bbb C)$-representations.)

The normal subgroup $SL_n(K)$ of matrices of determinant in $GL_n(K)$ is special, because its an algebraic group whose Lie algebra is simple, i.e., very special. This plays a similar role as simple groups among all groups. So special linear groups are quite special. For the orthogonal groups, the Lie algebra is simple, too, but there we have two groups, $O(n)$ and $SO(n)$ with the same Lie algebra. Over the real numbers, $O(n)$ has two connected components, and the one with the identity is $SO(n)$.