Why is $SU(3)$ chosen as the gauge group in QCD?

Let me first make a general remark about internal symmetry groups, unrelated to our problem of the correct symmetry group for QCD.

The symmetry must act on Hilbert space as a unitary operator for the conservation of probability.

Now let us turn to the strong interaction. The most important experimental facts were that

1. Observed hadron spectrum was understood as that of bound states of quarks.
2. The SLAC experiment found that in high energy, deep inelastic scattering, the bound quarks behaved as if they were weakly coupled.
3. The measured decay rate of $\pi\to\gamma\gamma$ was nine times greater than expected.

In theoretical language, we thus require the properties of confinement (hadrons) and asymptotic freedom (coupling gets weaker at high energies).

A theoretical result is that only Yang-Mills $SU(N)$ gauge theories exhibit asymptotic freedom.

The question is now, what is the value of $N$? Well, experimental fact number three helps us. If the quarks transform in the fundamental triplet of $SU(3)$, the decay rate is enhanced by $3^2$.