Why are there no composite particles with fractional charges?

To keep things manageable, I'll interpret the question like this: Given that quarks have their special pattern of electric charges with magnitudes $2/3$ and $1/3$ in units of the electron's charge, why do all hadrons (particles made of quarks) have integer electric charges in units of the electron's charge?

I'll use these inputs: Quarks are bound together by the strong force. Each quark species comes in three "colors" (this is what we call the strong-force charge, to distinguish it from electric charge), and the strong force ensures that only color-neutral combinations can occur as isolated particles. I'll explain what this means below.

Let $q_c$ denote a quark with color $c$, and let $\bar q_c$ denote an antiquark with color $c$. Isolated hadrons must be color-neutral, meaning that they must be invariant under the transformations \begin{align} q_c &\to \sum_{c'} U_{cc'}q_{c'} \\ \bar q_c &\to \sum_n \bar q_{c'}U^*_{c'c} \tag{1} \end{align} where $U$ is a $3\times 3$ unitary matrix with determinant $1$. (The group of all such matrices is called $SU(3)$.) The two basic color-neutral combinations are the meson-like combination $$ \sum_c \bar q_c q'_c \tag{2} $$ where $c$ is the color index, and the baryon-like combination $$ \sum_\pi (-1)^\pi q_{\pi(1)} q'_{\pi{2}} q''_{\pi{3}}. \tag{3} $$ The sum in (3) is over all permutations of the three color index-values, and the signs make the result completely antisymmetric. The fact that $U$ is unitary ensures that (2) is invariant, and the fact that $U$ has determinant $1$ ensures that (3) is invariant. The conjugate of (3) is also invariant, of course. Other invariants are sums and products of these basic invariants.

In units where an electron has charge $-1$, quarks $q$ have charge $+2/3$ modulo an integer, and antiquarks $\bar q$ have charge $-2/3$ modulo an integer. Since a meson-like combination involves the same number of quarks and antiquarks, we immediately conclude that it must have integer charge. And since a baryon-like combination involves three quarks or three anti-quarks, we immediately conclude that it must also have integer charge. All other color-neutral combinations are built from these, so all hadrons must have integer electric charge.

For more information:

  • This answer didn't try to explain why the strong force ensures that only color-neutral combinations can occur as isolated particles. If you want to learn more about that, the keywords include quantum chromodynamics and confinement.

  • This answer also didn't try to explain why quarks have their special pattern of electric charges. If you want to learn more about that, the keywords include electroweak symmetry breaking and chiral anomalies.

  • If you want to learn more about general conditions under which all electric charges must be integer multiples of some basic charge (which in the real world is the electric charge of a down-quark), the keywords are charge quantization and compact gauge group.

  • For an experimental perspective, which is what makes all of this mathematical stuff relevant, see anna v's answer.

Is there some theoretical reason for the lack of composite particles that would result in fractional charges?

It is an experimental fact that there are no fraction of the electron's charge particles in the data of the large number of experiments in hight energy physics.

So, a theory had to be developed that would fit mathematically this experimental observation.

This theory is the standard model of particle physics, given by the groups of $SU(3) \times SU(2) \times U(1)$ which give the allowed representations. It is chosen because it has no fractional charges in order to agree with the experimental data.. So the theory by construction cannot have a particle or particle-antiparticle combination of fractional charge for on mass shell observable particles.

The concept of color charges for the quarks and color neutrality for on mass shell particles can be seen here, which also contributes to the pairings. An interesting observation , though not directly relevant:

The rationale for the concept of color can be highlighted with the case of the omega-minus, a baryon composed of three strange quarks. Since quarks are fermions with spin 1/2, they must obey the Pauli exclusion principle and cannot exist in identical states. So with three strange quarks, the property which distinguishes them must be capable of at least three distinct values.

The standard model mathematics describes data we have at present as far as charges go completely, by construction. If in the future such a particle were detected, the standard model would have to be expanded or changed.

Quark jets have been experimentally studied to see whether their fractional charge is seen in the distribution of the jet particles, and the article says that their study verifies the fractional charges of the top-untitop pairs produced in the experiment.