Quarks in a hadron: where does the mass come from?

You say:

Now, when we talk about energetically favourably bound systems, they have a total mass-energy less than the sum of the mass-energies of the constituent entities.

and this is perfectly true. For example if we consider a hydrogen atom then its mass is 13.6ev less than the mass of a proton and electron separated to infinity - 13.6eV is the binding energy. It is generally true that if we take a bound system and separate its constituents then the total mass will increase. This applies to atoms, nuclei and even gravitationally bound systems. It applies to quarks in a baryon as well, but with a wrinkle.

For atoms, nuclei and gravitationally bound systems the potential goes to zero as the constituents are separated so the behaviour at infinity is well defined. If the constitiuents of these systems are separated to be at rest an infinite distance apart then the total mass is just the sum of the individual rest masses. So the bound state must have a mass less then the sum of the individual rest masses.

As Hritik explains in his answer, for the quarks bound into a baryon by the strong force the potential does not go to zero at infinity - in fact it goes to infinity at infinity. If we could (we can't!) separate the quarks in a proton to infinity the resulting system would have an infinite mass.

So the bound state does have a total mass less than the separated state. It's just that the mass of the separated state does not have a mass equal to the masses of the individual particles.

You can look at this a different way. To separate the electron and proton in a hydrogen atom we need to add energy to the system so if the added energy is $E$ the mass goes up by $E/c^2$. As the separation goes to infinity the energy $E$ goes to 13.6eV. If we try to separate the quarks in a proton by a small distance we have to put energy in and the mass also goes up by $E/c^2$ just as in any bound system. But with the strong force the energy keeps going up as we increase the separation and doesn't tend to any finite limit.


This happens because of a property of the strong force, called Asymptotic Freedom. This causes the interaction between quarks to get asymptotically weaker as the distance between them decreases. This is the reason why quarks are always found in a bound state and are not freely available in nature.

The strong force confines quarks to a region where they might possess a high amount of kinetic energy. (i.e. free movement for them.)

Some related math:

The strong force potential, a classical approximation, actually (from studies on bound states of quarks and antiquarks, can't say it is universal) can be represented by: $$V(r) = - \frac{4}{3} \frac{\alpha_s(r) \hbar c}{r} + kr$$

If you analyze this, the $\frac{1}{r}$ term dominates for short range, whereas the $r$ term is more significant at a relatively greater distance. The $r$ term is the confinement term, and shows that the potential actually increases at a further distance, and hence it is favorable for quarks to remain close together. It is similar for protons and other bound states too. (At least on a relatively small time scale, because the proton is generally assumed to be unstable, as @CuriousOne mentioned in the comments. The lower limit on a proton's half life is believed to be around $10^{35}$ years. )

This question will be an interesting read: What's inside a proton?