Is it possible for a system to have negative potential energy?

Yes, potential energy can be negative: consider Newton’s law of gravitation

$$V = -\frac{GMm}{r}$$

Where $G$ is Newton’s constant, $M$ and $m$ are masses, and $r$ is the distance between them. It can clearly be seen that this is always negative.

The key thing is that the absolute value of potential energy is not observable; there is no measurement that can determine it. The only thing that can be measured is differences in potential energy. So actually there is a redundancy in the equation above: if I add any constant to it, the difference in potential energy for two given separations is the same. The common form of Newton’s law of gravitation is set by the convention that two objects an infinite distance apart have zero gravitational potential energy, but this is purely a convention.

The idea of redundancies in physical descriptions is very important in theoretical physics, and is known as gauge invariance.


EDIT: following some comments by the original poster, I've added some more to this answer to explain the effect on total energy of a system of attracting objects at very short distances.

Let's consider two equal point masses $M$ separated by some distance $r$: the total energy of the system, using the above definition of potential energy, is

$$E = 2Mc^2 - \frac{GM^2}{r}.$$

If the total energy is negative, $E < 0$. We can rearrange this inequality to give a condition on the radius for negative total energy:

$$r < \frac{GM}{2c^2}.$$

Compare this to the Schwarschild radius $r_\mathrm{s} = 2GM/c^2$. The distance at which the Newtonian energy becmes negative is less than the Schwarzschild radius---if two point masses were this close they would be a black hole. In reality we should be using GR to describe this system; the negative energy is a symptom of the breakdown of our theory.

One can do the same calculation with two opposite charges $\pm e$ and find

$$r < \frac{e^2}{8 \pi M c^2 \varepsilon_0}.$$

We can then compare this to the classical electron radius $r_\mathrm{e}$ and similiarly find that $r < r_\mathrm{e}$ for a negative total energy. The classical electron radius is the scale at which quantum fluctuations must be taken into account, so again the negative energy is a symptom of the breakdown of the theory.


Basically the notion of absolute potential energy is undefined.

Definition: The change in potential energy of the system is defined as the negative of work done by the internal conservative forces of the system.

By this definition we can conclude that we are free to choose reference anywhere in space and define potential energy with respect to it.

For ex: Consider a system of 2 oppositely charged particles which are released from rest. Under the action of their mutual electrostatic forces they move towards each other. The internal electrostatic forces are doing positive work which results into the decrease in potential energy of the system.