Can something without mass exert a force?

Yes, photons can. See https://en.wikipedia.org/wiki/Radiation_pressure (and photons are certainly massless).

PS In fact, any massless particle has momentum(*) and if it is scattered on a body, it changes its own and the body's momentum, which is what a force does.

(*) $p = \hbar k = E/c$ where $E$ is its energy and $c$ is speed of light


Newton's 2nd Law of Motion gives the impressed force as $F=dp/dt$, so a physical theory for a massless particle exerting a force requires that the particle have momentum, $p$.

First we will discuss mass, momentum, the force law, and Special Relativity.

In Newtonian physics mass is identified in two ways: by it's inertia, or as the quantity of matter. The ordinary measurement is by comparison, with a known force, or a balance. Early experiments, 1905-06 with charged particles accelerated through a controlled voltage found that the inertial mass varied with the change in kinetic energy acquired, confirming earlier predictions of Lorentz, 1904, and Einstein, 1905.

The term rest mass, denoted $m_0$, entered the physics lexicon, along with the longitudinal and transverse mass; these two additional terms were required because the measurements vary depending on where you are. For some purposes these are still useful, but it turns out that the rest mass is a relativistic invariant, remaining unchanged under a Lorentz boost. So in modern terminology mass means rest mass, and is denoted by $m$, or for the old fashioned, occasionally $m_0$.

The Lorentz factor, $$\gamma=1/\sqrt{1-v^2/c^2},$$ provides the relativistic correction required for momentum, $p=\gamma mv$ replacing the Newtonian $p=mv$ for a particle with mass. So Newton's Second Law of Motion remains $F=dp/dt$.

Now we will discuss light, and how it carries momentum, and the concept of being massless. For the Physics FAQ summary, see here.

Light, as first noted by Maxwell, travels with the same speed in the vacuum, independent of the observer's inertial reference frame; for the calculation see Deriving the speed of the propagation of a change in the Electromagnetic Field from Maxwell's Equations. For the history, see here.

The relation between energy and momentum of light can be found from the Poynting vector, as derived from the field equations in vacuum. The result is that $p=E/c$, which follows from the radiation pressure.

The relativistic equation for total energy includes momentum, and is $E^2=(pc)^2+(mc^2)^2$; when $p=0$ this simplifies to the iconic $E=mc^2$. For the case with momentum, but no rest mass, we get $E=pc$, which provides the relativistic expression for the momentum of light: $p=E/c$, which is consistent with Maxwell's equations.

So for a self-consistent relativistic theory, if we start with Maxwell's equations, we end up with freely moving light having momentum, but no mass. If the light is trapped in a stationary box it will contribute to the weight of the box in proportion to the energy of the light, $m_L=E_L/c^2$ to the mass of the box without the trapped light.

At this point we have shown the road map for (a) relativistic force law, $F=dp/dt$, and (b) that light has momentum, $p=E/c$, and (c) that this momentum implies that light has no mass. So we now introduce the photon, a particle of light.

Historically, Planck introduced the hypothesis that the energy of light may be quantized, $E=hf$, where the energy of each quanta is determined by it's frequency. Einstein applied this concept to the photo-electric effect, and de Broglie, using Einstein'g Special Relativity plus the Planck relation proposed a complementary relation for the wavelength of a mass with momentum, $p=h/\lambda$. This expression is equivalent to the Planck relation when $p=E/c$ is inserted on the left hand side, because $\lambda f=c$.

Planck and de Broglie, together, provide the foundation for wave mechanics; the term "photon" for this massless quanta, or particle of light first appeared in the literature in 1926.

So in conclusion, yes, something without mass, the photon, can apply a force; this is done through it's momentum.

Experimental verification must be done carefully, for a force may be applied by absorption, or reflection. In the case of absorption the change of momentum is $|p|$, while for reflection it is doubled, as the momentum acts both coming and going.

For absorption the demonstration can be made with Crooke's light mill, often referred to as a radiometer; this clearly responds to light, but the analysis is complex, and does not directly show the pressure due to light.

The direct detectionof light pressure due to reflection requires a fine torsion balance mirror in a vacuum, first performed successfully in 1901; today it can be performed in an undergraduate advanced physics lab.