What are quantum fluctuations, really?

You asked for a qualitative picture, so here goes.

Consider a simplified example: the quantum harmonic oscillator.

Its ground state is given by

$$ \Psi(x) = \text{const} \cdot \exp \left( - m \omega_0 x^2 / 2 \hbar \right). $$

Now suppose that we are measuring the position of this oscillator in the ground state. We could get any real value, with probability density $|\Psi|^2$. In reality, because of the exponential decay, most of the values are distributed within the window of width

$$ \Delta x \sim \sqrt{\frac{2 \hbar}{m \omega_0}}, $$

with the mean concentrated at $x = 0$.

Because measuring an individual oscillator is a complicated process which results in it getting entangled with the measurement device, let's simplify the problem – say we have an ensemble of non-interacting oscillators all in ground states, and we measure them all independently. The distribution of values $\{x_i\}$ is expected to mostly lie within the mentioned above window, but the actual values are unknown. We usually say that those are due to quantum fluctuations of the position operator.

The same thing happens with the quantum field, which upon inspection is nothing more than a collection of weakly interacting harmonic oscillators. If we take an ensemble of vacuum quantum field configurations (say, independent experiments at a particle accelerator), and we measure a value of the field at a point, we will see that it is not equal to zero (as it would be in the classical theory), but instead the values are distributed within an error window and are otherwise random. This are quantum fluctuations of the QFT vacuum.

These fluctuations are sometimes attributed to "virtual particles", or "virtual pairs", which are said to be "born from the vacuum". Sometimes it is also said that they can "borrow energy from vacuum for a short period of time". AFAIK these are just analogies, appealing to the consequence of Erenfest's theorem (the so-called time-energy uncertainty relation).

But the fluctuations undisputably have very real, measurable effects. Qualitatively, those effects come from a difference between the physical picture of the same thing painted by classical fields and quantum fields. You can say that quantum fields reproduce classical fields on certain scales (measured in the field value), which are much greater than the size of the error window. But once the precision with which you measure field values becomes comparable to the size of the error window, quantum effects kick in. Those who like painting intuitive pictures in their heads say that this is caused by quantum fluctuations, or virtual particles.

UPDATE

Belief that observed Casimir effect has something to do with vacuum fluctuations of the fundamental QFT is misguided. In fact, in the calculation of the Casimir force we use an effective field theory – free electromagnetism in the 1D box, bounded by the two plates. Then we look at the effective vacuum state of this effective QFT, and we interpret the Casimir force as a consequence of the dependence of its properties on the displacement between the plates, $d$.

From the point of view of the fundamental QFT however (Standard Model, etc.) there is no external conducting plates in the first place. If there were, it would violate Lorentz invariance. Real plates used in real experiments are made of the same matter described by the fundamental QFT, thus the state of interest is extremely complicated. What we observe as Casimir force is really just a complicated interaction of the fundamental QFT, which describes the time evolution of the complicated initial state (which describes the plates + electromagnetic field in between).

It is hopeless to try to calculate this in the fundamental QFT, just like it is hopeless to calculate the properties of the tennis ball by studying directly electromagnetic interactions holding its atoms together. Instead, we turn to the effective description, which captures all the interesting properties of our setup. In this case it is free electromagnetic effective QFT in the 1D box.

So to summarize: we are looking at the vacuum state of the effective QFT and the dependence of its properties on $d$. Alternatively, we are observing an extremely complicated fundamental system in a state which we can't hope to describe.


I completely share your frustration that people often describe very complicated and precise results as coming from "quantum fluctuations" without ever defining what that term actually means. After several years of hearing the term thrown about, I've come to the conclusion that "quantum fluctuations" is simply synonymous with a state being in a superposition of classical states (e.g. position eigenstates for a particle, or product states for a spin system).

We often work in a semiclassical regime where the state of interest is in a superposition that is strongly weighted toward a single classical state (or a narrow range of "similar" classical states). Then we can think of the system as "mostly" being in that dominant classical state, but with "quantum fluctuations" that result in our occasionally measuring something other than that dominant value due to the Born rule. But sometimes (e.g. in strongly coupled quantum systems) the state of interest is a fairly uniform superposition over a very wide range of different classical states, so the "single classical state plus small quantum fluctuations" picture is no longer useful.

Another thing that might be helpful to note is that people often think of quantum superpositions by analogy with thermal mixtures of different states in statistical mechanics. So when they talk about "quantum fluctuations" they are making an analogy with thermal fluctuations, where there is always a chance (often small) of measuring something other than the expected value of a variable in a given measurement. (In field theory, this analogy can be made precise by noting that the the partition functions $$Z = \int D\varphi\ e^{i S[\varphi]/\hbar} \text{ and } Z = \int D\varphi\ e^{-\beta H[\varphi]}$$ for a quantum and a statistical field are simply related by a Wick rotation between real and imaginary time.) Many experts have internalized this analogy so completely that they say things like describing a quantum superposition as "spending most of its time in state $x$ but some of its time in state $y$" as if it were a thermally fluctuating statistical ensemble, even if it's actually a Hamiltonian eigenstate so that strictly speaking nothing actually changes over time at all.