Chemistry - Why does the exchange interaction in Hartree-Fock theory lower the total energy?
Solution 1:
I have been watching this question with keen interest, but wanted to let someone else go for the bounty -- However it expires tomorrow and no one has answered, so I will give my perspective.
Why does the exchange interaction in Hartree-Fock theory lower the total energy?
The answer is given in the question:
This integral is always positive, and so the exchange contribution to the total energy is always negative.
Again, I appreciate that this very simple answer may not be satisfactory for some, which is why I was waiting for someone else to go for the bounty.
A comment by the person who originally asked the question (not the user that offered the bounty) offers an attempt to explain why it might make phenomenological sense for the energy to go down, and I will not repeat that here because it's really them that deserves to get any points that might be awarded for that line of thinking.
However I have a different perspective and it is this: Feel free to try to make phenomenological models or analogies in your head (such there being an outward "pressure" or "reduction in effective volume") if it helps you in any way (maybe to remember things easier, or for your own peace of mind). But in my experience, such "analogies" using "classical physics" thinking (like pressure and forces) do not in quantum mechanics always work the way you would expect. A good example is quantum entanglement, which has absolutely no classical analog; and another example is quantum spin, which is a bit like classical angular momentum but is not precisely the same.
The best answer really, is that the exchange integral is always positive, and in the equation for the energy it has a negative sign, so it will always reduce the total energy.
You can come up with phenomenological ways to conceptualize this, and perhaps someone else will come up with a valid and convincing argument for why that conceptualization does not work, but the bottom line is that the integral is mathematically always positive, so by definition of the "exchange integral", the "exchange interaction" always lowers the energy here..
Solution 2:
I'll try to give my interpretation of the "physical" explanation of why exchange would lower the energy.
For the true wavefunction, the motion of all the electrons should be correlated, with the classical view being that the electrons are avoiding each other to minimize repulsion. With Hartree-Fock, we find an approximate wavefunction by solving for 1-electron functions using the average potential of the other electrons rather than having each electron feel the instantaneous potential of all the others. This eliminates most, but not all of that correlation.
Exchange terms arise due to the Pauli Exclusion Principle and the fact that electrons are fermions. This causes same spin electrons to be correlated, as they can't occupy the same space. While these electrons aren't fully correlated as they would be if you did Full CI, the exchange part of the correlation is reproduced exactly by Hartree-Fock. If not for exchange, these electrons could be located close to each other, which would be highly repulsive and increase the energy. So by avoiding these interactions, the energy is reduced and this correlative effect is what leads exchange to always be negative.
There is a little more in this same vein on Physics SE and the sources therein.
Solution 3:
Can anyone explain why the exchange contribution to the total energy is negative?
I find it misleading that exchange interaction is treated as something that changes total energy of the system. This lowering of energy is actually due to the Hartree-Fock scheme being in principle inexact, and is not really specific to indistinguishable particles.
Let me elaborate. Consider a hypothetical system of distinguishable particles with exactly the same masses, charges and spins (and all their other properties entering the Schrödinger's equation). Let these parameters be equal to those of an electron. Now, if you solve the Schrödinger's equation for this system in the electromagnetic field of an atomic nucleus, you'll get some (spinor-valued) wavefunctions $\Psi_n$, where $n$ enumerates the states. These wavefunctions won't in general be antisymmetric.
But the symmetry of the Hamiltonian, namely its invariance with respect to exchange of a pair of particles (since all their relevant properties are the same), tells us that there's a large degeneracy. If you linearly combine the degenerate states corresponding to the same energy so as to get symmetric $\Psi_k^{\mathrm s}$ and antisymmetric $\Psi_k^{\mathrm a}$ linear combinations, these will still be solutions of the Schrödinger's equation we started with.
If you throw away the symmetric solutions and only leave the antisymmetric ones, these will actually be the solutions for the case where our particles are indistinguishable. You see? We've just arrived at exchange interaction between indistinguishable particles, without any change in energy. What we got is just a possible increase in ground state energy due to losing a bunch of states (and thus renaming one of the formerly excited states to the ground one).
Now, why do we get lowering of states in inexact schemes when we introduce such antisymmetrization? That's directly related (at least for the antisymmetrized ground state) to the variational principle, which tells us that an approximation of a ground state will always have mean energy $E_{\text{approx}}$ (which is the Rayleigh quotient) obeying
$$E_{\text{approx}}\ge E_0,$$
where $E_0$ is the exact ground state energy.
Before introducing the Slater determinant, the trial wavefunction was smooth and generally nonzero at the loci of electron-electron collisions. But the exact (even non-symmetric) eigenfunction should have a local minimum (a cusp, see here for an example) or a (smooth) zero at such loci due to the Coulomb repulsion. After antisymmetrization, many of such loci of collisions will get zeros. This will make the approximation better, and the mean energy lower, closer to the exact eigenenergy.
If instead of Hartree-Fock method we used some other approximation that had begun with exact treatment of electron collisions, sacrificing something other, then such antisymmetrization might not lead to any improvement in estimation of eigenenergies (limit case being very precise numerical treatment yet disregarding Pauli principle, which is basically equivalent to an exact solution for distinguishable particles).