What does it mean for a Hamiltonian or system to be gapped or gapless?

This is actually a very tricky question, mathematically. Physicists may think this question to be trivial. But it takes me one hour in a math summer school to explain the notion of gapped Hamiltonian.

To see why it is tricky, let us consider the following statements. Any physical system have a finite number of degrees of freedom (assuming the universe is finite). Such physical system is described by a Hamiltonian matrix with a finite dimension. Any Hamiltonian matrix with a finite dimension has a discrete spectrum. So all the physical systems (or all the Hamiltonian) are gapped.

Certainly, the above is not what we mean by "gapped Hamiltonian" in physics. But what does it mean for a Hamiltonian to be gapped?

Since a gapped system may have gapless excitations at boundary, so to define gapped Hamiltonian, we need to put the Hamiltonian on a space with no boundary. Also, system with certain sizes may contain non-trivial excitations (such as spin liquid state of spin-1/2 spins on a lattice with an ODD number of sites), so we have to specify that the system have a certain sequence of sizes as we take the thermodynamic limit.

So here is a definition of "gapped Hamiltonian" in physics: Consider a system on a closed space, if there is a sequence of sizes of the system $L_i$, $L_i\to\infty$ as $i \to \infty$, such that the size-$L_i$ system on closed space has the following "gap property", then the system is said to be gapped. Note that the notion of "gapped Hamiltonian" cannot be even defined for a single Hamiltonian. It is a properties of a sequence of Hamiltonian in the large size limit.

Here is the definition of the "gap property": There is a fixed $\Delta$ (ie independent of $L_i$) such that the size-$L_i$ Hamiltonian has no eigenvalue in an energy window of size $\Delta$. The number of eigenstates below the energy window does not depend on $L_i$, the energy splitting of those eigenstates below the energy window approaches zero as $L_i\to \infty$.

The number eigenstates below the energy window becomes the ground state degeneracy of the gapped system. This is how the ground state degeneracy of a topological ordered state is defined. I wonder, if some one had consider the definition of gapped many-body system very carefully, he/she might discovered the notion on topological order mathematically.

Gapped or gapless is a distinction between continuous and discrete spectra of low energy excitations. For a Hamiltonian $H$ with gapped spectrum, the first excited state has an energy eigenvalue $E_1$ that is separated by a gap $\Delta > 0$ from the ground state $E_0$. For example, a dispersion relation of the form $E = |k|$ is an example of a gapless (continuous) spectrum, whereas $E = \sqrt{k^2 + m^2}$ is an example of a gapped one. $k$ denotes the wave vector and can be any real number. $m$ is the mass which in this case is the cause of the gap.

This distinction leads to a qualitative difference in the physical behavior of gapped and untapped systems - most importantly it determines whether a material is a conductor or an insulator. There are quite fascinating processes that can give rise to a gap such as interactions (interesting examples are the mass gap in Yang-Mills theory, or the gap in BCS superconductivity).

Gapped and gapless are usually attributes for many-body Hamiltonians. A gapped Hamiltonian is simply one for which there is a non-zero gap between the ground state and the first excited state.