How can black holes be so dense?
It's actually more extreme than you think.
The short story is this:
Associated with any amount of matter, there is an associated radius know as the Schwarzschild radius.
There is theorem in General Relativity that essentially states that if ever all of the matter is contained within the associated Schwarzschild radius, that matter must collapse to infinite* density, to a singularity.
Roughly speaking, there is a "runaway" effect where the pressure of the compressed material actually contributes to the gravity acting to compress the matter further. The process "feeds" on itself.
This collapse forms a black hole which consists of the singularity as well as an event horizon.
In the case of the Earth, the event horizon would enclose a volume roughly comparable to the volume of a peanut.
However, the matter does not fill that volume; the matter is compressed to an infinitely* dense singularity. Once an outside entity falls within that volume, it cannot get back out; it must fall "into" the singularity.
*Strictly true only for non-spinning, charge neutral matter. See Ron Maimon's comment.
You ask how atoms can be that tightly compressed. Atoms are made of electrons and quarks (the protons and neutrons are made of quarks) and as far as we know electrons and quarks are point like i.e. they have no size. So in principle they can be compressed to infinite density if you squeeze hard enough. At this point someone is going to point out that all particles are ultimately made from strings, and assuming string theory is correct we do expect the rules to change at sizes of around the string scale. However this is probably about $10^{-34}$ of a metre so let's ignore it for now.
Anyhow, quarks and electrons resist being compressed together for various reasons, so under everyday conditions we see everyday densities. The thing is that gravity is always additive - more mass means more gravity and more pressure, and we can keep piling on more matter and the gravity and pressure will keep rising. At some point the pressure gets so great that electrons react with proton to form neutrons, and we get matter made up just from neutrons. This is called neutronium and it's the state of matter found in neutron stars. Neutronium has a density of around $10^{18}$ kg/m$^3$ while ordinary matter is between $10^{3}$ and $10^{4}$ kg/m$^3$
When you try to compress neutrons they resist due to a phenomenon called degeneracy pressure, but you can keep adding mass and this keeps increasing the pressure until even the degeneracy pressure can't keep the neutrons apart. At that point it's unclear exactly what happens because we don't understand the physics that well. However it's possible that the next stage is that the neutrons dissolve into a sea of quarks and it may form something like strange matter, which has it's own degeneracy pressure that resists further compression.
But if you add even more mass you can overcome even the quark degenracy pressure and at that point the quarks start to collapse. Remember that I started out saying quarks are point like, so when they start to collapse they can collapse without limit and the density will become infinite.
That in a nutshell is how you can collapse the Earth to the size of a pea.
Well, maybe not. If string theory is correct quarks aren't point like particles. Once you go down to length scales around the Planck length the quark would start behaving like a string not a point particle, and we expect the rules to change. No-one knows what will happen, but it seems likely the collapse would be stopped before the density becomes infinite.