Top says 100% CPU used, but 50% of cores idle?
To expand upon Max's answer: Processors with hyperthreading report two cores to the operating system when there is really a single physical core. Linux sees 4 cores, but your processor has 2 physical cores.
The physical core has advanced scheduling capabilities and some extra transistors to handle the second thread, but for many processor-intensive applications the hyperthreading does not provide an actual benefit.
In my field, we benchmark the applications that will be running on a system and disable hyperthreading in the BIOS if it hinders performance.
EDIT:
If I load half the cores using cat /dev/urandom > /dev/null here is what I get on a modern Xeon running CentOS 6:
top - 19:32:38 up 1 day, 8:17, 3 users, load average: 12.13, 8.73, 4.07
Tasks: 694 total, 13 running, 681 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 50.4%sy, 0.0%ni, 49.6%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 49552656k total, 2693452k used, 46859204k free, 186448k buffers
Swap: 15999992k total, 0k used, 15999992k free, 469560k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
21193 root 20 0 98.5m 560 452 R 100.0 0.0 6:23.31 cat
21194 root 20 0 98.5m 560 452 R 100.0 0.0 6:22.76 cat
21195 root 20 0 98.5m 560 452 R 100.0 0.0 6:22.38 cat
21201 root 20 0 98.5m 560 452 R 100.0 0.0 6:21.98 cat
21202 root 20 0 98.5m 560 452 R 100.0 0.0 6:21.62 cat
21203 root 20 0 98.5m 560 452 R 100.0 0.0 6:21.36 cat
21206 root 20 0 98.5m 560 452 R 100.0 0.0 6:20.43 cat
21187 root 20 0 98.5m 560 452 R 99.6 0.0 6:24.45 cat
21204 root 20 0 98.5m 560 452 R 99.6 0.0 6:21.03 cat
21205 root 20 0 98.5m 560 452 R 99.6 0.0 6:20.73 cat
21213 root 20 0 98.5m 560 452 R 99.6 0.0 6:20.09 cat
21214 root 20 0 98.5m 560 452 R 99.6 0.0 6:19.70 cat
It's been over four years, and to be fair, I completely forgot about this question and only came back because I saw that I had received a Notable Question badge for it.
The problem was tangentially related to hyperthreading, but as I continually pointed out to responders, that was not the cause for the 50% idling.
The cause had to do with an inappropriately configured kernel dynamic ticks configuration. I was running Gentoo and using a custom-built kernel. After I upgraded the kernel, sometime in mid-2012, the issue resolved itself.