Why do some AC adapters and power supplies generate a whining noise, and what can I do about it?
Most power conversion devices contains coils, such as transformers or inductors. These components use electromagnetism to convert AC mains power to low-voltage DC power. The varying magnetic fields generated by these components can cause them to physically vibrate at high frequency, resulting in a high-pitched noise.
Most modern AC adapters are switched-mode power supplies. The internal switching frequency of an SMPS is typically low when unloaded and increases with load up to a certain point depending on the design. The no-load frequency is often low enough to be within the human hearing range. In addition, in low or no-load situations, the PWM used to regulate voltage at the inverter stage will be at a low duty cycle creating a "spikey" output profile which is more prone to causing vibration in coils, and the transformer itself will tend to vibrate as well (see Daniel R Hick's answer below for more details). Together, these can lead to audible noise especially in cheaper units which fail to suppress this noise.
Under load, a properly functioning SMPS should operate at a frequency well above the human hearing range, typically 50 kHz or higher (although some older designs operate at 33 kHz). However, the same noise can occur under load with a poorly designed or defective power supply as the coils may vibrate under electrical stress at a subharmonic frequency.
Coils used as inductors or transformers in other electronic devices, including those on motherboards, graphics cards, or other computer components, can also vibrate during operation. As such, a defective device can generate audible coil whine during operation.
This is why you sometimes see weird gobs of glue on coils inside electronic devices. The glue helps reduce the vibration and noise the coils generate during normal operation. It is entirely possible for users to apply glue onto coils using a glue gun to suppress coil whine, and people have successfully done so on their computer parts. However, you generally can't do this easily on small wall chargers of the sort you mentioned without risking damage to the charger or exposure to potentially dangerous voltages.
Ultimately, a whining noise isn't necessarily a sign of trouble in cheaper wall chargers when little or no power is being drawn from them. However, a computer PSU or laptop charger that generates coil noise especially when under load may be defective and you may want to consider replacing it.
More information on coil noise can be found in this Wikipedia article.
A "switching" power supply (like virtually all modern computer power supplies) works by "rectifying" the incoming 120V 60Hz (in the US) AC power into DC (at around 170 volts), "filtering" with capacitors, then using a semiconductor circuit to "chop" the DC voltage around 1000 times a second to turn it back into crude AC. (What's referred to as a "square wave", vs the "sine wave" of ordinary AC.) This "chopped" voltage then runs through a transformer to produce the desired output voltages. The outputs are again rectified to DC and filtered, to produce the desired voltages for the computer.
With this scheme, basic voltage regulation is performed by adjusting the "duty cycle" of the chopped voltage. When the power supply is lightly loaded the circuitry doesn't produce a nice symmetrical "square wave" but instead a series of narrow spikes, and that "spikey" waveform is more likely to produce annoying audible noise in the transformers and other components and is also more likely to produce "electrical noise" that you would, eg, hear in a nearby radio.
Additionally, when a power supply is lightly loaded more of the magnetic field inside a transformer escapes to the case of the transformer and to surrounding components (since less is captured by the "secondary" coil of the transformer), and this "escaping" magnetic field is more apt to cause noise.
The transformers are created in part by gluing plates of metal together. The AC fields causes back and forth forces in the metal plates. As the transformer ages the plates begin to separate and allows for movement on the plates which vibrate causing the humming sound you hear.