Why is it desirable in an amplifier to have high input impedance and low output impedance?
Actually, the premise of your question is only true if the signals you are interested in are voltages. In that case, if the amplifier draws no current through its input (has infinite, or at least very high input impedance), then connecting it to a source won't affect the signal voltage, regardless of what the source impedance is.
Similarly, when you connect a load to the output of your amplifier, if the amplifier has zero output impedance, the signal voltage won't change, regardless of the current drawn by the load.
These properties make it much easier to analyze the behavior of the system overall.
However, if the signals you're interested in are currents rather than voltages, you want your amplifier to have zero input impedance and infinite output impedance for the same reasons.
The ideal amplifier should not draw any current at all from its input. Assuming a two input amplifier the signal current in both input probes is zero. In other words the input impedance must be infinite!
The output, shoul operate as the ouptut of an ideal voltage source. This means that the pottential between the output and the ground must be \$A(v_2-v_1)\$, no matter how much current would a load conected to the output would draw. In other words the output impedance must be zero!
For a real amplifier, the input impedance must be as large as possible while the output impedance must be as low as possible!
Simplified answer relating mainly to audio amplifiers: -
An audio amplifier with a low output impedance can deliver larger powers to its loudspeaker more efficiently than an amplifier with higher output impedance. Thus, you will find that audio amps have output impedances measured in less than 1 ohm and in a lot of cases in milli-ohms.
On the other hand, a weak and feeble signal from (say) a microphone doesn't want to struggle with feeding its signal into an amplifer with a too low an input impedance - this may potentially (and significantly) attenuate the signal and require higher levels of amplification to compensate thus increasing noise pick-up etc..
If, an audio power amplifier's impedances were reversed, as you suggest, it would potentially produce a noiser signal on the loudspeaker and, be power-inefficient to the point that it would get significantly warmer in generating the equivalent sound level from the speaker.
There are other problems with low input impedances in that reshaping of the frequency response of some microphones may occur. This is also true of the high output impedance - the electro-mechanical nuances of the speaker may cause some signals to appear louder than what they should be.
As an aside, there are many amplifiers that do have a fairly low input impedance and these are typically in the field of RF where you need to match impedances to prevent signal reflections.
No-doubt there are other examples I've missed.