when should high z mode of oscilloscope be used to measure signals?

When working with a lot of non-RF situations, and dealing with voltage signals, you want to use the high-Z mode. Many lower frequency, more general purpose scopes (generally sub-1 GHz bandwidth) only have a high-Z mode, or if they do have a 50 ohm mode, it is not the default.

Once you start dealing with RF systems and higher frequencies, people tend to talk about power signals and not voltages or currents. Everything is impedance matched and we have to take care that our measurement does not impact this. Reflections due to mismatch could not only give errors on the readings, but cause amplifiers to start oscillating or mixers to perform incorrectly. At these frequencies we tend to either use a 50 ohm (or whatever your systems impedance is, 75 ohms is still commonly used in television broadcast and related applications) or use an active probe which connects to our scope with a 50 ohm line.

A key reason why is not only the fact that we are dealing with matched impedance systems, but also noise. Oscilloscopes are generally used from DC to their bandwidth. As we increase the bandwidth, so does our thermal noise. As a result, the voltage noise becomes very large. To put this into perspective: with 1 MOhm impedance and 1 GHz bandwidth, we have 4 mV RMS noise. This means that if we probe a 50 Ohm system with our 1 MOhm probe, the noise floor of our measurement is about -35 dBm. For a 10 MOhm probe (common in a times 10 probe) that would go to a noise floorof just -25 dBm. These are very bad noise floors, making the scope almost useless for any measurement! For comparison: a 50 ohm probe wit the same 1 GHz bandwidth would have a noise floor of just 28 micro volts or -83 dBm. Keep in mind that the oscilloscope itself will add noise on top of this, but will likely still perform far better than it possibly could wit a 1 MOhm impedance.

In my experience, the Hi-Z mode (and a 10X probe for even higher impedance) is the normal mode - and on many 'scopes, the only mode.

The 50 Ohm mode would only be used for RF or other situations where you are measuring signals in a 50 Ohm trasmission line.

Good question! The High Z input of a scope is typically used for measurements up to 700 MHz or so, then people switch over to 50 ohm mode. The 50 ohm path is typically much lower noise.

Use the high Z path to avoid loading down your circuit - and the 50 ohm path to match impedance and avoid reflections and other mismatch effects.