Practices for using a scope, scope probe and termination resistor for a proper measurement setup

Ideally, an oscilloscope would give you an option between 1M ohm input resistance (in parallel with perhaps 15pf capacitance) and a 50-ohm input termination - often cautioned not to apply too much power, lest it over-heat. This 50-ohm internal termination is quite vulnerable to burning up into an open-circuit, so it is wise to do a measurement check to see that it is still there, and still 50-ohms.

Many 'scopes give you no option - the default 1M ohm input applies. In this case find a BNC "T", and add a 50-ohm termination right at the 'scope input. Use the 1X scale, and use 50-ohm coax to connect to your signal source, not the 1X probe. Not as good as the internal termination described above, (a short unterminated section often remains), but its about the best you can do. Don't forget that its there - when you go to measure that +24V DC supply, you may smell smoke.
BNC T with 50 ohms Most 1X/10X probes have cable impedance higher than 50 ohms between probe tip and BNC connector, so don't use a probe in a 50-ohm measuring system to make a careful amplitude measurement.

A good function generator takes care to drive its output through a 50-ohm resistance, so its open-circuit output should be twice as high as its 50-ohm terminated output. It is common for function generator outputs to be 20V p-p open-circuit, and 10V p-p when terminated with 50 ohms. Such a generator might say in its manual, "Will deliver 10v p-p to a 50 ohm load".
A source calibrated to deliver 0 dBm power will deliver that power into a proper load (usually 50 ohm) - of course it delivers no power to an open-circuit, and very little more to a 1M 'scope input. Your oscilloscope (set to 1X) might make an RMS amplitude measurement - power is simply \$ \frac {{V_{rms}}^2}{50} \$ when you've got that BNC-T with 50 ohm termination attached.


I simple terms:

Your default everyday setup with a low-mid range oscilloscope should always be to use 1M input impedance probes and scope set to 10X if available. This setup is non-invasive, and gives you good bandwidth.

If you are a beginner, you can basically stop reading here.

1X mode is meaningful basically only when you need to measure voltages that are too low for 10X mode. Which is never. Almost.

50ohm input mode is only used in certain more advanced cases, where you want to push the bandwidth or have the scope simulate a missing load to avoid reflections. If you connect a signal generator straight to the scope using a coax with no other load on the coax, then 50ohm mode is appropriate. If it is connected usign a probe, then you shall use 1M mode. If you connect the signal generator to a load, and hook up the scope halfway on the coax using a T-connctor, then 1M is the appropriate mode. 50ohm is not "better" than 1M. It is "different", and shall only be used when appropriate.

On a side note, if 10X mode does not offer sufficient bandwidth for a measurement you need to make, and you can't afford high frequency active probes, then there is a neat trick you can do with the 50ohm mode. By using 50ohm mode, a 50ohm coaxial cable, and a series resistor at your measurement point, you can get the bandwidth of an active probe. With the disadvantage of lower impedance. But for digital signals this is often acceptable. The method is explained here: https://www.youtube.com/watch?v=9GqfZMcrAFY


And if we use a 50 Ohm termination then our signal's amplitude halves. How is this problem compensated?

There is no "problem" at all. Signal is whatever it is, and scope probe has nothing to do with the reality of it. If you don't use 50-Ohm termination, the signal "doubles" as compared to what the signal generator dial assumes (although there could be scaling options depending on assumed termination setup. But is done for dial convenience only). If you load the other cable end with 50-Ohm terminator, the voltage will be half of that. It is what it is, and you should always take measurements with high-impedance probe (>> than 50 Ohms). There are coax probes with 2K input impedance, or even with 500-Ohms, which are still qualified as "non-invasive" for 50-Ohms signaling environment, technically.

There are however situations when you want to use INTERNAL 50-Ohm IMPEDANCE input mode that can be found in modern high-speed digital scopes, for direct connection to coax cables. In this case everything gets matched automatically, yet the signal is still what is is, and no "2x correction" is necessary.