What do the colors mean on damage numbers?

• Yellow numbers are critical hits that you deal.
• White numbers are normal hits that you deal.
• Green numbers are healing you receive.
• Red numbers are damage you receive.
• Added in patch 2.4, orange numbers are the largest hit you dealt recently.

Blizzard's philosophy behind orange (and other) damage numbers is explained thoroughly in their blog post, but here are the rules regarding when they appear:

• Damage numbers must be over 10,000 to be considered
• If the damage number to be displayed is larger than the last that was displayed in orange, then display this new number in orange
• Decay the value of the largest number by 3% every second
  • This reduces the likelihood that you’ll go on too long without seeing any highlighted numbers
• Ignore the first 10 large numbers
  • This allows for the system to calibrate itself
• If no damage has been dealt for 10 seconds, reset the system

In addition to dpatchery's answer, there is now (patch 2.4 also an orange mode, which is your most damage done in some time (see also this question).