What are all the font styles I can use in math mode?

In texdoc symbols (the Comprehensive LaTeX Symbol List) you can find Table 316:

The footnotes are explained in the document. Table 327 will additionally explain bold math.

Oh and needless to say but if you were asking this question because you need more mathematical symbols, the Comprehensive List is just your document. Greek variants, Hebrew, Tables 139 to 147 are letter-like symbols ... you'll probably never run out of symbols again.

Edit: After reading this answer https://tex.stackexchange.com/a/60018/13450 I feel the need to also point to the mathalfa package that is referenced there, providing even more alternatives than shown above.

Edit 2: This answer seems so popular that I decided to include the mathalfa table as an image as well. This is taken from the mathalfa documentation and some of the fonts are commercial or need to be installed from external sources. See the documentation for more information if you consider using any of these fonts. Warning, very long table ahead (stitched together from a multi-page table).

Edit 3: With this thread being so popular for reasons I don't entirely understand, I feel compelled to say that there is rarely a point in using more styles than regular, bold, italic, script/calligraphic (I wouldn't even mix those) and blackboard bold. What these tables really show are typefaces you can use for these styles, not a huge number of styles (which would be pointless and ugly anyway). If, however, you are just searching for math fonts to go with your main font, the overview you probably actually want before even consulting these tables is the list of math fonts on the LaTeX Font Catalogue.

LaTeX kernel

The LaTeX kernel defines several math alphabets in fontmath.ltx

\DeclareSymbolFontAlphabet{\mathrm}    {operators}
\DeclareSymbolFontAlphabet{\mathnormal}{letters}
\DeclareSymbolFontAlphabet{\mathcal}   {symbols}
\DeclareMathAlphabet      {\mathbf}{OT1}{cmr}{bx}{n}
\DeclareMathAlphabet      {\mathsf}{OT1}{cmss}{m}{n}
\DeclareMathAlphabet      {\mathit}{OT1}{cmr}{m}{it}
\DeclareMathAlphabet      {\mathtt}{OT1}{cmtt}{m}{n}

This must be completed by the previous declarations

\DeclareSymbolFont{operators}   {OT1}{cmr} {m}{n}
\DeclareSymbolFont{letters}     {OML}{cmm} {m}{it}
\DeclareSymbolFont{symbols}     {OMS}{cmsy}{m}{n}

so that we know precisely to which font correspond each math command:

1. \mathrm is the normal upright Roman font

2. \mathnormal is the normal math italic font: $\mathnormal{a}$ and $a$ give the same result

3. \mathcal is the special calligraphic font for uppercase letters only

4. \mathbf gives upright Roman boldface letters

5. \mathsf gives upright sans serif letters

6. \mathit gives text italic letters: $different\ne\mathit{different}$

7. \mathtt gives upright letters from the typewriter type font

One should notice that the argument to each of those commands is typeset in math mode, so spaces are ignored and hyphens become minus signs. Using those commands with arguments not consisting only of normal letters can give unexpected (and sometimes bizarre) results.

No package has to be loaded for being able to use those alphabets.

AMSfonts

With the amsfonts package, which is loaded automatically by amssymb one has access also to

1. \mathfrak for Fraktur (aka Gothic) letters, upper and lower case

2. \mathbb for "blackboard bold" uppercase letters

RSFS

The package mathrsfs makes available the "Ralph Smith's Formal Script" font as a math alphabet, with the command \mathrsfs, while calrsfs will do the same but also turning \mathcal to choose the RSFS font (only uppercase letters).

Zapf's Euler Script

The package euscript will load a different calligraphic alphabet, Euler Script, designed by Hermann Zapf. Depending on the options, it will provide \mathscr (to go along with the original \mathcal) or change \mathcal to use Zapf's font:

\usepackage{eucal}
\usepackage[mathcal]{eucal}
\usepackage[mathscr]{eucal}

The first two calls are equivalent and \mathcal will use Euler Script; the last call will provide \mathscr for Euler Script and leave \mathcal as is.

_{In my opinion only one type of calligraphic letters should be used, be it Knuth's, RSFS or Euler Script.}

I believe your question is of a (deliberately) introductory or basic nature. Apologies if this is not your intention. A preliminary remark: Most of the text snippet you show is actually set in text mode (italics and bold-upright, resp.), and I won't have anything to say about text-mode alphabets.

• Math italics: In many math texts since (at least?!) the middle or late 18th century (Euler's influence?), variables and letter-like symbols are typeset by convention in a slanted or, rather, italicized style. In TeX's math mode, letters are automatically set in math italics unless one explicitly chooses a different style. A very importance difference between math italics and ordinary italics is that the latter obey kerning and ligature rules that are of no relevance (and are, in fact, undesirable) for math. This is especially noticeable when using letters such as f which have (in italics mode) both ascenders and descenders.

  • In TeX's Computer Modern font family, lowercase greek symbols are also set in slanted mode in math -- I suppose in order to harmonize nicely with any nearby italicized latin letters.

  • When using LaTeX, \mathit{} instructs TeX to typeset the argument in text-mode italics. (In contrast, in Plain TeX the instruction \mit serves to set its argument in math italics.) In the argument of \mathit{}, spaces are ignored. To generate a snippet of text that contains spaces in math mode, use the command \textit{} instead.

• Math roman: In math mode, again by convention, abbreviations of operators such as sin, cos, tan, ln, and many others, are set in upright or "roman" style to distinguish them from ordinary symbols and letters (which are set in italics, see above). Matth roman and text roman generally look very similar but can differ when ligatures and kerning rules are involved. When in math mode, \mathrm{} instructs TeX to typeset the argument in math roman; as with \mathit, \mathrm ignores spaces. (Use \textrm to generate upright text with spaces.)

• Bold versions of mathrm and mathit: To further help the visual definition of symbols, it's possible to embolden letters. This is often (but not always!) done for vectors. In the Computer Modern math font world, bold letters generated via \mathbf are available only in upright or "Roman" style; other math font families, though, also offer bold-italics for math.

• Math sans-serif and Math typewriter: Use of sans-serif fonts in math settings isn't commonly done, but it can be used to set off visually certain entities that have special meanings. In your example, gen, enc, and dec are typeset with \mathsf{}, while Combine and Reveal are typeset with \mathtt.

• Special math alphabets: In addition to the preceding four math alphabets, there are also special alphabets -- such as "Math Blackboard Bold" (invoked with \mathbb{}, "Math Caligraphic" (invoked with \mathcal{}), and "Math Fraktur" (invoked with mathfrak). They are frequently used for sets (e.g., \mathbb{R} and \mathbb{C} for the sets of real and complex numbers) and other entities the author wants to distinguish visually from the surrounding material. In the basic TeX distributions with the Computer Modern fonts, there are only uppercase letters for the math blackboard-bold, caligraphic, and Fraktur alphabets; however, there are packages that provide lowercase letters for these math alphabets as well.

  • In LaTeX when using the Computer Modern fonts, it's usually necessary to load one or more specialized packages, such as amsfonts, in order to access some of the special math alphabets.

• Unlike what can be done with text-mode alphabets (to be precise, since the advent of LaTeX2e in 1994 and its "new font selection system"), math-mode alphabets can't be combined, at least not in "basic" LaTeX when using CM fonts. E.g., the commands \mathcal{\mathbb{}} and \mathbf{\mathsf{}} won't work as one might expect (or wish?).

OK, this is admittedly a very basic introduction to math alphabets in TeX.