how to select math font in document

Update Aug. 2013: Since posting the initial version of this answer back in Sept. 2011, (at least) two more Unicode math fonts have joined the TeX/LaTeX scene: TeX Gyre Termes Math (to go with TeX Gyre Termes, a Times Roman clone) and Tex Gyre Pagella Math (to go with TeX Gyre Pagella, a Palatino clone). I've edited the example code below to include these two math fonts; the total of math fonts shown is now up to six 1.

---

Update May 2018: Over the past five years or so, several additional Unicode math font families have become available. I've updated the code and screenshots shown below to include the following, additional math fonts: Stix, Stix Two, Libertinus, and DejvaVu. I have deliberately not included Neo Euler as its development status is "abandoned". I also haven't included Minion Math since it is a commercial font that's not available free of charge.

---

As the other answers and comments have already pointed out, it's not easy in pdfLaTeX to switch from one math font group (say, Computer/Latin Modern) to another. The packages I'm familiar with which have matching text and math fonts are the default Computer/Latin Modern group (loaded if no other font group is specified), mathptmx (Times Roman look), and mathpazo (Palatino). There are also the txfonts and pxfonts packages, which give you Times and Palatino, but the hinting and glyph substitution isn't as good as with mathptmx and mathpazo. Within just the last few months, the newtxmath (and newtxtext) packages have come along; they correct virtually all of the shortcomings of the txfonts package and add quite a bit of new functionality as well. I'm not familiar with math fonts designed specifically to go with Charter text fonts.

Even though it's not straightforward (or even possible?!) to switch between entire math font families within the same document if you use pdfLaTeX, it's actually quite easy to do so in XeLaTeX and LuaLaTeX. The following MWE gives a demonstration of ten different matched text and math fonts: Latin Modern, XITS/XITS Math, TeXGyre Termes, TeXGyre Pagella, Palatino/Asana Math, and Cambria Math, as well as (addition 2018-May) Stix, Stix Two, Libertinus, and DejaVu.

See also the web page A Survey of Free Math Fonts for TeX and LaTeX for a comparison of various free math fonts available for (pdf)LaTeX users.

---

% !TeX program = lualatex
\documentclass{article}
\usepackage{xfrac,unicode-math}
\defaultfontfeatures{Scale=MatchLowercase}

%% Load 10 math fonts; 1 of them with and without some "stylistic sets"
\setmathfont{Latin Modern Math}[version=lm]
\setmathfont{XITS Math}[version=xits]
\setmathfont{Stix Math}[version=stix]
\setmathfont{Stix Two Math}[version=stix2]  % https://github.com/stipub/stixfonts/
\setmathfont{Stix Two Math}[StylisticSet={1,2,8}, version=stix2_128] 
\setmathfont{TeX Gyre Termes Math}[version=termes]
\setmathfont{Cambria Math}[version=cambria]
\setmathfont{TeX Gyre Pagella Math}[version=pagella]
\setmathfont{Asana Math}[version=asana]
\setmathfont{Libertinus Math}[version=libertinus]
\setmathfont{TeX Gyre DejaVu Math}[version=dejavu]
%%\setmathfont{Neo Euler}[version=euler] % status: "abandoned"

\setlength\parindent{0pt} % just for this example
\newcommand{\abc}{abcdefghijklmnopqrstuvwxyz}
\newcommand{\ligs}{\ ff fi fl ffi ffl}
\newcommand{\abctextrm}{Text roman:  \abc\ligs}
\newcommand{\abctextit}{Text italic: \ \ \textit{\abc\ligs}}
\newcommand{\abcmathit}{Math italic: \ $\abc$}
\newcommand{\formulas}{%
  $\displaystyle
   \int_{0}^{1}\! x^{2}\,\mathrm{d}x = \sfrac{1}{3} 
   \qquad
   \sum_{k=0}^{\infty} \frac{1}{k^{2}}=\frac{\pi^{2}}{6}
   \qquad 
   \mathcal{ABC}\ \mathfrak{ABC}\ \mathbb{CNQR}$}
\newcommand{\doall}{\abctextrm\par\abctextit\par\abcmathit\par\formulas}

\begin{document}

%% Latin Modern (the default on most systems)
\setmainfont{Latin Modern Roman}
\mathversion{lm}
\section*{Latin Modern Roman, Latin Modern Math}
\doall

%% Various Times Roman clones
\setmainfont{XITS}
\mathversion{xits}
\section*{XITS, XITS Math}
\doall

\setmainfont{STIX}
\mathversion{stix}
\section*{STIX, STIX Math}
\doall

\clearpage
\setmainfont{Stix Two Text}
\mathversion{stix2}
\section*{Stix Two Text, Stix Two Math}
\doall

\setmainfont{Stix Two Text}
\mathversion{stix2_128}
\section*{Stix Two Text, Stix Two Math w/ stylistic sets 1, 2 \& 8}
\doall

\setmainfont{TeX Gyre Termes}
\mathversion{termes}
\section*{Termes, Termes Math}
\doall

\clearpage
%% Palatino clones
\setmainfont{TeX Gyre Pagella}
\mathversion{pagella}
\section*{Pagella, Pagella Math}
\doall

\setmainfont{Palatino nova}[UprightFont=*-Regular,
             BoldFont=*-Bold, ItalicFont=*-Italic]
\mathversion{asana}
\section*{Palatino nova, Asana Math}
\doall

%% Others

\setmainfont[Ligatures={TeX,Common}]{Cambria}
\mathversion{cambria}
\section*{Cambria, Cambria Math}
\doall

\clearpage
\setmainfont{Libertinus Serif}
\mathversion{libertinus}
\section*{Libertinus Serif, Libertinus Math}
\doall

\setmainfont{DejaVu Serif}
\mathversion{dejavu}
\section*{DejaVu Serif, DejaVu Math}
\doall


%%  Neo Euler Math has status "abandoned"
%\setmainfont{Palatino Linotype}
%\mathversion{euler}
%\section*{Palatino Linotype, Neo Euler Math}
%\doall
\end{document}

That's a bit complex in LaTeX. Note that the definitions of math symbols are complex. Math fonts are quite different with text fonts.

You may want to read the document of NFSS: LaTeX2 font selection, section 3; and the documented source code of LaTeX2e: The LaTeX 2 Sources, section 42.

For examples, any math font package is OK. I suggest mathptmx.sty, it is typical and relatively simple. e.g.

\DeclareSymbolFont{letters}{OML}{ztmcm}{m}{it}

This (re)defines the letter math family. And the command

\DeclareSymbolFontAlphabet{\mathnormal}{letters}

in LaTeX2e kernel makes letters family to be default alphabet font.

People usually don't use many font families in one document. For an example of mixed math fonts, see also my previous answer.

It is not only annoying, but also impossilbe to define many math fonts in one document. In TeX, the number of math alphabet fonts is limited to 16. As Ulrike Fischer said, if you just want to show the results of a few math font packages, just produce some graphics and import them into the document. It is much easier.