What is the difference between unicode-math and mathspec?

(Please note that this answer has been revised to reflect issues raised in the comments.)

unicode-math

The unicode-math and mathspec packages have very different goals. The unicode-math package is designed to map math markup into unicode characters as supplied by real OpenType math fonts such as the Latin Modern Math, STIX, Asana Math. It also allows (as much as makes sense/is possible) Unicode input of math, as well as output so that you can enter unicode math characters in the source and have them be interpreted correctly as math by TeX.

For example, you could create the following document:

% !TEX TS-program = XeLaTeX

\documentclass[12pt]{article}
\usepackage{unicode-math}

\begin{document}

This is some math text entered with math in the source:
\[
∀X [ ∅ ∉ X ⇒ ∃f:X ⟶  ⋃ X\ ∀A ∈ X (f(A) ∈ A ) ]\]

This is some math text entered with regular markup

\[
\forall X [\emptyset \not\in X \Rightarrow \exists f:X \rightarrow  \bigcup X\ 
    \forall A \in X (f(A) \in A ) ]\]
\end{document}

which would produce

The unicode-math package is compatible with both XeTeX and LuaTeX.

mathspec

The mathspec package is not designed to allow you to use unicode characters in your source math input. Instead it is designed to allow you to use open-type fonts within math so that you can, for example, match the text font of your math with the text font in your document.

So for example you can use:

% !TEX TS-program = XeLaTeX

\documentclass[12pt]{article}
\usepackage{mathspec}
\setmainfont{Linux Libertine O}
\setmathfont(Digits,Latin)[Scale=MatchLowercase]{Linux Libertine O}


\begin{document}
This is some text in Libertine \emph{X(f(A))}
\[
\forall X [\emptyset \not\in X \Rightarrow \exists f:X \rightarrow  \bigcup X\ 
    \forall A \in X (f(A) \in A ) ]\]
\end{document}

which will produce:

The mathspec package can only be used with XeTeX; it cannot be used with LuaTeX.

Which should you use

If you want to use one of the available OpenType math fonts, then using unicode-math makes sense.

If you want to match a non-math font with the math font then mathspec can help, but because TeX does not apply kerning between characters from different fonts, using mathspec has its drawbacks, since you may find places in which the switch from the text font to the math font produces bad spacing, as in the example given.

mathspec is a clever attempt to make it possible to use math in XeLaTeX documents, with system fonts for the letters and symbols from standard math fonts.

With unicode-math one can basically use only specially tailored OpenType math fonts such as Latin Modern Math, TeX Gyre Termes Math, TeX Gyre Pagella Math, XITS Math, Asana Math (among the free ones) or Cambria Math and Lucida Bright Math (not free).

mathspec

With this package you can do limited math with the main system font. Just a silly example:

\documentclass{article}
\usepackage{mathspec}
\setmainfont[Ligatures=TeX]{Old Standard}
\setmathsfont(Digits,Latin,Greek){Old Standard}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

The emulated math is not bad, and the package has features for manually correcting the possible bad spacings, which happen because the fonts have really no support for math. For instance, inputting the formula as

$a+b=\int_{\xi}^{\theta} "f(x)\,dx$

would give the better result

The " means ”add some space on both sides of the following letter”.

Digits, Latin and Greek letters can be taken from different fonts.

unicode-math

The unicode-math is compatible both with XeLaTeX and LuaLaTeX. Here's the same example with Lucida Bright:

\documentclass{article}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX,Scale=0.85]{Lucida Bright OT}
\setmathfont[Scale=0.85]{Lucida Bright Math OT}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

Spacing is correct without manual intervention, because the font used for math is a real math font. Here's the same with TeX Gyre Pagella:

\documentclass{article}
\usepackage{unicode-math}
\setmainfont[Ligatures=TeX]{TeX Gyre Pagella}
\setmathfont{TeX Gyre Pagella Math}
\begin{document}
Some text and a formula $a+b=\int_{\xi}^{\theta} f(x)\,dx$.
\end{document}

Unicode input

One can input math with Unicode symbols, but this is not mandatory. It's possible both with mathspec and unicode-math. For instance, the integral with mathspec could be input as

$∫_ξ^θ "f(x)\,dx$

provided that ∫ is made known to the environment, for instance with

\usepackage{newunicodechar}
\newunicodechar{∫}{\int}

Some symbols (in particular letters) are already known.

Nothing particular is needed for inputting the integral as

$∫_ξ^θ f(x)\,dx$

with unicode-math. However the traditional syntax is understood by the package.

unicode-math is mainly concerned about facilitating the use of using Unicode math fonts, namely OpenType math fonts in LaTeX. While mathspec is about allowing to take math alphabets from text fonts in the absence of matching math font.

Unicode defines a large set of math symbols and many fonts include them, but proper math typesetting requires many font parameters to aid the typesetting engine and those have to be provided by the font. Traditionally TeX fonts would provide the needed parameters in the font dimension of TFM files, which meant that non-TFM fonts can not be used to typeset math in XeTeX (and later LuaTeX). When Microsoft started adding TeX-like math typesetting in Word, they extended OpenType with a special MATH table to hold the needed parameters for math typesetting and XeTeX implemented it (and later LuaTeX) and unicode-math was written to provide macro support for using the new fonts. So with unicode-math you get proper math typesetting with proper OpenType math fonts, like TeX Gyre Pagelle Math, but you are limited to those specially prepared fonts.