Incompatibility with mathtools and unicode-math in xelatex?

update:

The github issue related to this and opened by @UlrikeFischer was closed some time ago already at unicode-math v0.8e, and currently unicode-math is at v0.8m. So don't use the hack below.

---

---

Update. Here is an attempt at addressing the \newfam issue. We temporarily make innocent \newfam during unicode-math loading (I checked it was used only once; as it is used during patching of mathtools, mathtools must be loaded before unicode-math). Then we patch ourselves the mathtools macro. For this, I use the math family number 255 rather than creating a new one. It is not very probable the document uses 256 math families. One could make this more intrinsic by using \e@mathgroup@top - 1 but I don't know how to subtract 1 in expl3 lingua. Caveat: barely tested.

This addresses only one the two issues. The one of effect of empty range is not handled here.

\documentclass{article}
\usepackage{fontspec}

% *must* be before unicode-math
\usepackage{mathtools} 

\makeatletter
\let\ORInewfam\newfam
\def\newfam #1{\typeout{message from \string\newfam: killing myself}}
\usepackage{unicode-math}
\let\newfam\ORInewfam
\makeatother


\unimathsetup{math-style=ISO,bold-style=ISO,sans-style=italic}
\setmainfont{xits-regular.otf}
\setmathfont{xits-math.otf}
\setmathfont[range={\mathcal,\mathbfcal}]{latinmodern-math.otf}
%\setmathfont[range={}]{xits-math.otf} % Try with and without this line

\ExplSyntaxOn
\makeatletter
\def\MT_cramped_internal:Nn #1#2
     {
      \hbox_set:Nn \l_tmpa_box
       {
        \color@setgroup
        \c_math_toggle_token
        \m@th
        #1
        \dim_zero:N \nulldelimiterspace
        \XeTeXradical \c_two_hundred_fifty_five \c_zero { #2 }
        \c_math_toggle_token
        \color@endgroup
       }
      \box_set_ht:Nn \l_tmpa_box
       {
        \box_ht:N \l_tmpa_box
        - \__um_radical_vgap:N #1
       }
      \box_use_drop:N \l_tmpa_box
     }
\makeatother
\ExplSyntaxOff

\begin{document}

Test
$a+b = \mathcal{C H G}C \mathscr{D}$
$$-\frac{a}{b} \mathcal{O} . $$
\begin{equation}
    \mathrm{d} = \mathcal{G}(\varphi,x,y,z,t) , 
\end{equation}
% The symbol $\mathcal{G}$ is NOT missing.

% The following equation is wrong if I include the line  ``setmathfont[range=\{\}] \{xits-math.otf\}''
\begin{equation}
    \mathup{d} = \symbf{u}(\varphi,x,y,z,t) , 
\end{equation}
% The glyph $\symbf{u}$ is NOT missing.

Example from mathtools manual

\[
\cramped{x^2} \leftrightarrow x^2 \quad
\cramped[\scriptstyle]{x^2} \leftrightarrow {\scriptstyle x^2}
\]

\end{document}

_{code updated 2018/12/28 as per @egreg comment "\box_use_clear:N is deprecated as of 2018-12-31; the right function to use is \box_use_drop:N"}

please don't upvote this until it has received confirmation by competent people. It is true that removing some code from unicode-math-xetex.sty makes the thing work, but it may well be that OP's code is not correct use of unicode-math: as I am not familiar with it I can't judge.

please start upvoting this answer massively as I have grown competent enough to be certain of the origin of the problem (see bottom)

ok, you may stop upvoting and start transferring your votes to Ulrike's answer as she had already opened quite some time ago the ticket https://github.com/wspr/unicode-math/issues/345 mentioning the pertubation caused by naked use of \newfam on math alphabets and thus finding in anticipation which code line in unicode-math ultimately resulted in the disappearing glyphs as here in the OP's question.

Part I

I have identified the source of the problem in file unicode-math-xetex.sty.

Steps to reproduce:

kpsewhich unicode-math-xetex.sty
copy the file to your working repertory
comment out lines #L2040 to #L2091 inclusive

Then your document compiles fine with the expected glyphs with both mathtools and your  \setmathfont[range={}]{xits-math.otf}

Note: the code of unicode-math looks strange with the \AtEndOfPackageFile * { mathtools } nested into another one... but I checked balancing of braces. Thus, lines 2038 and 2039 must remain non-commented.

It could be that the important thing in the commented out code is that it does \newfam \g__um_empty_fam creating a new \mathgroup. This may cause a problem of numbering ?

Not being familiar with unicode-math I can't comment more. But it does look more like a problem in unicode-math than one in mathtools (as far as I can tell.)

extra info: I have checked that \mv@normal has exact same meaning in the two situations, confirming that there may be an offset in numbering of math families on the unicode-math side.

image showing it does work: (the math calligraphic are from XITS not Latin Modern, due to effect of range={})

Part II

Here is a minimal example to demonstrate the bug lies in unicode-math (see the various comments below this answer for more context).

\documentclass{article}

\usepackage{unicode-math}

\newfam\MyFam  % comment this to get correct glyph rather than invisible one

\setmainfont{lmroman10-regular.otf}
\setmathfont[range={cal}]{latinmodern-math.otf}

\begin{document}

$\mathrm{d} = \mathcal{G}$

\showoutput
\end{document}

%%% Local Variables:
%%% TeX-engine: xetex
%%% End:

The use of \newfam before \setmainfont/\setmathfont (the latter using the cal not the deprecated range={\mathcal}) is enough to create the problem: . Without the \newfam we get .

In case some doubt arises because Latin Modern is default use

\setmathfont[range={cal}]{texgyretermes-math.otf}

I have opened a ticket at https://github.com/wspr/unicode-math/issues/368.

And the point is that the single (direct) use of \newfam in unicode-math code is in the line

    \newfam \g__um_empty_fam

which is encountered at the start of its patch of mathtools which in Part I I advised to comment out. If this patch is executed too soon in the preamble a problem arises in using \math.. alphabet macros: the wrong font is used. This may have to do with unicode-math way of patching \use@mathgroup but I have not pursued further because the LaTeX code for math fonts is very complex and whenever I stay a few months without looking at it I forget it entirely and irrevocably, and it would take me quite some effort to re-immerse me into it, not to mention that I would need to read unicode-math code as well.

I don't quite understand why you are using an empty range, but imho your syntax is wrong. You are still using in many place \mathXX commands where \symXX should be used, and in the range command you should use the names for the blocks and not commands. This here works fine for me on texlive 2016:

\documentclass{article}
\usepackage{mathtools}
\usepackage{fontspec}
\usepackage{unicode-math}
\unimathsetup{math-style=ISO,bold-style=ISO,sans-style=italic}
\setmainfont{xits-regular.otf}
\setmathfont{xits-math.otf}
\setmathfont[range={cal,bfcal}]{latinmodern-math.otf}
\setmathfont[range={}]{xits-math.otf} % Try with and without this line
%
\begin{document}
Test
$a+b = \symcal{C H G}C \symscr{D}$
$$-\frac{a}{b} \symcal{O} . $$
\begin{equation}
    \symrm{d} = \symcal G(\varphi,x,y,z,t) ,
\end{equation}
The symbol $\symcal{G}$ is missing.

The following equation is wrong if I include the line  ``setmathfont[range=\{\}] \{xits-math.otf\}''
\begin{equation}
    \symup{d} = \symbf{u}(\varphi,x,y,z,t) ,
\end{equation}
The glyph $\symbf{u}$ is missing.
\end{document}