'Dimension too large' error when \setmathfont has the option Scale=MatchLowercase

Imho the definition of \__um_fontdimen_to_percent:nN is faulty, it should use \dim_to_decimal_in_sp instead of \dim_to_decimal:n

\documentclass{article}
\usepackage{unicode-math}
\ExplSyntaxOn
\cs_set:Nn \__um_fontdimen_to_percent:nN
  {
    \fp_eval:n { \dim_to_decimal_in_sp:n {  \fontdimen #1 #2 } / 100 }
  }
\ExplSyntaxOff
\setmainfont{Georgia}
\setmathfont[Scale=MatchLowercase]{Cambria Math}

\begin{document}
n $n \sqrt[n]{n}$
\end{document}

Update

As of unicode-math v0.8o (2019/03/04), this bug is fixed for normal Scale usage. The update includes a variant of Ulrike Fischer’s answer, and more crucially, a sloppier setup for font dimensions in fam 2 and 3.

The change from ScaleAgain = 1.00001 and ScaleAgain = 0.99999 to ScaleAgain = 1.0001 and ScaleAgain = 0.9999 effectively lowered the upper bound of the set B in Theorem 1 below. The set B now terminates at around k=10000, which is around Scale=0.153. This means, as long as the user requested Scale is above 0.153, then unicode-math would setup font dimensions correctly.

If the requested Scale is below 0.153, then the font dimension problem persists. But it was deemed that normal usage would never produce Scale=0.153 or lower, so we are safe for the most parts. See more detailed analysis here and here.

---

Old answers

Although I agree with Ulrike Fischer that this is a unicode-math bug, I’m afraid that I have to disagree with the claim that “\__um_fontdimen_to_percent:nN is broken”. Indeed, using \dim_to_decimal_in_sp:n is a great improvement, but it is not where the actual problem lies. I shall first present my solution and then try to discuss the root cause.

Note that the following solution is meant to be temporary until unicode-math fixes this in the next release.

---

Solution

For a relatively simple solution, you can use the new feature ScaleAgain to slightly distort the font size (it won’t be visible to the human eyes):

\documentclass{article}
\usepackage{unicode-math}% v0.8n, 2019/02/15
\setmainfont{Georgia}
\setmathfont[Scale=MatchLowercase,ScaleAgain=0.99999]{Cambria Math}
\begin{document}
n $n \sqrt[n]{n}$
\end{document}

In practice, you’ll have to try a range of ScaleAgain near 1 to be able to compile. Again, this shall be fixed in the next release of unicode-math.

---

A Theorem on the overflow behavior of unicode-math v0.8n

For those who are interested in the bizarre overflow behavior, here is a theorem, based on the two fixed ScaleAgain factors in unicode-math v0.8n.

First comes the visualizations on which Scale’s are safe and which may cause problem:

Near Scale=1:

Near Scale=1.2:

Near Scale=1.5:

All green line segments represent the safe Scale factors, while red line segments represent the problematic ones.

Here is the rigorous mathematical description:

In particular, Scale=1.031369386, Scale=1.031374755, Scale=1.031384644 and Scale=1.031390014 all lead to ! Dimension too large. Near Scale=1.03138:

---

Discussions

The x-heights of Georgia and Cambria Math are, respectively, 986/2048 and 956/2048. And Scale=MatchLowercase is correctly converted into Scale=1.03138 by fontspec. Now, if we were to apply the suggested re-definition of \__um_fontdimen_to_percent:nN when using Latin Modern Math, we would be surprise to find out that:

\documentclass{article}
\usepackage{unicode-math}% v0.8n, 2019/02/15
% https://tex.stackexchange.com/a/475802, by Ulrike Fischer:
\ExplSyntaxOn
\cs_set:Nn \__um_fontdimen_to_percent:nN
  {
    \fp_eval:n { \dim_to_decimal_in_sp:n { \fontdimen #1 #2 } / 100 }
  }
\ExplSyntaxOff

\newcommand*\tempscale{1.03138}% Also fails at 1.02, 1.05, 1.07
\setmathfont[Scale=\tempscale]{latinmodern-math.otf}

\begin{document}
n $n \sqrt[n]{n}$
\end{document}

still produces the ! Dimension too large. error.

Even more strangely, if we instead use a scale factor of either 1.03137 or 1.03139, then your Georgia + Cambria Math example compiles successfully, and so does my Latin Modern Math example (with or without the re-definition of \__um_fontdimen_to_percent:nN).

The root cause of the problem is the new feature ScaleAgain from fontspec, which is meant to solve a long standing problem (see Superscript placement using unicode-math with scaling and Scale option doesn't fully work with LuaLaTeX). Oh, and also the fact that TeX is “not good at math”.

To setup the math-related legacy font dimensions correctly, unicode-math must load the same math font three times. But TeX wouldn’t allow the same font to be loaded twice at the same size, so unicode-math has to load the font at slightly different sizes for the 2nd and the 3rd times. These slightly different sizes are obtained by compounding the previous scale factor. This was the main reason that ScaleAgain was introduced in fontspec, so unicode-math can do ScaleAgain=1.00001 for the 2nd time and ScaleAgain=0.99999 for the 3rd time. As my comment pointed out, sometimes unicode-math will fail to distinguish the three sizes due to TeX’s binary arithmetic.

Your example was “unlucky” enough to have Scale=1.03138, which translates to Round( 1.03138 * 2^16 ) = 67593. After ScaleAgain=1.00001, TeX sees 1.03139, which translates to Round( 1.03139 * 2^16 ) = 67593. So TeX thinks that the second family and the first family are the same font, and proceeds to overwrite fontdimen’s under the instruction of unicode-math. This causes the overflow, because the new fontdimen’s are no longer percentages which are less than one, but are now physical lengths which can get quite large.

With \setmathfont[Scale=MatchLowercase,ScaleAgain=0.99999]{Cambria Math}, we are basically trying to prevent TeX from loading the math font at the same size accidentally.