(unicode-math or newtxmath) + \newcommand{\nmodels}{\not\models} + $\not\models$ = stall

The definition of \not is getting too clever by half by looking up whether \nmodels or \notmodels already exist.

It works if you change the name:

\documentclass{standalone}
\usepackage{iftex}

\iftutex
  \usepackage{unicode-math}
\else
  \usepackage{newtxmath}
\fi

%\newcommand{\aintmodel}%
  %{\mathrel{\ooalign{\(\models\)\cr\hidewidth\(\mathslash\)\hidewidth}}}

\newcommand\aintmodel{\not\models}

\begin{document}
\( \aintmodel \)
\end{document}

This lookup gets performed whenever the symbol is used, and the definition of \not isn’t fully-expandable, so you can’t define the symbol with \let or \edef and sidestep this.

You could, however, create the symbol with \ooalign instead. (Use / with newtxmath instead of \mathslash.)

Clearly, unicode-math now needs to search for the prefix aint.

Update

You can also call \not without using the name \models, preventing the look-up. For example,

\documentclass{standalone}
\usepackage{iftex}

\usepackage{unicode-math}


\newcommand\nmodels{\mathrel{\not\symbol{"22A7}}}

\begin{document}
\( \nmodels \)
\end{document}

The current implementation of \not does essentially the same in pdflatex as in (Xe|Lua)LaTeX with unicode-math.

More precisely, \not takes the next token, say \foo and first checks whether \notfoo is defined; if the test succeeds, \notfoo is used. Otherwise the next check is whether \nfoo is defined. Again, if this test succeeds, \nfoo is used. Otherwise LaTeX does \n@tch@r\foo.

Now you see why an infinite loop is started: \nmodels does \not\models; since \nmodels is defined, it is used, which does \not\models…

You can avoid the issue by applying \not to \relax (hoping that neither \notrelax nor \nrelax are defined, which is unlikely).

\documentclass{standalone}
\usepackage{iftex}
\iftutex
  \usepackage{unicode-math}
\else
  \usepackage{newtxmath}
\fi

\newcommand{\nmodels}{\not\relax\models}%

\begin{document}

\(\nmodels\)

\(\not\models\)

\end{document}

Anyway, this is not a particularly good way to negate \models. Compare with the following.

\documentclass[border=4]{standalone}
\usepackage{iftex}
\iftutex
  \usepackage{unicode-math}
\else
  \usepackage{newtxmath}
\fi
\usepackage{centernot}

\newcommand{\nmodels}{\centernot\models}%

\begin{document}

\(\nmodels\)

\(\not\models\)

\end{document}