\tikzcdset{arrow style=math font} breaks the rendering of arrows

tikz-cd assumes two things when drawing the arrows:

1. That the line width of the arrow is the same as the width of a fraction rule, and
2. that the arrow goes along the math axis.

Both is not true for every math font. Imho it is not possible from inside luatex to get the correct values. You would have to look at the glyph definition with e.g. font forge. The following is not perfect but shows how one can adjust the values:

\documentclass{article}
\usepackage{unicode-math}
\usepackage{tikz-cd}
\tikzcdset{arrow style=math font}

\begin{document}

    \setmathfont{Libertinus Math}

    $\frac{1}{b}\rightarrow xxx$

    \begin{tikzcd}
        a\arrow[r]&b
    \end{tikzcd}

\tikzcdset{every arrow/.append style={line width=0.52pt}}%smaller    
\pgfmathdeclarefunction*{axis_height}{0}{%
    \begingroup%      
      \pgfmathreturn2.65pt % smaller than the original
    \endgroup}% 
    \begin{tikzcd}
        a\arrow[r]&b
    \end{tikzcd}


\end{document}

Based on the answer by Ulrike Fischer I wrote a small package which stores the values for a few commonly used fonts, namely: Libertinus Math, TeX Gyre Pagella Math, TeX Gyre Schola Math, TeX Gyre Bonum Math, STIX Two Math, Cambria Math. I thought I would share them here in case someone encounters the same problem. Maybe I will also updated the answer in case I add more fonts.

\NeedsTeXFormat{LaTeX2e}
\ProvidesPackage{mycd}

\RequirePackage{tikz-cd}

\DeclareOption*{\PassOptionsToPackage{\CurrentOption}{tikz-cd}}

\newcommand{\tikzcdarrowdimens}[2]{%
    \tikzcdset{every arrow/.append style={line width=#1}}%
    \pgfmathdeclarefunction*{axis_height}{0}{%
        \begingroup%      
        \pgfmathreturn#2%
        \endgroup%
    }%
}

\DeclareOption{Libertinus}{%
    \tikzcdset{arrow style=math font}%
    \tikzcdarrowdimens{0.052em}{0.250em}%
}

\DeclareOption{Pagella}{%
    \tikzcdset{arrow style=math font}%
    \tikzcdarrowdimens{0.060em}{0.250em}%
}

\DeclareOption{Schola}{%
    \tikzcdset{arrow style=math font}%
    \tikzcdarrowdimens{0.070em}{0.260em}%
}

\DeclareOption{Bonum}{%
    \tikzcdset{arrow style=math font}%
    \tikzcdarrowdimens{0.072em}{0.260em}%
}

\DeclareOption{STIX}{%
    \tikzcdset{arrow style=math font}%
    \tikzcdarrowdimens{0.068em}{0.259em}%
}

\DeclareOption{Cambria}{%
    \tikzcdset{arrow style=math font}%
    \tikzcdarrowdimens{0.06494140625em}{0.285888671875em}%
}

\ProcessOptions\relax

Basic usage is as follows:

\documentclass{article}

\usepackage{unicode-math}
\setmathfont{Libertinus Math}

\usepackage[Libertinus]{mycd}

\begin{document}
    \begin{tikzcd}
        a\arrow[r]&b
    \end{tikzcd}
\end{document}