How to correctly use newtxtext and newtxmath with luatex

It is better to use OTF fonts with luatex, perhaps:

\documentclass[12pt]{article}%
\usepackage{amsmath}
\usepackage{unicode-math}
\setmainfont{TeX Gyre Termes}
\setmathfont{TeX Gyre Termes Math}
\begin{document}

This is my equation
$\mathbf{\dot{x}}=A x \mathbf{x}$

although perhaps better
$\dot{\symbf{x}}=A x \symbf{x}$

or
$\dot{\symbfit{x}}=A x \symbfit{x}$
\end{document}

The package newtxtext won't work along with fontspec.

\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{fontspec}
\usepackage{newtxmath}

\setmainfont{Tempora} % or whatever Times clone you prefer

\begin{document}

This is my equation
$\dot{\mathbf{x}}=A x \mathbf{x}$

\end{document}

Note that also the pdflatex code

\documentclass[12pt]{article}
\usepackage{amsmath}
\usepackage{newtxtext,newtxmath}

\begin{document}

This is my equation
$\mathbf{\dot{x}}=A x \mathbf{x}$

\end{document}

would fail to produce the dot. Whether this is a bug of newtxmath is disputable. I believe that the correct syntax is

\dot{\mathbf{x}}

Although I agree with David Carlisle completely, the literal answer to the question you asked—how to use the newtxtext and newtxmath packages specifically—is this:

\documentclass[12pt]{article}%
\usepackage{amsmath}
\usepackage{newtxtext,newtxmath}
\usepackage[no-math]{fontspec}

\setmainfont{TeX Gyre Termes}[NFSSFamily = ntxtlf]

\begin{document}

This is my equation:
$\dot{\mathbf{x}} = A x \mathbf{x} = \boldsymbol{\dot{\mathrm{x}}}$

\end{document}

You should never do that, but you might realistically want to use fontspec with a legacy math package, and drop only newtxtext. If you really, truly, need to mix Unicode with legacy text encodings, the package for that is luainputenc. You might run into problems with things like capital Greek letters, which many legacy math packages load from the OT1 encoding.

For practical real-world use, you’re better off switching to unicode-math.