Space before right bracket

I'm not following you with the fraction; in this case I would prefer the slash notation and also the mleftright package that avoids the "spurious" space between "Q" and the parenthesis:

\documentclass{article}
\usepackage{amsmath}
\usepackage{newtxtext,newtxmath}
\usepackage{mleftright}
\begin{document}
\begin{equation}
   BER = Q\mleft(\sqrt{2E_{b}^{\textnormal{RX}}\Big/N_{0}}\,\mright)
\end{equation}
\end{document}

Here's the same with mathptmx instead of newtxmath

and in this case some corrections are needed:

\begin{equation}
   BER = Q\Bigl(\sqrt{2E_{b}^{\textnormal{RX}}\big/N_{0}}\,\Bigr)
\end{equation}

Please always post complete documents showing all packages used, the \dfrac caught me out first time.

There is is mathinner spacing from the \left\right You can use fixed size delimiters \biggl or simpler (but with spacing that differs again) using {} to get mathord spacing,

But actually I misread your question, you ask about the right bracket, yes I agree the spacing is very tight with the default cm or mathptmx setting, just add \, to compensate:

\documentclass{article}
\usepackage{amsmath}
%\usepackage{mathptmx}

\begin{document}

\begin{equation}
   BER = Q\left(\sqrt{\dfrac{2E_{b}^{\text{RX}}}{N_{0}}}\right)
\end{equation}
\begin{equation}
   BER = Q{\left(\sqrt{\dfrac{2E_{b}^{\text{RX}}}{N_{0}}}\right)}
\end{equation}
\begin{equation}
   BER = Q{\left(\sqrt{\dfrac{2E_{b}^{\text{RX}}}{N_{0}}}\,\right)}
\end{equation}

\end{document}

In the case of your equation, I'd say that there are three issues that should be addressed so that its form doesn't draw attention to itself for the wrong reasons, i.e., clumsy typography, rather than for the mathematics:

• the lack of space between the right-hand end of the square root sign and the closing right parenthesis (the issue you identify in your question);

• the excessive space between the opening left parenthesis and the square root sign's opener; and

• a sense that the parentheses are too large and thus visually quite dominant, relative to what's inside the parentheses.

Incidentally, it's not necessary to use \dfrac for your equation; \frac works just fine. The following MWE addresses these issues:

\documentclass{article}
\usepackage{amsmath,mathptmx}
\newcommand\tb\textbackslash
\begin{document}\pagestyle{empty}
\begin{align*}
Q \left(  \sqrt{\frac{2E_b^{\text{RX}}}{N_0}}  \right)
&\quad \text{initial form} \\
Q \left(\!\sqrt{\frac{2E_b^{\text{RX}}}{N_0}}\,\right)
&\quad \text{\texttt{\tb left(\tb !} and 
  \texttt{\tb ,\tb right)}}\\
Q\Biggl(\!\!\sqrt{\frac{2E_b^{\text{RX}}}{N_0}}\,\Biggr)
&\quad \text{\texttt{\tb Biggl(\tb !\tb !} and 
  \texttt{\tb ,\tb Biggr)}}
\end{align*}
\end{document}

Separately, the mathptmx package is known to generate math expressions that can look a bit cramped. If you're interested in perfecting the appearance of your formulas some more, you may want to look into using the MathTime Professional II font package. It's not a free package, unfortunately; however, its "lite" subset is free. :-) The following MWE shows what I would consider to be an even better looking version of your equation:

\documentclass{article}
\usepackage{amsmath,times,mtpro2}
\newcommand\tb\textbackslash
\begin{document}\pagestyle{empty}
\begin{align*}
Q\Biggl(\!\sqrt{\frac{2E_b^{\text{RX}}}{N_0}}\,\Biggr)
&\quad \text{\texttt{\tb Biggl(\tb !} and 
  \texttt{\tb ,\tb Biggr)}}
\end{align*}
\end{document}