Annotating a binary relation

\documentclass{article}
\usepackage{amsmath}
\begin{document}
$a \hat{\mathrel{\mathcal R}} b$

$a \mathrel{\hat{\mathcal R}} b$

$a \mathrel{\hat{\mathrel{\mathcal R}}} b$ Use this one
\end{document}

The hat is placed over the calligraphic R in an “optimal” way, according to the font designer's wishes.

If you want to have a different placement, more central, just add an empty subformula, so the hat is placed over the whole thing and not the single letter.

\documentclass{article}
\usepackage{amsmath}

\begin{document}

$a\mathrel{\hat{\mathcal{R}}}b$

$a\mathrel{\hat{\mathcal{R}{}}}b$

\end{document}

When TeX has to place a math accent, there are two cases.

First case: a single letter/symbol. The accent is placed over the “visual center” of the letter/symbol taking into account the slant of the font from which the letter/symbol is taken. This is how \hat{A} moves the hat right so it sits above the vertex of A.

Second case: a subformula, that is, math mode material enclosed in braces, or a \left-\right pair, or the argument to the accent comamand itself if it doesn't turn out to be a single math symbol; also something like \mathrel{<symbol>} is not considered to be a single math symbol. The accent is placed in the middle of the subformula.

With \hat{\mathcal{R}{}} we’re in the second case, because the argument to \hat isn’t a single letter/symbol, but also contains an empty subformula.

Thus \mathrel{\hat{\mathrel{\mathcal{R}}}} works, but is not really the best solution.