What is the right space between a math operator and its subscript?

If you use \operatorname{\mathcal{P}}_{\mathrm{ext}}, TeX cannot kern the subscript.

If you just need “ext” as subscript, you can define

\DeclareMathOperator{\Pme}{\mathcal{P}_{ext}}

If the subscript is variable,

\newcommand{\Pm}[1]{\operatorname{\mathcal{P}_{#1}}}

and call it as \Pm{ext}.

Full example:

\documentclass{article}
\usepackage{amsmath}

\DeclareMathOperator{\Pme}{\mathcal{P}_{ext}}

\newcommand{\Pm}[1]{\operatorname{\mathcal{P}_{#1}}}

\begin{document}

$\Pme(H)$

$\Pm{ext}(H)$

\end{document}

If you prefer the _ notation:

\documentclass{article}
\usepackage{amsmath}
\usepackage{xparse}

\NewDocumentCommand{\Pm}{e{_}}{%
  \operatorname{\mathcal{P}\IfValueT{#1}{_{#1}}}%
}

\begin{document}

$\Pm_{ext}(H)$

\end{document}

Note that in any of the above case you just type in ext, because it will be typeset as part of \operatorname; \mathrm is not needed.

\documentclass[fleqn]{article}
\usepackage{amsmath, amssymb}
\DeclareMathOperator{\Pm}{\mathcal{P}}
\showoutput
\begin{document}
\begin{align*}
    &\mathcal{P}_{\mathrm{ext}}(H) \\
    &\operatorname{\mathcal P}_{\mathrm{ext}}(H) \\
    &\Pm_{\mathrm{ext}}(H)\\
    &\mathop{\mathcal{P}_{\mathrm{ext}}}(H) 
\end{align*}


a $\log x$


a ${}\log x$

\end{document}

The specific example makes the answer a bit trickier than it would otherwise have been. I would say that the most logical markup would be \mathop{\mathcal{P}_{\mathrm{ext}}} as you want the subscripted term itself act as a \mathop.

However the reason that you see different spacing here is because you are on the right hand side of an alignment where implicitly the expression is prefixed by {} so you are getting a thin space from the \mathop to separate it from the (invisible) preceding term. So it is like the second \log example I added to the example here/ So why in the vast majority of cases the \mathop spacing for \log is the right thing the extra space here is probably not right and should be removed by \! or by using a form that does not give a \mathop, eg {\Pm}