Import mathcal symbols from txfonts

It's as easy as this:

\documentclass{article}

\DeclareMathAlphabet{\mathcal}{OMS}{ntxsy}{m}{n}   % or txsy
\SetMathAlphabet{\mathcal}{bold}{OMS}{ntxsy}{b}{n} % or txsy

\begin{document}
$x\mathcal{ABCDEF}y$

\boldmath$x\mathcal{ABCDEF}y$
\end{document}

Explanation. The TX/NewTX packages do no change to the basic math font setup, where \mathcal is defined by

\DeclareSymbolFontAlphabet{\mathcal}{symbols}

in order not to waste a math group. For your purpose a new math group is obviously necessary, so we just need to look in newtxmath.sty what's the definition of the symbols math symbol font and we find

178 \DeclareSymbolFont{symbols}{OMS}{ntxsy}{m}{n}
179 \SetSymbolFont{symbols}{bold}{OMS}{ntxsy}{b}{n}

(line numbers added for reference). So it's sufficient to change \DeclareSymbolFont{symbols} with \DeclareMathAlphabet{\mathcal} and \SetSymbolFont{symbols} with \SetMathAlphabet{\mathcal}.

It's better to use the fonts provided by NewTX that have refined metrics with respect to the older TX package and also is actively maintained, contrary to the old package.

The standard definition of \mathcal is in fontmath.ltx:

\DeclareSymbolFontAlphabet{\mathcal}   {symbols}

So, to use a different font, we need to do two things:

• define a new symbol font, with a distinct name (because we don't want everything else to come from the txfonts symbol font);
• redefine \mathcal so that it uses the new font.

So, let's use symbolstx for the new font. Looking for the definition of symbols in txfonts.sty, we find:

\DeclareSymbolFont{symbols}{OMS}{txsy}{m}{n}
\SetSymbolFont{symbols}{bold}{OMS}{txsy}{bx}{n}

So, we just need to adapt these lines to specify symbolstx rather than symbols:

\DeclareSymbolFont{symbolstx}{OMS}{txsy}{m}{n}
\SetSymbolFont{symbolstx}{bold}{OMS}{txsy}{bx}{n}

Then we can define \mathcal accordingly:

\DeclareSymbolFontAlphabet{\mathcal}{symbolstx}

Putting it altogether:

\documentclass{article}

\DeclareSymbolFont{symbolstx}{OMS}{txsy}{m}{n}
\SetSymbolFont{symbolstx}{bold}{OMS}{txsy}{bx}{n}

\DeclareSymbolFontAlphabet{\mathcal}{symbolstx}

\begin{document}

  \[
    \mathcal{ABCDEFGHIJKLMNOPQRSTUVWXYZ}
  \]

\end{document}