Using PTSerif-TLF for Cyrillic with TeX Gyre Pagella

XeLaTeX and LuaLaTeX use a completely different paradigm for fonts, because everything is available in a single output encoding, namely TU. Thus substitutefont does nothing sensible here.

You could use \babelfont as suggested in the other answer, but you will get several spurious warnings: support for fonts in babel is still in development.

With polyglossia:

\documentclass{article}

\usepackage{polyglossia}
\usepackage{amsmath}
\usepackage{unicode-math}

\setmainlanguage{english}
\setotherlanguage{russian}

\setmainfont{TeX Gyre Pagella}
\newfontfamily{\cyrillicfont}{PT Serif}
\setmathfont{TeX Gyre Pagella Math}

\usepackage[math]{blindtext}

\begin{document}

The topologist is \foreignlanguage{russian}{Алекс\'{а}ндров}.

\blindmathpaper

\end{document}

If you are using XeTeX or LuaTex you should use other ways of switching fonts. One nice way is offered by babel with its \babelfont command:

\documentclass{article}

\usepackage[russian,main=english]{babel}
\usepackage{amsmath} % load here to avoid \dddot clash
\babelfont{rm}[Ligatures=TeX]{TeX Gyre Pagella}
\babelfont[russian]{rm}[Ligatures=TeX]{PT Serif}
\usepackage{unicode-math}
\setmathfont{TeX Gyre Pagella Math}

\usepackage[math]{blindtext}

\begin{document}

The topologist is {\Russian Алекс\'{а}ндров}.

\blindmathpaper

\end{document}

Result: