Suppressing the 'Appendices' page

For the benefit of others who may have this problem but not have \appendixpage anywhere in their code, there are (at least?) two ways the Appendix page can be enabled:

1. Via the \appendixpage command, as mentioned above.
   • Remove this line from your code and the Appendix page will disappear.
2. The page option of the appendix package.
   • \usepackage[page]{appendix} WILL include the Appendix page.
   • \usepackage{appendix} will NOT include the Appendix page.

Delete the command \appendixpage.

The fact that you have to remove the line

\appendixpage

is already stated in Ian's answer.

There is also an error in your code. If you want the appendix "Mathematical Appendix" not to be numbered, but you want it to appear in the ToC, you have to issue the command

\addappheadtotoc

just after the line

\chapter*{Mathematical Appendix}

and not before it, otherwise you get a wrong page number in the ToC.

This is how your MWE should be changed:

\documentclass[12pt,openany]{book}
\usepackage{appendix}
\renewcommand\appendixtocname{Mathematical Appendix}

\begin{document}
\frontmatter
\tableofcontents
\mainmatter

\chapter{The first chapter}
Bla bla

\appendix

\chapter*{Mathematical Appendix}
\addappheadtotoc
\renewcommand\thesection{A.\arabic{section}}
Bla bla
\section{Functions}
Bla bla

\end{document} 

Output (ToC):