Hyperref: Changing the linkcolor locally in the TOC

\hypersetup{linkbordercolor=black} before \tableofcontents changes the link colour to black (also for links to Figures and Tables occurring in the TOC). Enclosing \hypersetup and \tableofcontents in a group (either by enclosing in curly braces {...}or by using \begingroup\hypersetup...\tableofcontents\endgroup) keeps the change local. (Using "plain colour names" like black requires the xcolorpackage.)

If the colorlinks=true option of the hyperref package is used, \hypersetup{linkcolor=black} must be used instead of \hypersetup{linkbordercolor=black}.

MWE:

\documentclass{article}
\usepackage{xcolor}
\usepackage{hyperref}
\begin{document}
{\hypersetup{linkbordercolor=black}
% or \hypersetup{linkcolor=black}, if the colorlinks=true option of hyperref is used
\tableofcontents
}
% \hypersetup{linkbordercolor=red} 
% is not necessary because the colour is only changed in the group
\section{First Section starting at page \pageref{s1}\label{s1}}

\ref{s1}

\end{document}

Because you are using "black" (and assuming your text is written in black) also

{\hypersetup{hidelinks}
\tableofcontents
}

would be possible, which would - well, hide the links in the TOC.