Further customize color of hyperref links

The hyperref options can be configured using the \hypersetup command and colors are enabled by colorlinks=true.

A MWE is below. citecolor is set to gray. For the table of contents link colors, linkcolor is set to green before the \tableofcontents command. Similarly for the list of figures, linkcolor is set to red before the \listoffigures command. Later linkcolor is set to blue for the internal links.

I was unable to make the table of contents's sub-section links appear in orange color.

\documentclass{article}
\usepackage{color}
\usepackage[usenames,dvipsnames,svgnames,table]{xcolor}
\usepackage[backref=true,backend=biber,natbib=true,hyperref=true]{biblatex}
\bibliography{refs}

\usepackage{hyperref}
\hypersetup{
     colorlinks   = true,
     citecolor    = gray
}
\begin{document}

\title{Title}
\maketitle
\hypersetup{linkcolor=green}
\tableofcontents
\hypersetup{linkcolor=red}
\listoffigures
\hypersetup{linkcolor=blue}

\section{Table of content's links should be green} \label{s1}
All text, captions, and section headers after the list of figures 
should be black. The citation links should be gray~\cite{cit2}. 
\subsection{Table of content's subsections links should be orange} 
Internal links like this section link \ref{s1} or this figure link 
\ref{f1} should be blue.

\begin{figure}[h]
\caption{List of figures links red}
\label{f1}
\end{figure}
\nocite{*}

\printbibliography
\end{document}

Here is a screenshot of the output:

For the question of changing the link colour of subsections in the table of contents, one can use the tocloft package:

\documentclass{article}
\usepackage{color}
\usepackage[usenames,dvipsnames,svgnames,table]{xcolor}
\usepackage{hyperref}
\hypersetup{
     colorlinks   = true,
     citecolor    = gray
}

\usepackage{tocloft}

\renewcommand{\cftsubsecfont}{\normalfont\hypersetup{linkcolor=orange}}
\renewcommand{\cftsubsecafterpnum}{\hypersetup{linkcolor=green}}

\begin{document}

\title{Title}
\maketitle
\hypersetup{linkcolor=green}
\tableofcontents
\hypersetup{linkcolor=red}
\listoffigures
\hypersetup{linkcolor=blue}

\section{Table of content's links should be green}
\label{s1}
All text, captions, and section headers after the list of figures
should be black. The citation links should be gray.
\subsection{Table of content's subsections links should be orange}
Internal links like this section link \ref{s1} 
or this figure link \ref{f1} should be blue.
\section{Making sure section titles turn back to green}
This should be green again in the ToC.

\begin{figure}[h]
\caption{List of figures links red}
\label{f1}
\end{figure}

\end{document}

It requires a little fiddling to find which commands need to be changed to make the colour what is desired. More information on page 10 of the documentation. To keep the font the same, I had to find the standard definition, p. 30 of the documentation.

The tocloft package also deals with all sort of options to change the list of figures (and list of tables), meaning you may be able to set all your wishes in the preamble using this package.