Label for tick mark interferes with graph

You can define only -1.732 as extra x ticks and 1.732 as a simple x tick, in this case you can define anchor for xticklabels separately and avoid overlap like this

xtick={1.732},
xticklabels={$\sqrt{3}$},
extra x ticks={-1.732},
extra x tick labels={$-\sqrt{3}$},
xticklabel style={font=\tiny, inner sep=0pt,below right},
extra x tick style={font=\tiny, inner sep=0pt,xticklabel style={below left,xshift=-3pt}},

Code

\documentclass{amsart}

\usepackage{tikz}
\usetikzlibrary{calc,intersections}

\usepackage{pgfplots}
\pgfplotsset{compat=1.11}

\usepackage{mathtools,array}

\begin{document}


\begin{tikzpicture}
\begin{axis}[width=5in, axis equal image, axis on top,
    axis lines=middle,
    xmin=-10,xmax=10,
    ymin=-11,ymax=10,
    restrict y to domain=-11:10,
    ytick={\empty},
    xtick={1.732},
    xticklabels={$\sqrt{3}$},
    extra x ticks={-1.732},
    extra x tick labels={$-\sqrt{3}$},
    xticklabel style={font=\tiny, inner sep=0pt,below right},
    extra x tick style={font=\tiny, inner sep=0pt,xticklabel style={below left,xshift=-3pt}},
    xlabel=$x$,ylabel=$y$,
    axis line style={latex-latex},
    axis line style={shorten >=-7.5pt, shorten <=-7.5pt},
    xlabel style={at={(ticklabel* cs:1)}, xshift=7.5pt, anchor=north west},
    ylabel style={at={(ticklabel* cs:1)}, yshift=7.5pt, anchor=south west}
]

\addplot[samples=501,domain=-9:-1.1789,blue] {(x^(2) - 3)/ (x + 1)} node[anchor=east,pos=0.9,font=\footnotesize]{$y=f(x)$};
\addplot[samples=501,domain=-0.72508:9,blue] {(x^(2) - 3)/ (x + 1)};

\addplot [dashed, latex-latex,domain=-10:10] {x - 1} node [pos=0.9, anchor=south, font=\footnotesize, sloped]{$y=x-1$};
\addplot [dashed, latex-latex,domain=-10:10] (-1,x) node [pos=0.1, anchor=south, font=\footnotesize, sloped]{$x=-1$};

\end{axis}
\end{tikzpicture}

\end{document}

Result

I don't know why, but axis on top doesn't seem to affect extra x ticks. However, in this case there is no need to use extra x ticks, just set and modify the standard ticks.

You may want to modify the inner sep, but that fine tuning I'll leave to you.

\documentclass{amsart}
\usepackage{pgfplots}
\pgfplotsset{compat=1.11}

\begin{document}    
\begin{tikzpicture}
\begin{axis}[width=5in, axis equal image,axis on top,
    axis lines=middle,
    xmin=-10,xmax=10,
    ymin=-11,ymax=10,
    restrict y to domain=-11:10,
    ytick={\empty},
    xtick={-1.732, 1.732},
    xticklabels={\hspace*{-5pt}\makebox[0pt][r]{$-$}$\sqrt{3}$, $\sqrt{3}$\hspace*{-5pt}},
    xticklabel style={font=\tiny, circle, fill=white, inner sep=0pt},
    xlabel=$x$,ylabel=$y$,
    axis line style={latex-latex},
    axis line style={shorten >=-7.5pt, shorten <=-7.5pt},
    xlabel style={at={(ticklabel* cs:1)}, xshift=7.5pt, anchor=north west},
    ylabel style={at={(ticklabel* cs:1)}, yshift=7.5pt, anchor=south west}
]

\addplot[samples=501,domain=-9:-1.1789,blue] {(x^(2) - 3)/ (x + 1)} node[anchor=east,pos=0.9,font=\footnotesize]{$y=f(x)$};
\addplot[samples=501,domain=-0.72508:9,blue] {(x^(2) - 3)/ (x + 1)};

\addplot [dashed, latex-latex,domain=-10:10] {x - 1} node [pos=0.9, anchor=south, font=\footnotesize, sloped]{$y=x-1$};
\addplot [dashed, latex-latex,domain=-10:10] (-1,x) node [pos=0.1, anchor=south, font=\footnotesize, sloped]{$x=-1$};

\end{axis}
\end{tikzpicture}

\end{document}