Monic polynomial with integer coefficients with roots on unit circle, not roots of unity?
There exist irreducible monic polynomials such that all their roots apart from two lie on the unique circle (and are not roots of unity). Such polynomials can be chosen among Salem polynomials and they exit in arbitrary high degree. By definition a Salem polynomial $S(x)\in \mathbb Z[x]$ is a monic irreducible reciprocal polynomial with exactly two roots off the unit circle, both real and positive. Of course non of the roots of these polynomials are roots of unity, since these polynomials are irreducible.
See for example theorem 1.6 in the article
Automorphisms of even unimodular lattices and unramified Salem numbers of Gross and Mcmullen:
http://www.math.harvard.edu/~ctm/home/text/papers/unim/unim.pdf
Theorem. For any odd integer $n\ge 3$ there exist infinitely many unramified Salem polynomials of degree $2n$.
For a class of concrete examples with at least asymptotically more than $n/2$ zeros on the unit circle, the Fekete polynomials, which were just mentioned recently by Franz Lemmermeyer at this question on class numbers, might be fruitful in this regard.
Defining them by $$ f_p(x)=\sum_{a=0}^{p-1}\left(\frac{a}{p}\right)x^a, $$
it seems that
$$ \frac{f_p(x)}{x(x-1)} $$ when $p\equiv 3 \bmod 4$
and $$ \frac{f_p(x)}{x(x-1)^2(x+1)} $$
when $p \equiv 1 \bmod 4$ are thought to be irreducible. (See another question here.) (I'd appreciate being corrected if I'm wrong on this.)
Now it has been proven in
B. Conrey, A. Granville, B. Poonen, K. Soundararajan, Zeros of Fekete Polynomials
that asymptotically more than half of the zeros lie on the unit circle.
Just a couple of minor top-ups to Dmitri's nice answer.
- For each even $n\ge2$ the polynomial $p_n(x)=x^n-x^{n-1}-\dots-x+1$ is a Salem polynomial.
- It is not known whether for any $\delta>0$ there exists a Salem polynomial such that $\theta<1+\delta$ for its largest root $\theta$ (which is a Salem number). Lehmer's conjecture suggests that the answer is no.