Is $\mathbb{Z}[\sqrt{15}]$ a UFD?
How to prove that $(3,\sqrt{15})$ is not principal.
Assume $(3,\sqrt{15})$ is principal - that is $(3,\sqrt{15})=(a+b\sqrt{15})$ for some $a,b\in\mathbb Z$.
Write $\alpha=a+b\sqrt{15}$.
What is $N(\alpha)=a^2-15b^2$? It must be a divisor of $N(3)=9$ and $N(\sqrt{15})=15$. We know $N(\alpha)$ cannot be $\pm 1$, or $\alpha$ would be a unit, and $(3,\sqrt{15})$ is not all of $R$.
So you need $N(\alpha)$ to be a divisor of $9$ and $15$ but not $\pm 1$. That means:
$$N(\alpha)=a^2-15b^2=\pm 3$$
Prove this isn't possible. Look modulo $5$.
How to prove $(3,\sqrt{15})^2$ is principal
As you've noted, $I=(3,\sqrt{15})^2=(9,3\sqrt{15},15)$. But if an ideal contains $9$ and $15$, it contains $2\cdot 9-15=3$.
Also, if an ideal of $R$ contains $3$, then it contains $3\sqrt{15}$.
So what is the ideal $I$?
The UFD question
We've shown that $3$ must be irreducible, by showing it has no non-unit divisors.
So $15=3\cdot 5=\sqrt{15}\sqrt{15}$. Can $3$ divide $\sqrt{15}$?
Maybe I've got it all backwards, but I think these questions are best answered in the reverse of the order you have posed them.
- How do I show that $R$ is (not) a UFD?
Given primes $p$ and $q$, if $\mathbb{Z}[\sqrt{pq}]$ is a unique factorization domain, then both $p$ and $q$ split in this domain. That's what happens to 2 and 3 in $\mathbb{Z}[\sqrt{6}]$, for example. If $p = (a - b \sqrt{pq})(a + b \sqrt{pq})$, $q = (c - d \sqrt{pq})(c + d \sqrt{pq})$ and $(a + b \sqrt{pq})(c + d \sqrt{pq}) = \sqrt{pq}$, then $pq = (\sqrt{pq})^2$ represents two incomplete factorizations.
But 3 is irreducible in $\mathbb{Z}[\sqrt{15}]$, as there is no solution to $x^2 \equiv \pm 3 \pmod 5$. Takes just a little more effort to show 5 is likewise irreducible. Then $15 = 3 \times 5 = (\sqrt{15})$ constitutes the presentation of two distinct, complete factorizations, and therefore $\mathbb{Z}[\sqrt{15}]$ is not a unique factorization domain.
- How do I show that $(3, \sqrt{15})^2$ is a principal ideal?
You have already determined that $\langle 3, \sqrt{15} \rangle^2 = \langle 9, 3 \sqrt{15}, 15 \rangle$. Two elements are enough to generate an ideal, which means that one of these three elements is kind of redundant. Since 9 and 15 are both integers of $\mathbb{Z}$ and $\mathbb{Z}$ is a Euclidean domain, we can boil $\langle 9, 15 \rangle$ to $\langle 3 \rangle$, leaving us with $\langle 3, 3 \sqrt{15} \rangle$. But since $\langle 3 \rangle$ absorbs multiplication of 3 by $\sqrt{15}$, then $\langle 3 \sqrt{15} \rangle$ turns out to be entirely redundant.
Congratulations, you have have factored the principal ideal $\langle 3 \rangle$ as the square of the ideal $\langle 3, \sqrt{15} \rangle$. The number 3 is irreducible, but the ideal $\langle 3 \rangle$ is "composite". Which brings us to your first question:
- How do I show that $(3, \sqrt{15})$ is a maximal ideal but not a principal ideal?
If $\langle 3, \sqrt{15} \rangle$ was not maximal, it could either be properly contained in a maximal ideal or it could be all of $\mathbb{Z}[\sqrt{15}]$. The latter is quite easy to disprove: show that $1 \not\in \langle 3, \sqrt{15} \rangle$. The former takes more work, though there probably is some shortcut I'm unaware of.
If it was the case that $\langle 3, \sqrt{15} \rangle$ is properly contained in a different ideal, that ideal would have to properly contain $\langle 3 \rangle$ and $\langle \sqrt{15} \rangle$ without actually being $\langle 3, \sqrt{15} \rangle$. The only ideal that satisfies this criterion is the whole ring.
So $\langle 3, \sqrt{15} \rangle$ is a maximal ideal. If it was a principal ideal, we could find a single number $a \in \mathbb{Z}[\sqrt{15}]$ such that $\langle a \rangle = \langle 3, \sqrt{15} \rangle$. This $\langle a \rangle$ must be such that it contains numbers with norms divisible by 9 and numbers with norms divisible by 15. Which would mean that $a$ itself has a norm of 3 and therefore 3 splits in this ring. But we already saw that the number 3 is in fact irreducible.
$$\mathbb Z[\sqrt{15}]\simeq\mathbb Z[X]/(X^2-15),$$ so $$\mathbb Z[\sqrt{15}]/(3,\sqrt{15})\simeq\frac{\mathbb Z[X]/(X^2-15)}{(3,X,X^2-15)/(X^2-15)}\simeq\mathbb Z[X]/(3,X)\simeq\mathbb Z/3\mathbb Z.$$ Thus $(3,\sqrt{15})$ is a maximal ideal.