Largest value of determinant

The matrix is the product $AB$, where $$A=\begin{pmatrix}\alpha^2&\alpha&1\\\beta^2&\beta&1\\ \gamma^2&\gamma&1\end{pmatrix}\mbox{ and }B=\begin{pmatrix}1&1&1\\\alpha&\beta&\gamma\\ \alpha^2&\beta^2&\gamma^2\end{pmatrix}.$$ So its determinant is the product of $\det(A)$ and $\det(B)$. The matrix $A$ can be obtained from $B$ by exchanging the first and the third row and then transposing the result. Therefore $\det(A)=-\det(B)$ and the determinant of the given matrix is $-\det(B)^2$. This is a non-positive expression and its maximal value $0$ is attained if and only if $\det(B)=0$.

Now $B$ is a Vandermonde matrix and its determinant is known to be $(\gamma-\beta)(\gamma-\alpha)(\beta-\alpha)$. This determinant is zero and thus the given determinant attains its maximal value if and only if two or more of the numbers $\alpha,\beta,\gamma$ are equal.

Hint: Using the rules of SARRUS we get after simplifying $$- \left( \alpha-\gamma \right) ^{2} \left( \beta-\gamma \right) ^{2} \left( \beta-\alpha \right) ^{2} $$

Being a $3 \times 3$ determinant, it is too small to be attacked with sophisticated techniques, but also too involved to be attacked by brute force. We shall compute it with successive simplifications of its rows and columns. In the following, $R_i$ and $C_j$ will mean the $i$-th row and the $j$-th column, and $[R_i \to aR_i + bR_j]$ means to replace the $i$-th row with the linear combination $aR_i + bR_j$ where $a$ and $b$ are numbers. The computation, then, goes as follows:

$$\begin{align*} & \begin{vmatrix} 3\alpha^2 & \beta^2 + \alpha \beta + \alpha^2 & \gamma^2 + \alpha \gamma + \alpha^2 \\ \alpha^2 + \alpha \beta + \beta^2 & 3\beta^2 & \gamma^2 + \beta \gamma + \beta^2 \\ \alpha^2 + \alpha \gamma + \gamma^2 & \beta^2 + \beta \gamma + \gamma^2 & 3\gamma^2\end{vmatrix} = [R_2 \to R_2 - R_3] = \\[10pt] & \begin{vmatrix} 3\alpha^2 & \beta^2 + \alpha \beta + \alpha^2 & \gamma^2 + \alpha \gamma + \alpha^2 \\ (\beta - \gamma) (\alpha + \beta + \gamma) & (\beta - \gamma) (2\beta + \gamma) & (\beta - \gamma) (\beta + 2\gamma) \\ \alpha^2 + \alpha \gamma + \gamma^2 & \beta^2 + \beta \gamma + \gamma^2 & 3\gamma^2\end{vmatrix} = \\[10pt] (\beta - \gamma) & \begin{vmatrix} 3\alpha^2 & \beta^2 + \alpha \beta + \alpha^2 & \gamma^2 + \alpha \gamma + \alpha^2 \\ \alpha + \beta + \gamma & 2\beta + \gamma & \beta + 2\gamma \\ \alpha^2 + \alpha \gamma + \gamma^2 & \beta^2 + \beta \gamma + \gamma^2 & 3\gamma^2\end{vmatrix} = [R_3 \to R_3 - R_1] = \\[10pt] (\beta - \gamma) & \begin{vmatrix} 3\alpha^2 & \beta^2 + \alpha \beta + \alpha^2 & \gamma^2 + \alpha \gamma + \alpha^2 \\ \alpha + \beta + \gamma & 2\beta + \gamma & \beta + 2\gamma \\ (\gamma - \alpha) (\gamma + 2\alpha) & (\gamma - \alpha) (\alpha + \beta + \gamma) & (\gamma - \alpha) (2\gamma + \alpha) \end{vmatrix} = \\[10pt] (\beta - \gamma) (\gamma - \alpha) & \begin{vmatrix} 3\alpha^2 & \beta^2 + \alpha \beta + \alpha^2 & \gamma^2 + \alpha \gamma + \alpha^2 \\ \alpha + \beta + \gamma & 2\beta + \gamma & \beta + 2\gamma \\ \gamma + 2\alpha & \alpha + \beta + \gamma & 2\gamma + \alpha \end{vmatrix} = [C_2 \to C_2 - C_1] = \\[10pt] (\beta - \gamma) (\gamma - \alpha) & \begin{vmatrix} 3\alpha^2 & (\beta - \alpha) (\beta + 2\alpha) & \gamma^2 + \alpha \gamma + \alpha^2 \\ \alpha + \beta + \gamma & \beta - \alpha & \beta + 2\gamma \\ \gamma + 2\alpha & \beta - \alpha & 2\gamma + \alpha \end{vmatrix} = \\[10pt] (\beta - \gamma) (\gamma - \alpha) (\beta - \alpha) & \begin{vmatrix} 3\alpha^2 & \beta + 2\alpha & \gamma^2 + \alpha \gamma + \alpha^2 \\ \alpha + \beta + \gamma & 1 & \beta + 2\gamma \\ \gamma + 2\alpha & 1 & 2\gamma + \alpha \end{vmatrix} = [C_3 \to C_3 - C_1] \\[10pt] (\beta - \gamma) (\gamma - \alpha) (\beta - \alpha) & \begin{vmatrix} 3\alpha^2 & \beta + 2\alpha & (\gamma - \alpha) (\gamma + 2\alpha) \\ \alpha + \beta + \gamma & 1 & \gamma - \alpha \\ \gamma + 2\alpha & 1 & \gamma - \alpha \end{vmatrix} = \\[10pt] (\beta - \gamma) (\gamma - \alpha)^2 (\beta - \alpha) & \begin{vmatrix} 3\alpha^2 & \beta + 2\alpha & \gamma + 2\alpha \\ \alpha + \beta + \gamma & 1 & 1 \\ \gamma + 2\alpha & 1 & 1 \end{vmatrix} = [C_3 \to C_3 - C_2] \\[10pt] (\beta - \gamma) (\gamma - \alpha)^2 (\beta - \alpha) & \begin{vmatrix} 3\alpha^2 & \beta + 2\alpha & \gamma - \beta \\ \alpha + \beta + \gamma & 1 & 0 \\ \gamma + 2\alpha & 1 & 0 \end{vmatrix} = \\[10pt] - (\beta - \gamma)^2 (\gamma - \alpha)^2 (\beta - \alpha) & \begin{vmatrix} 3\alpha^2 & \beta + 2\alpha & 1 \\ \alpha + \beta + \gamma & 1 & 0 \\ \gamma + 2\alpha & 1 & 0 \end{vmatrix} = \\[10pt] - (\beta - \gamma)^2 (\gamma - \alpha)^2 (\beta - \alpha)^2 \end{align*} $$

where the last $3 \times 3$ determinant has been expanded along the $3$rd column.

Since the determinant is the negative of the square of a product of real numbers, its maximum will be $0$, reached whenever any two of $\alpha, \beta, \gamma \in [-3,10]$ are equal.