Determinant of a $3\times3$ magic square
I don't have an explanation, but here is an outline of a proof (I've checked all the details myself, but it's laborious to write up correctly) that what you claim to happen actually does happen.
Result: Let $M$ be a 3x3 integer matrix whose columns, rows, diagonal, and anti-diagonal each total 15. The following are equivalent:
- The entries of $M$ are distinct and positive;
- The determinant of $M$ is $\pm 360$;
- Calling $h,i$ the (3,2) and (3,3) entries, $$(h,i)\in \{(1,6), (1,8), (3,4), (3,8), (7,2), (7,6), (9,2), (9,4)\}.$$
The proof is just writing down the equations for the sums, rows, etc., to be 15 and solving, and then writing down the formula for the determinant and simplifying (it turns out $det = 45(h-5)(h+2i-15)$) and solving the resulting diophantine equation.
The explanation may well be just that there aren't very many 3x3 magic squares (basically, it looks like there's just one that we rotate and flip to get eight). For 4x4, the solution space will be 7d instead of 2d, and that is a lot of extra freedom. If you find a simply-described structure in the determinants of those, I'd be surprised.
Kevin has done the magic squares; I'll do the semi-magics. So the rules are, we're using the 9 digits, once each, and all the row sums and all the column sums are equal. Note that this common sum must be 15. Now the only ways to use 1 and get 15 are $1+6+8$ and $1+5+9$. Since exchanging rows doesn't affect the (absolute value of) the determinant, and exchanging columns doesn't, either, and taking the transpose doesn't, and since further these operations don't affect semi-magicity, we may assume the matrix looks like this: $$\pmatrix{1&6&8\cr5&&\cr9&&\cr}$$ Now the only ways to use 9 are $1+5+9$, which we've already used, and $2+4+9$, so the matrix must look like $$\pmatrix{1&6&8\cr5&&\cr9&2&4\cr}\qquad{\rm or}\qquad\pmatrix{1&6&8\cr5&&\cr9&4&2\cr}$$ The first matrix fills out uniquely to $$\pmatrix{1&6&8\cr5&7&3\cr9&2&4\cr}$$ while the second one can't be completed. So, up to transpose, row swaps, and column swaps, there is only one order 3 semi-magic square, hence, only one determinant (up to sign).