Quaternions and 4-vectors

The object you're talking about is called, in mathematics, a Clifford algebra. The case when the algebra is over the complex field in general has a significantly different structure from the case when the algebra is over the real field, which is important in Physics. In Physics, in the specific case of 4 dimensions, using the Minkowski metric as you have in your Question, and over the complex field, the algebra is called the Dirac algebra. Once you have the name Clifford algebra, you can look them up in Google, where the first entry is, unsurprisingly, Wikipedia, http://en.wikipedia.org/wiki/Clifford_algebra, which gives you a reasonable flavor of the abstract construction methods that mathematicians prefer. The John Baez page that is linked to from the Wikipedia page is well worth reading (if you spent a year learning everything that John Baez has posted over the years, almost always with unusual clarity and engagingly, you would know most of the mathematics that might be useful for Physics).

It's not so much that the Clifford algebras are funny. Their quadratic construction is interrelated, often closely, with many other constructions in mathematics.

There are people who are enthusiastic about Clifford algebras, sometimes very or too much so, and a lot of ink has been spilled (Joel Rice's and Luboš Motl's Answers are rather inadequate to the literature, except that I think they chose to interpret your Question narrowly where I've addressed what your construction has led to in Mathematics more widely), but there are many other fish in the sea to admire.

EDIT: Particularly in light of Marek's comments below, it should be said that I interpreted Isaac's Question generously. There is a somewhat glaring mistake in the OP that is pointed out by Luboš (which I hope you see, Isaac). Nonetheless there is a type of construction that is closely related to what I chose to take to be the idea of the OP, Clifford algebras.

Isaac, this is how I think your derivation ought to go, if we just use quaternions, taking $q=t+ix+jy+kz$, $$q^2=(t+ix+jy+kz)(t+ix+jy+kz)=t^2-x^2-y^2-z^2+2t(ix+jy+kz).$$ The $xy,yz,zx$ terms cancel nicely, but the $tx,ty,tz$ terms don't, unless we do as Luboš did and introduce the conjugate $\overline{q}=t-ix-jy-kz$. This, however, doesn't do what I take you to be trying to do. So, instead, we introduce a fourth object, $\gamma^0$, for which $(\gamma^0)^2=+1$, and which anti-commutes with $i$,$j$, and $k$. Then the square of $\gamma^0t+ix+jy+kz$ is $t^2-x^2-y^2-z^2$. The algebra this generates, however, is more than just the quaternions, it's the Clifford algebra $C(1,3)$.

EDIT(2): Hi, Isaac. I've thought about this way too much overnight. I think now that I was mistaken, you didn't make a mistake. I think you intended your expression $(a,b,c,d)^2$ to mean the positive-definite inner product $a^2+b^2+c^2+d^2$. With this reading, however, we see three distinct structures, the positive-definite inner product, the quaternions, and the Minkowski space inner product that emerges from using the first two together. Part of what made me want to introduce a different construction is that in yours the use of the quaternions is redundant, because you'd get the same result that you found remarkable if you just used $(a,ib,ic,id)^2$ (as Luboš also mentioned). Even the positive-definite inner product is redundant, insofar as what we're really interested in is just the Minkowski space inner product. Also, of course, I know something that looks similar and that has been mathematically productive for over a century, and that can be constructed using just the idea of a non-commutative algebra and the Minkowski space inner product.

To continue the above, we can write $\gamma^1=i$, $\gamma^2=j$, $\gamma^3=k$ for the quaternionic basis elements, together with the basis element $\gamma^0$, then we can define the algebra by the products of basis elements of the algebra, $\gamma^\mu\gamma^\nu+\gamma^\nu\gamma^\mu=2g^{\mu\nu}$. Alternatively, for any vector $u=(t,x,y,z)$ we can write $\gamma(u)=\gamma^0u_0+\gamma^1u_1+\gamma^2u_2+\gamma^3u_3$, then we can define the algebra by the product for arbitrary 4-vectors, $\gamma(u)\gamma(v)+\gamma(v)\gamma(u)=2(u,v)$, where $(u,v)$ is the Minkowski space inner product. Hence, we have $[\gamma(u)]^2=(u,u)$. Now everything is getting, to my eye, and hopefully to yours, rather neat and tidy, and nicely in line with the conventional formalism.

It's just funny. Note that your equation doesn't actually use any single general quaternion. You only use the $i,j,k$ imaginary units in an ad hoc way to get three minus signs whenever you need them.

If you were using an actual quaternion $$ q = t + xi + yj + zk,$$ then the only semi-natural real bilinear invariant you may construct out of it is $$ q\bar q = (t + xi + yj + zk ) ( t - xi - yj - zk) = t^2+ x^2 +y^2 +z^2 $$ so the 4 real components in a quaternion still have the Euclidean, rather than Minkowskian, signature. But even for a 4-dimensional Euclidean space, the quaternions are actually just a game because we haven't really used the main nontrivial structure of the quaternions, their multiplication, in any nontrivial way. Quaternions are not genuine quaternions if you never use the relations $ij=-ji = k$ and its cyclic permutations - and we haven't used them above. We only used the fact that $i,j$ etc. anticommute with each other, but we didn't really care what their product is.

Because we haven't really used those relations, we haven't used full quaternions - except as a meaningless bookkeeping device. In the same way, one may organize 8 real numbers under the umbrella of a single "octonion" except that if the complicated and cool octonion multiplication table - with the $G_2$ automorphism group - is never employed, it's clear that the "octonion" interpretation was just a game to give a name to a collection of 8 numbers. But not every collection of 4 or 8 numbers deserves to be called "quaternion" and "octonion", even though, of course, one may get the individual components out of the "quaternion" and "octonion", too.

In the very same way, a general pair of two real numbers is simply not a complex number. By its very essence, a complex number must act as one number - so there must be a notion of holomorphy required somewhere or everywhere in the formalism - rather than two numbers. The references linked in the other answers don't understand the purpose and relevance of all those mathematical structures, so they lead to incorrect answers to the fundamental question whether the trick is real or just a fun. The right answer is that it is just a fun, and your fun even used a wrong signature that differs from a somewhat more natural fun.

Cornelius Lanczos has a chapter on quaternions and special relativity in his "The Variational Principles of Mechanics". So, is has been used. But it seems more straightforward to consider the multivector algebra of spacetime so t,x,y,z really are on the same footing.