Notation/definition problem for commutative binary operation
I don't really understand the question, but the structure you've discovered is the unique ITSQ on three elements (well-done!).
In detail:
An idempotent totally-symmetric quasigroup (ITSQ) is a set $X$ together with a binary operation $\#$ such that the following hold:
- $x\#y = y \#x$
- $(x\#y)\#y=x$
- $x\#x=x$.
Now let $A = \{a,b,c\}$ and $\#$ be as defined in your question. It's clear that $(A,\#)$ is an ITSQ. I claim that:
Proposition. If $f : A^2 \rightarrow A$ has the property that $(A,f)$ is an ITSQ, then $f=\#$.
Proof. We know that $$f(a,a) = a \qquad f(b,b) = b \qquad f(c,c) = c$$
If we can show that $f(a,b) =c,$ then by symmetry, this completes the proof.
Assume toward a contradiction that $f(a,b) \neq c$. Then we get two cases:
Case 0. $f(a,b) = a$.
Case 1. $f(a,b) = b$.
Case 1: Suppose $f(a,b)=b$. Then $f(f(a,b),b)=f(b,b)$. So $a=b$, a contradiction.
Case 0: Similar.
I just came across this question from last year. If you're still interested, here's a simple solution:
Use the set $\mathbb{Z}_3$ of integers modulo $3$ as your three-element set, and define $$x\,\#\,y=2(x+y) \pmod{3}.$$
By the way, if you liked @BarryCipra's idea of using the cube roots of unity as your three-element set, you can define $x\,\#\,y$ to be $x^2 y^2$ or, equivalently, $\dfrac1{xy}.$ This is isomorphic to the solution using $\mathbb{Z}_3$ in the first part of this answer.
It's somewhat unclear what your criterion is for an acceptable definition, but if you want something that "looks" mathematical, you can identify your three-member set with the cube roots of unity, $\{1,e^{2\pi i/3},e^{-2\pi i/3}\}=\{1,{-1+\sqrt{-3}\over2},{-1-\sqrt{-3}\over2}\}$ and then, using complex conjugation, define
$$x\#y= \begin{cases} x\quad\text{if }x\overline y=1\\\\ \overline{xy}\quad\text{if }x\overline y\not=1 \end{cases}$$
If you like, you can let $\phi$ be an arbitrary bijection between your set $\{a,b,c\}$ and the cube roots of unity, in which case the definition is
$$x\#y= \begin{cases} x\quad\text{if }\phi(x)\overline{\phi(y)}=1\\\\ \phi^{-1}(\overline{\phi(x)\phi(y)}\quad\text{if }\phi(x)\overline{\phi(y)}\not=1 \end{cases}$$
(There is a tacit theorem here that the definition truly is a binary operation, and is independent of the choice of $\phi$.)
Added later: Here is a math-y alternative that avoids breaking into cases: Identify your three-member set with the unit vectors $(1,0,0)$, $(0,1,0)$ and $(0,0,1)$ and then, using the dot and cross product in $\mathbb{R}^3$, define
$$u\#v=(u\cdot v)u+((1,1,1)\cdot(u\times v))u\times v$$
The main trick here is that $u\times v=0$ if $u=v$ and $u\cdot v=0$ is $u\not= v$. The $(1,1,1)\cdot(u\times v)$ is to get the sign right when $u\not=v$.