How can one define an infix operator with an arbitrary unicode character?

You can get the syntax highlighting that you desire by modifying your UnicodeCharacters.tr file (path given by System`Dump`unicodeCharactersTR), though I don't know how advisable this practice is.

For example, adding:

0x20B0      \[PennyOp]      ($penny$)    Infix       155     None        5       5

I can use EscpennyEsc to enter:

---

I am not aware of documentation of the format of this file but as best I can tell the columns are:

1. the Unicode address in hex

2. the FullForm String representation

3. input aliases separated by tabs

4. the use or type of the symbol

5. parsing precedence

6. Left, Right or None -- I assume an associativity control

7. left whitespace padding to place around the character (in StandardForm)

8. right whitespace padding

---

Completing the operator

Additional code is required to turn such a character into a valid operator. Please see these additional questions for the rest of the story:

• How is + as an infix operator associated with Plus?
• How to define a number of infix operators with predefined relative precedences

Instead of using the Notation package, you can achieve the translation by doing the following:

MakeExpression[RowBox[{x_, "⟗", y_}], StandardForm] := 
 MakeExpression[
  RowBox[{"FlatJoin", "[", x, ",", y, "]"}], 
  StandardForm
 ]

This takes care of the input translation. Now it's possible to enter expressions like

1 ⟗ (3 + 4 ⟗ 2)

and have Mathematica understand that it means

FlatJoin[1, 3 + FlatJoin[4, 2]]

If you want FlatJoin to remain undefined and just have the symbolic display, you may also want to define an output format:

FlatJoin /: MakeBoxes[FlatJoin[x_, y_], StandardForm] := 
 RowBox[{"(", MakeBoxes[x, StandardForm], "⟗", 
   MakeBoxes[y, StandardForm], ")"}]

But as soon as you provide a definition for FlatJoin, this output format won't really be needed.

Edit

The above gets rid of the error messages reported in the question. To get syntax highlighting for incomplete expressions, Mathematica needs to know that it's dealing with a binary operator. The easiest way to do that is of course to make use of one of the operators that is already pre-defined, see the list in the documentation; you can select any operator that has no built-in meaning, see this doc page.

The reason why syntax highlighting doesn't work for general (unicode or other) atomic characters in the abbreviated infix notation 1 ⟗ 2 is that they are interpreted as multiplications by a variable of that name.

You can fix the syntax coloring by using a TagBox to customize AutoStyleOptions (the syntax colorer thinks the operator is an undefined symbol):

AddInputAlias["4" -> ParsedBoxWrapper @ TagBox[
    "⟗",
    FlatJoin,
    BaseStyle->{AutoStyleOptions->{System`UndefinedSymbolStyle->{FontColor->GrayLevel[0]}}},
    SyntaxForm->"↔"
    ]
]

InfixNotation[
    ParsedBoxWrapper @ TagBox[
        "⟗",
        FlatJoin,
        BaseStyle->{AutoStyleOptions->{System`UndefinedSymbolStyle->{FontColor->GrayLevel[0]}}},
        SyntaxForm->"↔"
    ],
    FlatJoin
]

Note that this also fixes the error messages you get when defining the notation. Here is a screen shot showing that the operator turns black as desired: