What is the definition of Curl in Mathematica?

The definition used (motivated by exterior calculus) is as follows:

Given a rectangular array $a$ of depth $n$, with dimensions $\{d, ..., d\}$ (so there are $n$ $d$'s) and a list $x = \{x_1, ..., x_d\}$ of variables, then

Curl[a, x] == (-1)^n (n+1) HodgeDual[Grad[a, x], d]

If $a$ has depth $n$, then Grad[a, x] has depth $n+1$, and therefore HodgeDual[Grad[a, x], d] has depth $d-(n+1)$. Clearly then we need $n < d$. Note that $n = 0$ is admitted, i.e. we can take the curl of a scalar function.

In the traditional case of $d=3$ and $n=1$ we have $d-(n+1)=1$ and that's why the curl of a vector is also a vector.

The HodgeDual operation starts by antisymmetrizing its first argument, and hence implicitly we really have something like

Curl[a, x] == (-1)^n (n+1) HodgeDual[Symmetrize[Grad[a, x], Antisymmetric[All]], d]

Finally, one more comment: The definition given assumes that we work with tensors given in components in a Cartesian coordinated and orthonormal basis. If that wasn't the case we would have to insert some additional metric factors. That's handled by the third argument of Curl for a variety of alternative coordinate systems.

I fail to find a reference for the definition used by Curl, but manage to figure out how the Curl is defined.

I'll use Einstein summation convention for simplicity. The defintion used by Curl is summarized as follows:

1. Curl of a scalar / zero-order tensor is defined as

$$\text{curl}\ \phi= \epsilon_{kim}\frac{\partial\phi}{\partial x_m}$$

2. Curl of a vector / first-order tensor is defined as

$$\text{curl}\ \mathbf{v}= \epsilon_{kim}\frac{\partial v_m}{\partial x_i}$$

3. Curl of a matrix / second-order tensor is defined as

$$\text{curl}\ \mathbf{T}=\color{red}{\frac{1}{2}} \epsilon_{kim}\frac{\partial T_{im}}{\partial x_k}$$

Check

ϵ = LeviCivitaTensor[3];

Table[Sum[ϵ[[k, i, m]] D[Sca, Var[[m]]], {m, 3}], {k, 3}, {i, 3}] == 
ϵ.D[Sca, {Var}] == 
TensorContract[ϵ\[TensorProduct]D[Sca, {Var}], {{3, 4}}] == 
Curl[Sca, Var]
(*True*)

Table[Sum[ϵ[[k, i, m]] D[Vec[[m]], Var[[i]]], {m, 3}, {i, 3}], {k, 3}] == 
TensorContract[ϵ\[TensorProduct]D[Vec, {Var}], {{3, 4}, {2, 5}}] == 
Curl[Vec, Var]
(*True*)

1/2 Sum[ϵ[[k, i, m]] D[Ten[[i, m]], Var[[k]]], {k, 3}, {i, 3}, {m, 3}] == 
1/2 TensorContract[ϵ\[TensorProduct]D[Ten, {Var}], {{2, 4}, {3, 5}, {1, 6}}] == 
Curl[Ten, Var]
(*True*)

The second question is simple:

Table[Sum[ϵ[[k, i, m]] D[Ten[[i, j]], Var[[k]]], {k, 3}, {i, 3}], {m, 3}, {j, 3}] == 
TensorContract[ϵ\[TensorProduct]D[Ten, {Var}], {{2, 4}, {1, 6}}] == 
(Curl[#, Var] & /@ Transpose@Ten // Transpose)
(* True *)

I suggest to modify your simple piece of code to make the output more informative. I define a function dim[] which is similar to the Dimensions[] function but is restricted to matrices and arrays. For example, try Dimensions[x+y] which returns {2} but dim[x+y] returns 0.

ClearAll[dim, id];
dim[x : (_List | _StructuredArray)] := Dimensions@x;
dim[x_Div] = $Failed;
dim[x_] = 0;
id[x_] := (Print[dim[x // Expand]]; x // MatrixForm);

Print["Grad"];
id@Grad[Sca, Var]
id@Grad[Vec, Var]
id@Grad[Ten, Var]
Print["Div"];
id@Div[Sca, Var]
id@Div[Vec, Var]
id@Div[Ten, Var]
Print["Curl"];
id@Curl[Sca, Var]
id@Curl[Vec, Var]
id@Curl[Ten, Var]

You wrote

I couldn't find a precise definition of Mathematica for Curl in the documents

The documentation for Curl[] is long and typical of of some of the more obscure Mathematica function documentation. The answer is buried in Properties & Relations where it states:

Curl[f, {x, y, z}] == (-1)^r (r + 1) HodgeDual[ Symmetrize[Grad[f, {x, y, z}], Antisymmetric[All]]]

but HodgeDual[] is even more obscure so I am not surprised that the precise definition is not given. It would be a good idea if it was available to users. So the answer to your first question is that it is still obscure. Even with a definition, one usually tries various examples and examines the results. Sometimes you have to use the function in not obvious ways. In any case, if a function does not do what you want, you can attempt to write your own function that does exactly what you want. You may decide that this is the better option in your situation.

It is good that you seem to have implemented your own version. I suggest an alternative version and a simple test

CurlTen == Transpose@Array[Curl[Ten[[All, #]], Var] &, 3]

which returns True. I hope this helps you in some small way even though I can't answer your first question.