Derivation of the Curl formula in cartesian coordinates.

A simpler approach is via integral theorems. As stated in the question, the special cases for a rectangle in the $xy$ , $yz$ , $zx$ planes are well understood. According to Green's theorem : $$ \begin{cases} \iint_{xy} \left(\frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y}\right) dx\, dy = \oint_{xy} \left( F_x\, dx + F_y\, dy \right) \\ \iint_{yz} \left(\frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z}\right) dy\, dz = \oint_{xy} \left( F_y\, dy + F_z\, dz \right) \\ \iint_{zx} \left(\frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x}\right) dz\, dx = \oint_{xy} \left( F_z\, dz + F_x\, dx \right) \end{cases} $$ But instead of rectangles, we take half rectangles, or better: the triangles $OAB$ , $OBC$ , $OAC$ respectively:

enter image description here

Thanks to Green's theorem we can replace area integrals by line-integrals; mind that they are counter-clockwise. Then it is clear that, irrespective of any further content: $$ \oint_{OAB} + \oint_{OBC} + \oint_{OAC} + \oint_{ABC} = 0 $$ Assuming that the operator rot(ation) is not defined yet in general, this means that we now have an expression for it: $$ 2 \iint_{ABC} \vec{\operatorname{rot}}(\vec{F}) \cdot \vec{n}\, dA = \\ - \iint_{xy} \left(\frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y}\right) dx\, dy - \iint_{yz} \left(\frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z}\right) dy\, dz - \iint_{zx} \left(\frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x}\right) dz\, dx $$ Continuing with infinitesimal volumes / areas and flipping normals on the right hand side, so that they become the components of the normal at the left hand side: $$ \vec{\operatorname{rot}}(\vec{F}) \cdot \vec{n}\, \Delta A = \\ \left(\frac{\partial F_z}{\partial y} - \frac{\partial F_y}{\partial z}\right)\cdot n_x\, \Delta A + \left(\frac{\partial F_x}{\partial z} - \frac{\partial F_z}{\partial x}\right)\cdot n_y\, \Delta A + \left(\frac{\partial F_y}{\partial x} - \frac{\partial F_x}{\partial y}\right)\cdot n_z\, \Delta A $$ Leaving out the infinitesimal area $\,\Delta A\,$ gives us the same answer as found by the OP themselves.
A somewhat neater approach is to calculate mean values and let the area of the (red) triangle go to zero: $$ \vec{\operatorname{rot}}(\vec{F}) \cdot \vec{n} = \lim_{ABC \to 0} \frac{\iint_{ABC} \vec{\operatorname{rot}}(\vec{F}) \cdot \vec{n}\, dA}{\iint_{ABC} dA} $$ Note. I've encountered essentially the same method at several places elsewhere in physics (I think it's with stress and strain). Aanyway, a related subject is : What does shear mean?


It is possible to prove this by taking Green's theorem and applying rotations. So what we do is rotate the whole space so that the normal is in the z direction (or whatever Green's Theorem configuration we like) and then use the rotation information to get an expression for the original space.

We start by applying a rotation around the x and y axis

$$ \left(\begin{array}{c} x' \\ y' \\ z' \end{array}\right) = \left(\begin{array}{ccc} a & b & c \\ d & e & f \\ g & h & i\end{array}\right) \cdot \left(\begin{array}{c} x \\ y \\ z\end{array}\right) $$

This rotates the surface so that its normal at the required point, points upwards. This means,

$$\left(\begin{array}{c} 0 \\ 0 \\ 1 \end{array}\right) = \left(\begin{array}{ccc} a & b & c \\ d & e & f \\ g & h & i\end{array}\right) \cdot \vec{n} $$

and by using the inversion property of rotation matrices,

$$ \vec{n} = \left(\begin{array}{ccc} a & d & g \\ b & e & h \\ c & f & i\end{array}\right) \cdot \left(\begin{array}{c} 0 \\ 0 \\ 1 \end{array}\right) = \left(\begin{array}{c} g \\ h \\ i \end{array}\right) $$

Notice that the normal $\vec{n}$ is the last row of our rotation matrix. Since the first and second row are also unit vectors orthogonal to $\vec{n}$ we can write $\vec{n}$ as their cross product,

$$ \vec{n} = \left|\begin{array}{ccc} \vec{i} & \vec{j} & \vec{k} \\ a & b & c \\ d & e & f \end{array}\right| $$

and hence,

$$ n_x = \left|\begin{array}{cc}b & c \\ e & f \end{array}\right| n_y = \left|\begin{array}{cc}d & f \\ a & c \end{array}\right|, n_z = \left|\begin{array}{cc}a & b \\ d & e \end{array}\right|$$ We also want to rotate our vector field appropriately: $$ \left(\begin{array}{c} F_{x'}(P') \\ F_{y'}(P') \\ F_{z'}(P') \end{array}\right) = \left(\begin{array}{ccc} a & b & c \\ d & e & f \\ g & h & i\end{array}\right) \cdot \left(\begin{array}{c} F_x(P) \\ F_y(P) \\ F_z(P)\end{array}\right) $$

Finally we can derive our expression using simple determinant properties and the linear relationships that we wrote above,

$$\begin{gather}\frac{\partial F_{y'}}{\partial x'} - \frac{\partial F_{x'}}{\partial y'} \end{gather} = \\ \left|\begin{array}{cc} \frac{\partial}{\partial x'} & \frac{\partial}{\partial y'} \\ F_{x'} & F_{y'} \end{array}\right| = \\ \left|\begin{array}{ccc} a\frac{\partial}{\partial x} + b\frac{\partial}{\partial y} + c\frac{\partial}{\partial z} & d\frac{\partial}{\partial x} + e\frac{\partial}{\partial y} + f\frac{\partial}{\partial z} \\ aF_{x} + bF_{y} + cF_{z} & dF_{x} + eF_{y} + fF_{z} \end{array}\right| = \\ \left| \begin{array}{cc} a\frac{\partial}{\partial x} & e\frac{\partial}{\partial y} \\ aF_x & eF_y\end{array}\right| + \left| \begin{array}{cc} a\frac{\partial}{\partial x} & f\frac{\partial}{\partial z} \\ aF_x & fF_z\end{array}\right| + \left| \begin{array}{cc} b\frac{\partial}{\partial y} & d\frac{\partial}{\partial x} \\ bF_y & dF_x\end{array}\right| + \\ \left| \begin{array}{cc} b\frac{\partial}{\partial y} & f\frac{\partial}{\partial z} \\ bF_y & fF_z\end{array}\right| + \left| \begin{array}{cc} c\frac{\partial}{\partial z} & d\frac{\partial}{\partial x} \\ cF_z & dF_x\end{array}\right| + \left| \begin{array}{cc} c\frac{\partial}{\partial z} & e\frac{\partial}{\partial y} \\ cF_z & eF_y\end{array}\right| = \\ (bf-ce)\left| \begin{array}{cc} \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\ F_y & F_z \end{array}\right| + (af-cd)\left| \begin{array}{cc} \frac{\partial}{\partial x} & \frac{\partial}{\partial z} \\ F_x & F_z \end{array}\right| + (ae-db)\left| \begin{array}{cc} \frac{\partial}{\partial x} & \frac{\partial}{\partial y} \\ F_x & F_y \end{array}\right| = \\ \left|\begin{array}{cc}b & c \\ e & f \end{array}\right|\left| \begin{array}{cc} \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\ F_y & F_z \end{array}\right| - \left|\begin{array}{cc}d & f \\ a & c \end{array}\right|\left| \begin{array}{cc} \frac{\partial}{\partial x} & \frac{\partial}{\partial z} \\ F_x & F_z \end{array}\right| + \left|\begin{array}{cc}a & b \\ d & e \end{array}\right|\left| \begin{array}{cc} \frac{\partial}{\partial x} & \frac{\partial}{\partial y} \\ F_x & F_y \end{array}\right| = \\ n_x\left| \begin{array}{cc} \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\ F_y & F_z \end{array}\right| - n_y\left| \begin{array}{cc} \frac{\partial}{\partial x} & \frac{\partial}{\partial z} \\ F_x & F_z \end{array}\right| + n_z\left| \begin{array}{cc} \frac{\partial}{\partial x} & \frac{\partial}{\partial y} \\ F_x & F_y \end{array}\right| = \\ \left| \begin{array}{ccc} \vec{i} & \vec{j} & \vec{k} \\ \frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\ F_x & F_y & F_z \end{array}\right| \cdot \vec{n}$$