Number Spiral Problem

C (gcc),  44  43 bytes

f(x,y,z){z=x>y?x:y;z=z*z-~(z%2?y-x:x-y)-z;}

Try it online!

The spiral has several "arms":

12345
22345
33345
44445
55555

The position \$(x, y)\$ is located on arm \$\max(x, y)\$ (assigned to variable z). Then, the largest number on arm \$n\$ is \$n^2\$, which alternates between being in the bottom left and top right position on the arm. Subtracting \$x\$ from \$y\$ gives the sequence \$-n+1, -n+2, \ldots, -1, 0, 1, \ldots, n-1, n-2\$ moving along arm \$n\$, so we choose the appropriate sign based on the parity of \$n\$, adjust by \$n-1\$ to get a sequence starting at 0, and subtract this value from \$n^2\$.

Thanks to Mr. Xcoder for saving a byte.

Python,  54   50  49 bytes

def f(a,b):M=max(a,b);return(a-b)*(-1)**M+M*M-M+1

-4 bytes thanks to @ChasBrown

-1 bytes thanks to @Shaggy

Try it Online!

First time golfing! I'm more than aware this is not optimal, but whatever.

Essentially runs on the same principle as @Doorknob C code.

MATL, 15 bytes

X>ttq*QwoEqGd*+

Try it online!
Collect and print as a matrix

How?

Edit: Same technique as @Doorknob's answer, just arrived at differently.

The difference between the diagonal elements of the spiral is the arithmetic sequence \$ 0, 2, 4, 6, 8, \ldots \$. Sum of \$ n \$ terms of this is \$ n(n - 1) \$ (by the usual AP formula). This sum, incremented by 1, gives the diagonal element at position \$ (n, n) \$.

Given \$ (x, y) \$, we find the maximum of these two, which is the "layer" of the spiral that this point belongs to. Then, we find the diagonal value of that layer as \$ v = n(n-1) + 1 \$. For even layers, the value at \$ (x, y) \$ is then \$ v + x - y \$, for odd layers \$ v - x + y \$.

X>        % Get the maximum of the input coordinates, say n
ttq*      % Duplicate that and multiply by n-1
Q         % Add 1 to that. This is the diagonal value v at layer n
wo        % Bring the original n on top and check if it's odd (1 or 0)
Eq        % Change 1 or 0 to 1 or -1
Gd        % Push input (x, y) again, get y - x
*         % Multiply by 1 or -1
          % For odd layers, no change. For even layers, y-x becomes x-y
+         % Add that to the diagonal value v
          % Implicit output

---

Alternate 21 byte solution:

Pdt|Gs+ttqq*4/QJb^b*+

Try it online!
Collect and print as a matrix
From the above, we know that the function we want is

$$ f = m * (m - 1) + 1 + (-1)^m * (x - y) $$

where \$ m = max(x, y) \$.

Some basic calculation will show that one expression for max of two numbers is

$$ m = max(x, y) = \frac{x + y + abs(x - y)}{2} $$

Plugging one into another, we find that one alternate form for \$ f \$ is:

$$ f = (x-y)\cdot i^{k} + \frac{1}{4}((k-2)\cdot k) + 1 $$

where \$ k = abs(x-y) + x + y \$.

This is the function the solution implements.