Cycles on the torus

Python 2, 87 bytes

f=lambda m,n,z=0,l=[]:z==0if z in l else sum(f(m,n,(z+d)%m%(n*1j),l+[z])for d in(1,1j))

Try it online!

The interesting thing here is using a complex number z to store the coordinate of the current position. We can move up by adding 1 and move right by adding 1j. To my surprise, modulo works on complex numbers in a way that lets us handle the wrapping for each dimension separately: doing %m acts on the real part, and %(n*1j) acts on the imaginary part.


JavaScript (ES6), 67 bytes

This shorter version is derived from a Python 3.8 alternate version found by @xnor. However, this works only for \$m\times n<32\$ in JS.

Takes input as (m)(n).

m=>n=>(g=(k,l)=>l>>k&1?!k:g((k+m)%(m*n),l|=1<<k)+g(k-~k%m-k%m,l))``

Try it online!

To have it work for any input, we could use BigInts for 73 bytes:

m=>n=>(g=(k,l=k)=>l&(b=1n<<k)?!k:g((k+m)%(m*n),l|=b)+g(k-~k%m-k%m,l))(0n)

Try it online!


JavaScript (ES6),  76 73  72 bytes

Takes input as (m)(n).

m=>n=>(g=(x,y)=>g[x+=y*m]?!x:g(-~x%m,y,g[x]=1)+g(x%m,-~y%n)+--g[x])(0,0)

Try it online!

Commented

m => n => (         // m = width; n = height
  g = (             // g is a recursive function taking:
        x, y        //   the current coordinates (x, y) on the torus
      ) =>          //
    g[              // the surrounding object of g is also used for storage
      x += y * m    // turn x into a key for the current coordinates
    ] ?             // if this cell was already visited:
      !x            //   return 1 if we're back to (0, 0), or 0 otherwise
    :               // else:
      g(            //   first recursive call:
        -~x % m,    //     move to the right
        y,          //     leave y unchanged
        g[x] = 1    //     mark the current cell as visited by setting the flag g[x]
      ) +           //   add the result of
      g(            //   a second recursive call:
        x % m,      //     restore x in [0...m-1]
        -~y % n     //     move up
      ) +           //
      --g[x]        //   clear the flag on the current cell
)(0, 0)             // initial call to g with (x, y) = (0, 0)

Jelly, 28 bytes

ạƝ§=1Ȧ
²‘p/’ŒPÇƇḢÐṂ%⁸QƑƇṪÐṂL

A monadic Link accepting a list, [m,n], which yields the count.

TIO-jt1qe1v9 ...although there is little point, it's way too inefficient.
(I can't even run [2,3] locally with 16GB ram)!

How?

Brute force - creates coordinates of a tiled version big enough then filters the power-set of these points to those paths with neighbours only increasing by one in a single direction, then filters to those starting at a minimal coordinate (i.e. the origin) and, at the same time, removes this start coordinate from each. Then uses modulo arithmetic to wrap back to a torus and filters out any containing duplicate coordinates (i.e. those containing intersections) and, finally, filters to those with minimal ending coordinates (i.e. ending back at the origin) and yields the result's length.

ạƝ§=1Ȧ - Link 1: all neighbours differ by 1 in exactly one direction
 Ɲ     - for neighbours:
ạ      -   absolute difference
  §    - sum each
   =1  - equal to one (vectorises)
     Ȧ - any and all? (falsey if empty or contains a falsey value when flattened)

²‘p/’ŒPÇƇḢÐṂ%⁸QƑƇṪÐṂL - Main Link: list of integers, [m,n]
²                     - square (vectorises) -> [m*m, n*n]
 ‘                    - increment (vectorises) -> [m*m+1, n*n+1]
   /                  - reduce with:
  p                   -   Cartesian product
    ’                 - decrement (vectorises) -> all the coordinates of an m*m by n*n grid
                      -                           including [0, 0] and [m*m, n*n] 
     ŒP               - power-set -> all paths going either up OR right at each step, but not
                      -              necessarily by only 1, and
                      -              necessarily both up and right (e.g. [...[1,3],[5,7],[6,2],...])
        Ƈ             - filter keep those for which:
       Ç              -   call last Link (1) as a monad
                      -              ...now all remaining paths do only go in steps
                      -              of one up or one right
          ÐṂ          - filter keep those minimal under:
         Ḣ            -   head - removes the 1st coordinate from each and yields them for the filter
                      -          ...so only those which started at [0,0] but without it
            %⁸        - modulo by the left argument ([m,n]) (vectorises)
                Ƈ     - filter keep those for which:
               Ƒ      -   is invariant when:
              Q       -     de-duplicated
                      -          ...so no repetitions of torus coordinates (and we already removed
                      -          the first [0,0] which must be present exactly twice)
                  ÐṂ  - filter keep those minimal under:
                 Ṫ    -   tail
                      -          ...so only those which ended at [0,0] 
                    L - length