Arithmetics in LuaLaTeX

With the code you posted, evaluating fact(5) causes the following expression to be evaluated by the Lua interpreter inside LuaTeX:

tex.sprint(5*fact(4))

which is

tex.sprint(
  5*tex.sprint(
    4*fact(3)))

which is

tex.sprint(
  5*tex.sprint(
    4*tex.sprint(
      3*fact(2))))

which is

tex.sprint(
  5*tex.sprint(
    4*tex.sprint(
      3*tex.sprint(
        2*fact(1)))))

which is

tex.sprint(
  5*tex.sprint(
    4*tex.sprint(
      3*tex.sprint(
        2*tex.sprint(
          1*fact(0))))))

which is

tex.sprint(
  5*tex.sprint(
    4*tex.sprint(
      3*tex.sprint(
        2*tex.sprint(
          1*1)))))

The first fact() call to return a value when the Lua interpreter evaluates fact(5) is thus fact(0). According to your definition of fact(), this evaluates to the Lua integer 1. Then 1*1 is evaluated, results in the Lua integer 1, and the next recursion level thus tries to evaluate 2*tex.sprint(1). This is where your code fails, since tex.sprint(1) evaluates to nil and 2*nil isn't a valid arithmetic expression (tex.sprint() prints contents to the DVI or PDF output, but as a Lua function, returns nil). Because of this error, the recursion stops.

You can fix the problem this way:

\documentclass{article}
\usepackage{luacode}

\begin{luacode}
function fact(n)
 if n == 0 then
    return 1
 else
    return n * fact(n - 1)
 end
end

function printfact(n)
  tex.sprint (fact(n))
end

\end{luacode}

\newcommand*{\factorial}[1]{\directlua{printfact(#1)}}

\begin{document}
\factorial{5}
\end{document}

which produces the following output (DVI or PDF):

In this code, the fact() function always returns a Lua number (not nil), while printfact() writes the corresponding number to the TeX output file (DVI or PDF).

Here's my recommendation: Move the tex.sprint directive to the definition of the \factorial macro, and use return, not tex.sprint, to return numbers from the fact function.

Note also that since there are no TeX-special characters (such as #, %, and \) in the setup of the Lua function fact, one can define it via a \directlua call, as is done in the code below. No need, in this case, to load the luacode package.

\documentclass{article}
\directlua{

function fact ( n )
   if n == 0 or n == 1 then
       return 1
   else
       return ( n * fact (n - 1) )
   end
end

}

\newcommand\factorial[1]{\directlua{tex.sprint(fact(#1))}}

\begin{document}
\factorial{5}
\end{document}