My first luaLaTeX example

(Answering the question rather than voting to close, because I can find no other occurrences of “malformed number near” on this site, so it would be useful to have this question around for anyone else who is likely to make a similar typo.)

The error message

[\directlua]:5: malformed number near '2d'.

tells you where to look: on line 5 within the luacode block, near the string 2d. And this is what your file contains there (part of line 5 replaced with ...):

        tex.print(string.format(
        ’%2d$^{\\circ}$ ...

note the character before the %2d: it's ’ which is U+2019 RIGHT SINGLE QUOTATION MARK, and the Lua interpreter doesn't know what to do with it. Actually it does: it thinks that ’ is the name of a variable, then looks at the % and decides you're trying to take the remainder, but on the right side of the % operator you have 2d which is not a number. (Hence the error message about a malformed number.) If you had ’ = 23 and then string.format(’%4), the latter would correctly give you 3. (In this case even though you haven't defined ’ to be a variable, the syntax-analysis part of Lua cannot give you an error immediately, because you could define a variable with that name later.)

How did ’ get in there? Possibly a bug in your editor; make sure you fix it or it will happen again. Just change ’ to ' in both places, and your code works; congratulations on getting your first LuaLaTeX file working correctly!

---

Aside: I have found it better to put all Lua code in a separate .lua file and in your .tex file have only \directlua{dofile('myluascript.lua')} (or whatever). This way you get better syntax highlighting for your lua code, when editing your file or when showing it on a site like this.

Your code, with the lua code moved out and with @marsupilam's advice of using [[ ... ]] strings:

.tex file:

\documentclass{article}
\directlua{dofile('lualatex-bug.lua')}
\newcommand{\trigtable}{\directlua{trigtable()}}

\begin{document}
    \begin{tabular}{rcccc}
    \hline
    & $x$ & $\sin(x)$ & $\cos(x)$ & $\tan(x)$ \\
    \hline
    \trigtable
    \hline
    \end{tabular}
\end{document}

lualatex-bug.lua:

function trigtable ()
    for t=0, 45, 3 do
        x=math.rad(t)
        tex.print(string.format([[%2d$^{\circ}$ & %1.9f & %1.9f & %1.9f & %1.9f \\]],
                                t, x, math.sin(x), math.cos(x), math.tan(x)))
    end
end

You may be interested in lua long strings to discard all special characters in the string, and have them raw.

This is especially relevant to luaLaTeX, as the escape character in lua is the backslash \.

If you want to pass a command to TeX, the \ has to be doubled, and it may thus be less annoying to type [[\mycmd]] than "\\mycmd", or [[\\]] than "\\\\"

The output

The code

\RequirePackage{luatex85}
\documentclass[12pt,border=2pt]{standalone}
\usepackage{luacode}
\begin{luacode*}
  function trigtable ()
    for t=0, 45, 3 do
      local x=math.rad(t)
      local sf =[[%2d$^{\circ}$ & %1.9f & %1.9f & %1.9f & %1.9f \\]]
      tex.print(string.format(sf, t, x, math.sin(x), math.cos(x), math.tan(x)))
    end
  end
\end{luacode*}

\newcommand{\trigtable}{\luadirect{trigtable()}}

\begin{document}
    \begin{tabular}{rcccc}
    \hline
    & $x$ & $\sin(x)$ & $\cos(x)$ & $\tan(x)$ \\
    \hline
    \trigtable
    \hline
    \end{tabular}
\end{document}