Is the partition function of non-conformal theories on a torus modular invariant?

No, a generic theory on a torus is not modular invariant.

A conformal field theory cannot depend on the metric of the space it is defined on but only on its conformal equivalence class. It is a fact that the space of such classes for the torus is given by a complex parameter $\tau$, the (conformal) modulus, defining the lattice we quotient out of $\mathbb{R}^2$ to get the 2-torus: The basic lattice vectors are $(1,0)$ and the vector in the plane corresponding to the complex number $\tau$, i.e. $\tau$ is the "ratio" between the two fundamental circles on the torus. If we did not have conformal invariance of the theory we would not be allowed to identify the lattice given by $(1,0)$ and $\tau$ with the lattice given by $(2,0)$ and $2\tau$ because while related by a rescaling the area of these parallelograms and hence the resulting torus is different, so we would not have only this one parameter $\tau$ describing the relevant torus structure.

The modular group $\mathrm{PSL}(2,\mathbb{Z})$ sends $\tau$ to values that are equivalent in the sense that they define the same lattice after rescaling one of the basis vectors to $(1,0)$ as allowed by conformal invariance, and so the partition functions, as a function of $\tau$, must be modular invariant since $\tau$ related by modular transformation define tori with the same conformal equivalence class.


Let $T = \mathbb{C}/\left( {\mathbb{Z} \oplus \tau \mathbb{Z}} \right)$. This torus is conformally equivalent to any torus $T' = \mathbb{C}/\left( {\mathbb{Z} \oplus \tau '\mathbb{Z}} \right)$ with $\tau ' = \frac{{a\tau + b}}{{c\tau + \operatorname{d} }}$ such that a,b,c,d integral and $\left| {\begin{array}{*{20}{c}} a&b \\ c&\operatorname{d} \end{array}} \right| = 1$.

This is a modular transformation.

We you take a partition function in a 2D CFT like you would see in String Theory, you are integrating over the moduli space of conformally inequivalent Riemann surfaces. So it is important that your Partition function has modular invariance so it does not distinguish between tori which are conformally equivalent.

If you are working with a QFT which is not conformal, then this argument is not really relevant.