What is box tensor product?

Let $R,S$ be $k$-algebras. If $M$ is an $R$-module and $N$ is an $S$-module, then the $k$-module $M|_k \otimes_k N|_k$ carries the structure of an $R \otimes_k S$-module. This is sometimes denoted by $M \boxtimes_k N$.

More generally, if $X \to S$ and $Y \to S$ are morphisms of schemes, $M$ is an $\mathcal{O}_X$-module and $N$ is an $\mathcal{O}_Y$-module, then the external tensor product $M \boxtimes_{\mathcal{O}_S} N$ is by definition the $\mathcal{O}_{X \times_S \, Y}$-module $\mathrm{pr}_X^*(M) \otimes_{\mathcal{O}_{X \times_S \, Y}} \mathrm{pr}_Y^*(N)$.

Depending on the context, $\boxtimes$ can also denote something different.