Undefined font shape with lualatex

Loading lmodern along with fontspec does nothing relevant for text fonts, because all it does in this respect is

\renewcommand{\rmdefault}{lmr}
\renewcommand{\sfdefault}{lmss}
\renewcommand{\ttdefault}{lmtt}

which is done anyway by fontspec. It can make sense to load it nonetheless, because it also sets Latin Modern as the math font.

The warning

LaTeX Font Warning: Font shape `TU/lmr/bx/sc' undefined
(Font)              using `TU/lmr/bx/n' instead on input line 17.

is due to Latin Modern lacking boldface small caps. The last warning is due to cmr lacking a font definition file for TU, so LaTeX does the default substitution in this case, that is, changing the family name to \familydefault, which expands to \rmdefault and so to lmr.

The other three warnings derive from you defining \englishfont in an incorrect way.

\documentclass{memoir}

% Font packages
\usepackage{lmodern} % serif

% Define language
\usepackage{polyglossia}
\setmainlanguage[]{french}
\setotherlanguage[]{english}
% Define english font
\newfontfamily{\englishfont}{lmroman}[
  Ligatures=TeX,
  UprightFont=*10-italic,
  ItalicFont=*10-italic,
  BoldFont=*10-bolditalic,
  BoldItalicFont=*10-bolditalic,
  SmallCapsFont=*caps10-oblique,
]

\begin{document}

\textsc{\fontname\font}

\textsc{\textbf{\fontname\font}}

\textenglish{\textbf{\fontname\font}}

\textenglish{\textit{\fontname\font}}

\textenglish{\textit{\textbf{\fontname\font}}}

\textenglish{\textsc{\fontname\font}}

\end{document}

This will only raise one warning about the missing boldface small caps font.