Difference between \scriptsize and \scriptstyle

While it is true that one is to be used in text mode, the other in math mode, also remember, smaller than \scriptsize is \tiny, but a script of a \scriptstyle is a \scriptscriptstyle. Thus, the context of the two terms (\scriptsize and \scriptstyle), while both using the word script, is actually different. One refers to font size, the other to sub- and super-scripting.

We can see that they are fundamentally different in the following MWE, where \scriptstyle does not produce something in \scriptsize if one was already in \scriptsize to begin with. That is to say, the final X is not in \scriptsize, even though \scriptstyle is invoked at the time.

\documentclass{article}
\begin{document}
$X \scriptstyle X$
versus
\scriptsize$X \scriptstyle X$
\end{document}

As to how it is brought about in TeX, know that \scriptstyle is one of the boxes built via \mathchoice (the other three are \displaystyle, \textstyle, and \scriptscriptstyle) whereas none of that process applies to the invocation of \scriptsize which changes the font size characteristics. You could even say that \scriptstyle shares more in common with \displaystyle (in terms of how it is implemented) than with \scriptsize.

\scriptstyle is a tex primitive and gives a way to force switching to the style that is automatically used in subscripts. (There are eight math styles, display, text, script and scriptscript, plus "cramped" versions of those four)

The math setup defines which fonts to use for each math family (default, \mathrm etc) completely independently. It is common but in no way forced to specify smaller fonts in scriptstyle that text style.

Conversely \scriptsize is a latex macro defined in the document class that specifies whatever the document class wants to specify for text at that size. the standard classes such as article specify vertical spaces around math and lists as well as font size so \small for example is

\newcommand\small{%
   \@setfontsize\small\@ixpt{11}%
   \abovedisplayskip 8.5\p@ \@plus3\p@ \@minus4\p@
   \abovedisplayshortskip \z@ \@plus2\p@
   \belowdisplayshortskip 4\p@ \@plus2\p@ \@minus2\p@
   \def\@listi{\leftmargin\leftmargini
               \topsep 4\p@ \@plus2\p@ \@minus2\p@
               \parsep 2\p@ \@plus\p@ \@minus\p@
               \itemsep \parsep}%
   \belowdisplayskip \abovedisplayskip
}

For sizes such as \scriptsize that are less likely to be used for large blocks article does in fact just set the font size, \scriptsize is

\newcommand\scriptsize{\@setfontsize\scriptsize\@viipt\@viiipt}

that is, 7pt font on an 8pt baselineskip,