\str_case:nnF not finding match

You have the syntax for \str_case:nnF wrong; the first argument is the string to look for matches; the second argument should be of the form

{<string-a>}{<code>}
{<string-b>}{<code>}
...

and you have additional braces that are wrong.

\documentclass{article}
\usepackage{mathtools}
\usepackage{xparse}

\ExplSyntaxOn

% Render underscored with some variation of strict
\NewDocumentCommand{\strict}{O{strict} m}
  {
    \str_if_eq:eeTF {\str_item:nn {#1} {-1}} {-}
      {
        \show_strict:nn {{#1}strict} {#2}
      }
      {
        \show_strict:nn {#1} {#2}
      }
  }

\cs_new_protected:Npn \show_strict:nn #1 #2
  {
   \iow_term:n {show_strict ~ P1 ~ = ~ #1}
   \iow_term:n {show_strict ~ P2 ~ = ~ #2}
    \str_case:nnF {#1}
      {
       {*}  { \underset {\textup{(semi-strict,~strict)}} {#2} }
       {**} { \underset {\textup{semi-strict~(strict)}} {#2} }
      }
      {
        \underset {\textup{#1}} {#2}
      }
  }

\ExplSyntaxOff

\begin{document}

\begin{enumerate}
\item Test $\strict{{default}}$ \\*
      expect underset with upright "strict" 
\item Test $\strict[semi-]{{hyphen}}$
      expect underset with upright "semi-strict"
\item Test $\strict[*]{{star}}$
      expect underset with upright "(semi-strict,strict)"
\item Test $\strict[**]{{starstar}}$
      expect underset with upright "semi-strict (strict)"
\end{enumerate}

\end{document}

Rather than \mathrm you should use \textup (the hyphens would be minus signs otherwise and spaces would be ignored).

Here's the console output:

*************************************************
* show_strict P1 = strict
*************************************************
*************************************************
* show_strict P2 = {default}
*************************************************
*************************************************
* show_strict P1 = {semi-}strict
*************************************************
*************************************************
* show_strict P2 = {hyphen}
*************************************************
*************************************************
* show_strict P1 = *
*************************************************
*************************************************
* show_strict P2 = {star}
*************************************************
*************************************************
* show_strict P1 = **
*************************************************
*************************************************
* show_strict P2 = {starstar}
*************************************************

When you do

\str_set:Nn \l_tmpa_str {#1}
\str_set:Nn \l_tmpb_str {#2}

you are setting the variables to TeX strings, removing the meaning from the tokens. In particular, { and } cease to be 'special', and come out as the 'random' quote mark/hyphen you are seeing. In your example, there's no reason to store these at all, but if you do, use a tl.

The second issue is

\str_case:nnF {\l_tmpa_str}

which compares exactly the text \l_tmpa_str with each case. They never match, so you get the F branch. You want to look at the text itself, easiest with

\str_case:nnF {#2}

or if you really want to store it, perhaps

\cs_generate_variant:Nn \str_case:nnF { V }

...

\str_case:VnF \l_tmpb_str

to access the value of the variable. Again, I suspect you want a token list test here, not a string one:

\tl_set:Nn \l_tmpb_tl {#2}
\tl_case:NnF \l_tmpb_tl

although as previously-mentioned, I don't see the need to store the input.