Increasing the size of a symbol "|"

You can use \middle|, and you have to add some horizontal spacing on the right and on the left.

Even if you don't want to use external package modules, I suggest using the DeclarePairedDelimiter from mathtools (which is just an extension of amsmath, which takes care of all the spacing problems, and loading xparse to simplify the syntax (both arguments are separated by a ;). Here is a code for each solution:

\documentclass{article}
\usepackage{xparse, mathtools, amsfonts}

\DeclarePairedDelimiterX{\set}[1]\{\}{\setargs{#1}}
\NewDocumentCommand{\setargs}{>{\SplitArgument{1}{;}}m}
{\setargsaux#1}
\NewDocumentCommand{\setargsaux}{mm}
{\IfNoValueTF{#2}{#1}{\nonscript\,#1\nonscript\;\delimsize\vert\nonscript\:\allowbreak #2\nonscript\,}}

 \begin{document}

\[ G=\left\{ \begin{pmatrix}t & 0\\
0 & \frac{1}{t}
\end{pmatrix}\,\middle|\, t\in\mathbb{R}^{\times}\right\} \]
\bigskip

\[ G=\set*{ \begin{pmatrix}t & 0\\
0 & \frac{1}{t}
\end{pmatrix} ; t\in\mathbb{R}^{\times}} \]

 \end{document}

First off, I would use inline-fraction notation, 1/t, instead of \frac{1}{t} in order to typeset the matrix more compactly. If you're OK with using psmallmatrix instead of pmatrix, you can then get by with \bigl(, \bigm| (or \bigm\vert), and \bigr). If you prefer using pmatrix, use \biggl(, \biggm|, and \biggr) instead.

Observe that \biggm| automatically adds the typographically required amount of whitespace around the vertical bar. This is in contrast to \bigg|, wwhere no whitespace is inserted.

\documentclass{article}
\usepackage{mathtools} % for 'psmallmatrix' env.
\usepackage{amssymb}   % for '\mathbb' macro
\begin{document}
\begin{align*}
G&=\bigl\{ 
\begin{psmallmatrix} t & 0\\ 0 & 1/t \end{psmallmatrix} 
\bigm| 
t\in\mathbb{R}^{\times} 
\bigr\} \\
&=\biggl\{ 
\begin{pmatrix} t & 0\\ 0 & 1/t \end{pmatrix} 
\biggm| 
t\in\mathbb{R}^{\times} 
\biggr\}
\end{align*}
\end{document}

I don't think this is a Lyx-specific issue. Does this work for you?

\documentclass{article}

\usepackage{amsmath}

\begin{document}

$
G=\left\{
\left. %ADDED (as a blank bracket)
\begin{pmatrix}
t & 0\\
0 & \frac{1}{t}
\end{pmatrix}
\right| %ADDED (as expandable bracket)
t\in\mathbb{R}^{\times}\right\}
$

\end{document}

The way this works is be wrapping another set of brackets around the pmatrix environment. The first one is empty (\left.) while the you replace the current | as the companion right bracket (\right|). This allows it to match the height automatically.