Adjust the padding provided by Colorbox macro from the realboxes package

It uses the same \fboxsep parameter as the standard \colorbox

\documentclass{article}

\usepackage{xcolor}
\usepackage{listings}
\usepackage{realboxes}

\begin{document}
Some inline code \Colorbox{gray!20}{\lstinline{@#$%^&*()}} here

\setlength\fboxsep{0pt}
Some inline code \Colorbox{gray!20}{\lstinline{@#$%^&*()}} here
\end{document}

You can locally set a different value for \fboxsep; in order to have all those backgrounds the same height, a phantom seems the best solution. You can also reduce the padding on the sides by using kerns.

\documentclass{article}
\usepackage{xpatch}
\usepackage{xcolor}
\usepackage{listings}
\usepackage{realboxes}

\definecolor{mygray}{rgb}{0.8,0.8,0.8}

\lstset{
  basicstyle=\ttfamily,
  backgroundcolor=\color{mygray},
}

\makeatletter
\xpretocmd\lstinline
  {%
   \bgroup\fboxsep=1.5pt
   \Colorbox{mygray}\bgroup\kern-\fboxsep\vphantom{\ttfamily\char`\\y}%
   \appto\lst@DeInit{\kern-\fboxsep\egroup\egroup}%
  }{}{}
\makeatother

\begin{document}

\noindent\lstinline[language=TeX]|\my code| \lstinline|axe| \lstinline|lll|

\begin{lstlisting}[language=TeX,columns=fullflexible]
\my code
\end{lstlisting}

\end{document}