How to format K notation for continued fractions

You can download the source of the arXiv paper at https://arxiv.org/format/1909.13597 (choose Download source to download the .tex file).

The paper uses

\def\contFracOpe{%
    \operatornamewithlimits{%
        \mathchoice{% * Display style
            \vcenter{\hbox{\huge $\mathcal{K}$}}%
        }{%           * Text style
            \vcenter{\hbox{\Large $\mathcal{K}$}}%
        }{%           * Script style
            \mathrm{\mathcal{K}}%
        }{%           * Script script style
            \mathrm{\mathcal{K}}%
        }
    }
}

That definition seems to come from projetmbc's answer to How to typeset a continued fraction in the following format? which in term refers to user2478's answer to How to create my own math operator with limits?.

I usually prefer \newcommand over \def for commands in the preamble and the \mathrm's are unnecessary, so I would probably make that definition read

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}

\newcommand\ContFracOp{%
  \operatornamewithlimits{%
    \mathchoice
     {\vcenter{\hbox{\huge $\mathcal{K}$}}}
     {\vcenter{\hbox{\Large $\mathcal{K}$}}}
     {\mathcal{K}}
     {\mathcal{K}}}}


\begin{document}
$\ContFracOp\dots\ContFracOp_{k=1}^m x_k e^{\ContFracOp_{k=1}^m x_k}$

\[\ContFracOp\dots\ContFracOp_{k=1}^m x_k e^{\ContFracOp_{k=1}^m x_k}\]
\end{document}

An alternative would be to base \ContFracOp on egreg's answer that makes use of graphicx's \resizebox to scale the symbol to the same size as \sum.

\documentclass{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{graphicx}

\makeatletter
\DeclareRobustCommand\bigop[2][1]{%
  \mathop{\vphantom{\sum}\mathpalette\bigop@{{#1}{#2}}}\slimits@
}
\newcommand{\bigop@}[2]{\bigop@@#1#2}
\newcommand{\bigop@@}[3]{%
  \vcenter{%
    \sbox\z@{$#1\sum$}%
    \hbox{\resizebox{\ifx#1\displaystyle#2\fi\dimexpr\ht\z@+\dp\z@}{!}{$\m@th#3$}}%
  }%
}
\makeatother

\newcommand{\ContFracOp}{\DOTSB\bigop[.96]{\mathcal{K}}}

\begin{document}
$\ContFracOp\dots\ContFracOp_{k=1}^m x_k e^{\ContFracOp_{k=1}^m x_k}$

\[\ContFracOp\dots\ContFracOp_{k=1}^m x_k e^{\ContFracOp_{k=1}^m x_k}\]
\end{document}

I found the magic number .96 in the optional argument to \bigop in the definition of \ContFracOp by trial and error. It helps manually correct the scaling of the symbol to match the height of \sum as closely as possible.

A simplification of moewe's answer. As a bonus, also the alternative notation for continued fractions.

\documentclass{article}
\usepackage{amsmath,graphicx}

\makeatletter
\DeclareRobustCommand{\gaussk}{\DOTSB\gaussk@\slimits@}
\newcommand{\gaussk@}{\mathop{\vphantom{\sum}\mathpalette\bigcal@{K}}}

\newcommand{\bigcal@}[2]{%
  \vcenter{\m@th
    \sbox\z@{$#1\sum$}%
    \dimen@=\dimexpr\ht\z@+\dp\z@
    \hbox{\resizebox{!}{0.8\dimen@}{$\mathcal{K}$}}%
  }%
}
\newcommand{\cfracplus}{\mathbin{\cfracplus@}}
\newcommand{\cfracplus@}{%
  \sbox\z@{$\dfrac{1}{1}$}%
  \sbox\tw@{$+$}%
  \raisebox{\dimexpr\dp\tw@-\dp\z@\relax}{$+$}%
}
\newcommand{\cfracdots}{\mathord{\cfracdots@}}
\newcommand{\cfracdots@}{%
  \sbox\z@{$\dfrac{1}{1}$}%
  \sbox\tw@{$+$}%
  \raisebox{\dimexpr\dp\tw@-\dp\z@\relax}{$\cdots$}%
}
\makeatother

\begin{document}

\[
e^n=\sum_{k=0}^{n-1}\frac{n^k}{k!}+\frac{n^{n-1}}{(n-1)!}\biggl(
1+n+\gaussk_{m=1}^{\infty}\Bigl(\frac{-n(m+n-1)}{m+2n-1}\Bigr)\biggr)
\]

\[
\gaussk_{m=1}^{\infty}\Bigl(\frac{a_m}{b_m}\Bigr)=
\frac{a_1}{b_1}\cfracplus
\frac{a_2}{b_2}\cfracplus
\frac{a_3}{b_3}\cfracplus\cfracdots
\]

\[
\sum_{n=1}^{\infty}\gaussk_{n=1}^\infty
\quad
\textstyle
\sum_{n=1}^{\infty}\gaussk_{n=1}^\infty
\quad
\scriptstyle
\sum_{n=1}^{\infty}\gaussk_{n=1}^\infty
\]

\end{document}

An elementary solution, not as general, based on \genfrac and package scalerel:

\documentclass{article}
\usepackage{scalerel}
\usepackage{mathtools, amssymb}
\newcommand{\cadd}[1][0pt]{\mathbin{\genfrac{}{}{#1}{0}{}{+}}}
\newcommand{\Cdots}[1][0pt]{\genfrac{}{}{#1}{0}{\mbox{}}{\cdots}}
\DeclareMathOperator*{\Kont}{\mathcal{K}}
\DeclareMathOperator*{\bigKont}{\scalerel*{ \mathcal{K}}{\big(}}

\begin{document}
    \[\bigKont_{m = 1}^{\infty}\Bigl(\frac{a_m}{b_m}\Bigr)\coloneqq \frac{a_1}{b_1}\cadd \frac{a_2}{b_2}\cadd \frac{a_3}{b_3}\cadd\Cdots \]%

\end{document}