Define a macro for a vector to display differently in inline/display math mode

Distinguishing inline and display math is different from distinguishing \displaystyle from \textstyle and can be tricky, since the align-like environments of amsmath use internally $\displaystyle...$. Your definition with \ensuremath will furthermore always use \textstyle if you call the macro outside of math mode. In my (of course questionable) opinion \ensuremath shouldn't be used unless the life of sweet kittens and fluffy bunnies depends on it; and also in that case it shouldn't be used lightly.

Luckily, amsmath provides a boolean \if@display, so we can (ab)use it. I'd load mathtools, an extension to amsmath, which among other things defines already a psmallmatrix environment.

\documentclass{article}

\usepackage{mathtools}% loads amsmath

\makeatletter

\newcommand*{\ve}[1]{%
   \begingroup
   \if@display
      \def\@tempa{{pmatrix}}
   \else
      \def\@tempa{{psmallmatrix}}
   \fi
   \expandafter\begin\@tempa
   \@ve#1,\@@nil,
   \expandafter\end\@tempa
   \endgroup
}
   
\def\@ve#1,{\ifx\@@nil#1\else#1\\\expandafter\@ve\fi}

\makeatother

\begin{document}

\hsize=4cm % for the snapshot

\[
\ve{a}\ve{a,b}\ve{a,b,c}
\]

\begin{align*}
\ve{a}\ve{a,b}\ve{a,b,c}
\end{align*}

$\ve{a}\ve{a,b}\ve{a,b,c}$

\end{document}

EDIT A version which like egreg's answer uses \mathchoice. I've added for completeness also a row vector. This required some more trickery, becase there is no harm in adding \\ at the end of a column vector (like my first coded did) but it goes wrong when you add a & at the end of a row.

\documentclass{article}

\usepackage{mathtools}

\makeatletter

\newcommand*{\cvector}[1]{\cr@vector{\\}{#1}}
\newcommand*{\rvector}[1]{\cr@vector{&}{#1}}

\newcommand*{\cr@vector}[2]{%
   \mathchoice{\cr@@vector{#1}{#2}{}}%
              {\cr@@vector{#1}{#2}{small}}%
              {\cr@@vector{#1}{#2}{small}}%
              {\cr@@vector{#1}{#2}{small}}%
}

\newcommand*{\cr@@vector}[3]{%
   \def\@tempa{\gdef\@tempa{#1}}% first time does nothing, afterwards it's \\ or &
   \def\@tempb##1,{\ifx\@@nil##1\else\@tempa##1\expandafter\@tempb\fi}%
   \begin{p#3matrix}
   \@tempb#2,\@@nil,
   \end{p#3matrix}
}

\makeatother


\begin{document}

\hsize=5cm % for smaller snapshot

Display
\begin{align*}
\rvector{a,b,c} \cvector{a,b,c}
= \frac{\rvector{x,y} \cvector{u,v}}{\rvector{1,2,3}\cvector{4,5,6}}
\end{align*}
and inline $\rvector{a,b,c}$, $\cvector{a,b,c,d,e}$

\end{document}

One could use \mathchoice directly in the definition of \cvector and \rvector and spare one intermediate macro but I find the code would be less readable.

The idea is good. Not so \ensuremath, of course.

\documentclass[twocolumn]{article} % twocolumn is just to get a smaller picture
\usepackage{amsmath}

\ExplSyntaxOn
\NewDocumentCommand{\ve}{m}
 {
  \mathchoice
    {\raymo_ve_display:n {#1}}
    {\raymo_ve_inline:n {#1}}
    {\raymo_ve_inline:n {#1}}
    {\raymo_ve_inline:n {#1}}
 }
\cs_new_protected:Nn \__raymo_ve_make:n
 {
  \seq_set_split:Nnn \l__raymo_ve_body_seq { , } { #1 }
  \seq_use:Nn \l__raymo_ve_body_seq { \\ }
 }
\cs_new_protected:Nn \raymo_ve_display:n
 {
  \begin{pmatrix}
  \__raymo_ve_make:n { #1 }
  \end{pmatrix}
 }
\cs_new_protected:Nn \raymo_ve_inline:n
 {
  \left(\begin{smallmatrix}
  \__raymo_ve_make:n { #1 }
  \end{smallmatrix}\right)
 }
\ExplSyntaxOff

\begin{document}

This is inline $\ve{1,2,x}$ and there is also a display
\[
\ve{1,2,x}
\]

\end{document}

If you're not running the latest LaTeX kernel (2020-10-01), you need \usepackage{xparse}.

I find expl3 much better than xstrings.

“Optimized” code for also supporting row vectors:

\documentclass[twocolumn]{article} % twocolumn is just to get a smaller picture
\usepackage{amsmath,mathtools}

\ExplSyntaxOn
\NewDocumentCommand{\gve}{mm}
 {% #1 = c or r
  \mathchoice
    {\raymo_ve_build:nnn {#1}{#2}{}}
    {\raymo_ve_build:nnn {#1}{#2}{small}}
    {\raymo_ve_build:nnn {#1}{#2}{small}}
    {\raymo_ve_build:nnn {#1}{#2}{small}}
 }
\NewDocumentCommand{\ve}{m}{\gve{c}{#1}}
\NewDocumentCommand{\rve}{m}{\gve{r}{#1}}

\cs_new_protected:Nn \__raymo_ve_make:nn
 {
  \seq_set_split:Nnn \l__raymo_ve_body_seq { , } { #2 }
  \seq_use:Nn \l__raymo_ve_body_seq { \str_case:nn {#1}{{c}{\\}{r}{&}} }
 }
\cs_new_protected:Nn \raymo_ve_build:nnn
 {
  \begin{p#3matrix}
  \__raymo_ve_make:nn { #1 } { #2 }
  \end{p#3matrix}
 }
\ExplSyntaxOff

\begin{document}

This is inline $\ve{1,2,x}\rve{1,2,x}$ and there is also a display
\[
\ve{1,2,x}\rve{1,2,x}
\]

\end{document}