Unstandardized brackets in matrix

I guess you're looking for \left\lceil and \right\rfloor.

\documentclass{article}
\usepackage{amsmath} % for 'matrix' environment
\begin{document}
\[
\left\lceil
\begin{matrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n 
\end{matrix}
\right\rfloor
\]
\end{document}

It makes sense to define new environments for this task:

\documentclass{article}
\usepackage{amsmath}

\makeatletter
\newcommand{\newmatrix}[3]{% #1 = prefix, #2/#3 = left/right delimiters
  \newenvironment{#1matrix}{\left#2\env@matrix}{\endmatrix\right#3}%
}
\makeatother

\newmatrix{cf}{\lceil}{\rfloor}
\newmatrix{fc}{\lfloor}{\rceil}

\begin{document}

\[
\begin{cfmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n
\end{cfmatrix}
\ne
\begin{fcmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n
\end{fcmatrix}
\]

\end{document}

The “prefix” can be any string not already in use.

With a bit more work it can support defining the *-variant in the mathtools style, that is, accepting an optional argument to set the column alignment, and the “small” variant all in one swoop.

\documentclass{article}
\usepackage{amsmath,mathtools}

\MHInternalSyntaxOn
\makeatletter
\newcommand{\newmatrix}[3]{%
  % #1 = prefix, #2/#3 = left/right delimiters
  \newenvironment{#1matrix}
    {\left#2\env@matrix}
    {\endmatrix\right#3}%
  \newenvironment{#1matrix*}[1][c]
    {\left#2\MT_matrix_begin:N ##1}
    {\MT_matrix_end:\right#3}
  \MT_fenced_sm_generator:nnn{#1smallmatrix}{#2}{#3}
}
\makeatother
\MHInternalSyntaxOff

\newmatrix{cf}{\lceil}{\rfloor}

\newmatrix{fc}{\lfloor}{\rceil}

\begin{document}

\begin{gather*}
\begin{cfmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n 
\end{cfmatrix}
\ne
\begin{fcmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n 
\end{fcmatrix}
\\
\begin{cfmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n 
\end{cfmatrix}
\ne
\begin{fcmatrix*}[r]
111 & 44 & \dots & -2 \\
2 & 333 & \dots & 5 
\end{fcmatrix*}
\\
\begin{cfsmallmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n 
\end{cfsmallmatrix}
\ne
\begin{fcsmallmatrix}
a_1 & a_2 & \dots & a_n \\
b_1 & b_2 & \dots & b_n 
\end{fcsmallmatrix}
\end{gather*}

\end{document}