Parenthesis vs brackets for matrices

It is just a question of notation and what you are comfortable with. The only thing you might want to be aware is |A| stands for determinant of the matrix A and hence cannot be used in place of () or [] to represent the matrix

They're interchangeable. I think a lot of people tend to use parentheses just because they're easier to write by hand.

There are two notations for matrices, ( ), [ ]. Use the one with which you are comfortable with.
Below are examples of both for the 3x3 Identity matrix. $$ \begin{pmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{pmatrix} \\ \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} $$