How do we decide whether to visualize a matrix with its rows or columns?

There's a lot of ways to interpret matrices, some of which involve reading it by rows and some by columns. But in this particular case, it is columns both times: you were misled by the fact that the matrix $$\begin{bmatrix}3 & 1 \\ 1 & 1\end{bmatrix}$$ is symmetric, so its columns are the same as its rows.

Here, the idea is that for any $2 \times 2$ (or more generally $k \times 2$) matrix $A$, we have $$ A \begin{bmatrix}3 & 1 \\ 1 & 1\end{bmatrix} = \begin{bmatrix}A \begin{bmatrix}3 \\ 1\end{bmatrix} & A\begin{bmatrix}1 \\ 1\end{bmatrix} \end{bmatrix}. $$ In other words, each column of the product is equal to $A$ times a column of the second matrix we multiplied.

In the picture you have, the vector $\begin{bmatrix}3 \\ 1\end{bmatrix}$ (in pink) gets sent to $\begin{bmatrix}1 \\ -3\end{bmatrix}$, and the vector $\begin{bmatrix}1 \\1\end{bmatrix}$ (in yellow) gets sent to $\begin{bmatrix}1 \\ -1\end{bmatrix}$, and all of these are columns of the respective $2 \times 2$ matrix.

As long as your main objects of study are column vectors, and you multiply matrix and (column) vector together by writing the matrix on the left and the vector on the right, a matrix is more naturally seen as a collection of columns rather than rows.

A matrix represents a linear transformation. The columns of the matrix are given by where this linear transformation sends your basis vectors. The result of a matrix-vector product similarly becomes a linear combination of the columns of the matrix (where the entries in the vector are the coefficients of this linear combination).

When multiplying two matrices, of course you can choose. Either you say "Apply the left-hand matrix to each column in the right-hand matrix, and collect the results in a new matrix" (in which case you see both matrices as collections of columns), or you say "Apply the right-hand matrix to each row in the left-hand matrix, and collect the results in a new matrix" (in which case both matrices are collections of rows). They both give the same result. Which one is most convenient comes down to whether one happens to be significantly easier to calculate than the other for some reason, and what you're going to do with the result afterwards.

Of course, the final answer is "it depends on the situation". Because what else could it be? But columns is much more common than rows.