How to typeset the subscript of a matrix?

An index can be a scalar, or a vector in its own right: compare $\vec{x}_r$ with $\vec{x}_{\vec{r}}$ - these are different mathematical entities! The dimensions of the index may not coincide with the dimensions of the indexed object.

Therefore as a general rule your variant 1 should be preferred: it shows that the quantity is a vector or a matrix with a scalar index (or vector index if appropriate).

Of course some journals and publishers introduce their own "house styles" which may have different recommendations. Sadly enough, sometimes these "house styles" are written without input from the mathematicians, so I can envision a style that requires (mathematically unsound) variant 2.

In general, if the \vec and \mat commands are designed to typeset their arguments (vectors and matrices, respectively) in bold characters, it would be rather unusual to include any associated subscripts and superscripts inside the arguments of these commands -- unless the resulting entities are themselves matrices and/or vectors. For instance, if A is a four-dimensional matrix, then \mat{A_{ij}} (with bold-faced i and j) should be used to denote the two-dimensional i,j-th submatrix of A. Put differently, the subscripts and superscripts should always be typeset in the ordinary font unless there's a really good reason to do otherwise.

On your second question: I can think of no reasonable justification for bold-facing the transpose symbol as a function of whether or not the item being transposed is typeset in boldface or not. Thus: don't boldface the transpose symbol unless you are required to do so by some house style.

A subject contained, at least implicitly, in your question: How should one typeset the transpose symbol? Personally, I have no opinion on which single symbol is best -- that's a matter akin to asking whether vanilla, strawberry, or chocolate ice cream is best. I do have the following recommendation, though: if you have lots of matrix transpose symbols in your document and if there's a chance, at some point in the future, that you'll need to switch from one symbol to another, it should really be worth your time to define a command that denotes the transpose explicitly, and to use this command instead of hardcoding the transpose symbol. Here are some possibilities:

\newcommand{\tran}{^T}
\newcommand{\tran}{'}
\newcommand{\tran}{^{\mathstrut\scriptscriptstyle{\top}}} % ooh, fancy!

This way, if you ever have to switch from one symbolic notation convention to another --say, because some journal or book style says so -- then you can comply without having to track down and examine every instance of ^T and/or ' to decide whether the symbol stands for a transpose, a terminal point of time (as in \sum_{t=1}^T), a first derivative, or what not. The MWE below illustrates the "look and feel" of various symbol choices for the transpose operator, as well as the use of ordinary vs boldface fonts for subscripts.

Finally, an issue may arise if you have to typeset subscripts of a matrix and a transpose symbol: If the transpose symbol is typeset to the immediate right of the matrix, and therefore above the subscripts to the matrix, your readers may be unsure (at least at first) if the ij subscript pertains to A or to its transpose. Hence, if there's a chance that such confusion may arise, you may help out your readers by not placing the transpose symbol and the matrix's subscripts above one another, as is illustrated in the third example line of the MWE. Remember also that there's no shame in using parentheses occasionally to aid the legibility of a formula... :-)

\documentclass{article}
\usepackage{bm,amsmath}
\renewcommand{\vec}[1]{\bm{#1}}
\newcommand{\mat}[1]{\bm{#1}}
\newcommand{\tran}{^{\mathstrut\scriptscriptstyle\top}} 
\setlength\parindent{0pt}
\begin{document}
Subscripts non-bold (normal case):\\
$\vec{b}_i$; $\vec{b}\tran$, $\vec{b}'$, $\vec{b}^T$; 
$\mat{A}_{ij}\tran$, $\mat{A}_{ij}'$, $\mat{A}_{ij}^T$ 

\bigskip
Subscripts bold (needs a good reason):\\    
$\vec{b_i}$, $\mat{A_{ij}}$

\bigskip
Transpose symbol in various possible positions\\
relative to the matrix and its subscripts:\\
$\mat{A}\tran_{ij}$\,: ${\mat{A}\tran}_{ij}$ or ${\mat{A}_{ij}}\tran$?
\end{document}

If you look at it purely semantically, \vec{x_i} could be something like the i-th vector from a collection of vectors (for example a generating set), while \vec{x}_i would more likely mean the i-th element of the vector x, regardless how they are actually typeset. As others have noted, though, it may look weird to have boldface subscripts.

To add to this, there seems to be at least the following four cases:

1. \vec{x}_i which could mean the i-th element of vector x
2. \vec{x}_\vec{i} which could mean a subvector (a slice) of a vector x
3. \vec{x_i} which one could interpret as an i-th element from a collection of vectors
4. \vec{x_\vec{i}} which could be an element of a collection of vectors indexed by vectors.

When using bold face to typeset vectors, there would be no way to distinguish between 2, 3 and 4, and 3 would be somewhat misleading, as it would seem that the index is a vector, which it isn't.

If you used an extensible arrow to indicate vectors, each of those would be typeset differently, in somewhat logical way, but something like \vec{x_{i+2j-1}} would look rather ugly, I think.