Real and quaternionic representations according to weights

There is some misleading information in the literature about this issue, due (as mentioned by Henrik Winther) to special properties of compact groups.

Suppose $\pi$ is an irreducible representation of a real group.

If $\pi$ is self-dual then it supports an invariant bilinear form, which is symmetric or skew-symmetric. Which one is given by the Frobenius Schur indicator $\epsilon(\pi)=\pm1$.

If $\pi$ is self-conjugate, it is real or quaternionic. Write the real/quaternionic indicator $\delta(\pi)=\pm1$ respectively.

If $\pi$ is Hermitian (admits an invariant Hermitian form, not necessarily positive definite), then self-dual and self-conjugate are equivalent (a stronger statement than the first comment above). If $\pi$ is furthermore unitary then $\delta(\pi)=\epsilon(\pi)$.

There is an elementary formula for $\epsilon$ when $\pi$ is finite dimensional. See Bourbaki, Lie Groups and Lie Algebras, Chapter 7-9, or see The Real Chevalley Involution, arXiv:1203.1901 for a simpler proof.

Proposition: $\epsilon(\pi)=\chi_\pi(\exp(2\pi i\rho^\vee))$

where $\chi_\pi$ is the central character and $\rho^\vee$ is one-half the sum of the positive coroots. Obviously this is independent of the real form. In general, however, $\delta(\pi)$ is sensitive to the real form.

Corollary: If $\pi$ is unitary, in particular if $G$ is compact, the real-quaternionic indicator is $\delta(\pi)=\chi_\pi(\exp(2\pi i\rho^\vee))$.

This gives the simplest answer to the original question, assuming it was only about compact groups. For non-compact groups the relationship is more complicated, although there is still a closed formula; this is the subject of a forthcoming dissertation by Ran Cui at the University of Maryland.

It is unlikely that there is a procedure of the kind you would like, because the weight structure of representations depends only on the complexified Lie algebra, while distinguishing between real and quaternionic type depends on which real form is being considered. This is something that is obscured, I think, by restriction to only compact forms, because then you have a bijection between real and complex algebras. For example, the first fundamental (complex) representation of $\mathfrak{sl}_2(\mathbb{C})$ corresponds to a quaternionic type (real) irreducible of the real form $\mathfrak{su}(2)$, but real type irreducible for the other form $\mathfrak{sl}_2(\mathbb{R})$. Both of these have the same weights.

Another piece of evidence is that there is no mention of such a procedure to determine the type in Onishchik - Lectures on Real Semisimple Lie Algebras and Their Representations - EMS (2003). This book is the most comprehensive guide to computing types of fundamental representations from scratch that I know of, but the way described there is rather involved.

If you are willing to accept the types of the fundamental representations as given (e.g. from a table), then there is however a rather simple formula dating to Cartan for determining the type of other arbitrary irreducibles in terms of highest weight. The details for this is also given in Onishchik's book.

$\mathrm{Hom}_G(V^*,V) \cong \mathrm{Hom}(V^*,V)^G \cong (V\otimes V)^G \cong (\mathrm{Sym}^2 V\oplus \mathrm{Alt}^2 V)^G$, so the first is nonzero ($V$ is self-dual) iff $V$ possesses a symmetric or alternating invariant form. By Schur's lemma it can have only one of the two.

It's not as easy to do in your head, but you can indeed compute the weight multiplicities of $Alt^2 V$ from those of $V$: $m_\nu(\mathrm{Alt}^2 V) = {m_{\nu/2}\choose 2} + \sum_{\{\lambda,\mu\}, \lambda\neq \mu} m_\lambda(V) m_\mu(V)$.

Now you need to determine whether those multiplicities give a representation with an invariant vector. This is the most annoying part. For each positive root $\beta$, we have a differencing operator "at $\mu$, subtract the value located at $\mu+\beta$", and these commute. Apply all of them and see if the value at the origin is $1$ or $0$. (This is undoing the denominator in the Weyl character formula, just Fourier transformed to deal with weight multiplicities.) If $1$, then $V$ is quaternionic (for $G$ compact); if $0$, then $V$ is real (for $G$ compact, and $V$ assumed self-dual).

The latter half of this answer may not be so different from answering your boldface question with "Yes, in principle you must be able to, since the representation is characterized by its multiplicity diagram."