Estimate for the order of the outer automorphism group of a finite simple group
Check article "Probabilistic generation of wreath products of non-abelian finite simple groups" by Martyn Quick. In Section 3.1 he consider this question and get $|Out G|\leq |G|/30$ for every non-abelian finite simple group $G$, which was enough for his needs.
You can derive bounds in a straightforward way from the complete list of finite simple groups in Wikipedia, which has references (particularly to ATLAS). If you look through all of the cases (Note: I had not thoroughly checked all of the cases in an earlier version of this answer, and got a wrong bound), you find two worst cases asymptotically:
- $A_2(2^{2k})$ has order $8^{2k}(4^{2k}-1)(8^{2k}-1)/3$ and outer autmorphism group of order $12k$, for $k \geq 1$.
- ${}^2A_2(2^{2k+1})$ (written ${}^2A_2(4^{2k+1})$ in Wikipedia) has order $8^{2k+1}(4^{2k+1}-1)(8^{2k+1}+1)/3$ and outer automorphism group of order $6(2k+1)$, for $k \geq 1$.
In both cases, the asymptotics imply the outer automorphism group has order not much larger than $6\frac{\log(3|G|)}{\log(2^8)}$. Optimal bounds are given by taking $k=1$, and the first case gives a slightly larger function, namely $6 \frac{\log (2^{10}|G|/315)}{\log 2^8}$. This is an upper bound for all non-abelian simple groups, and is sharp for $A_2(4)$.
Regarding your last question, this bound is far better than what you get for finite groups in general. For example, the elementary abelian group $(\mathbb{Z}/2\mathbb{Z})^n$ has order $2^n$, but its outer automorphism group is $GL_n(\mathbb{Z}/2\mathbb{Z})$, which has order $\prod_{k=0}^{n-1} (2^n-2^k)$. In other words, a group of order $m$ can have outer automorphism group with size about $m^{\log m}$ instead of $\log m$.
I recommending listing all the possibilities. A number of sources already do this, so it is fairly easy to do again; the table in the ATLAS is quite reasonable. Basically, the outer automorphism group is ridiculously small "most" of the time, so you might care about the details.
I think you'll get |Out(G)| ≤ C*log(|G|) as worst case, but this is pretty pessimistic most of the time.
The outer automorphism group of an alternating group has order at most 4, and almost always has order 2. There are finitely many sporadic groups, and so will not matter asymptotically, but you can quickly check over the list to see they have outs of size at most 2.
The groups of Lie type have a 3-part outer-automorphism group; the diagonal, the field, and the diagram parts. The diagram part has order at most 6 (and only for D4). The field part is cyclic, but can be "large", as in, if "q" of your group is p^f, then it is cyclic of order f. The diagonal part is usually small (order at most 4, or even order at most 2), but can be larger for PSL(n,q) and PSU(n,q). Even there it is cyclic of order at most n.
So basically you handle the case of PSL/PSU a little more carefully, then the case of a general group of Lie type using bounds of 4 and 6 for diagonal and diagram so getting something like O(log(|G|)), then you handle the rest which are bounded by a constant.