If randomness doesn't exist, how come the universe isn't a perfect sphere with predictable distribution of matter?

This is a superb question, because it gets to the core of one of the great debates of 20th century physics - the nondeterministic interpretation of the laws of quantum mechanics (that is, the universe is truly random), vs the "hidden variables interpretation" (that is, the universe is non-random, but the underlying variables that control it can't be measured).

Almost all serious scientists these days accept nondeterminism, usually in the form of the Copenhagen interpretation; so your presumption that "the scientific community pretty much agrees that randomness doesn't exist" is the complete opposite of true.

I think your presumption is entirely incorrect. Quantum mechanics says is that physical observable quantities of systems are given by probability distributions, so there is intrinsic randomness in any quantum mechanical system. The laws of physics, as we know them now, are fundamentally random in some sense.

Your question still makes sense if we ignore that, though. The degree of randomness should be small for large systems. One can ask about why there are anisotropies (deviations from uniformity) in the distribution of matter and energy in the universe. Most people who study this do so in the context of cosmic microwave background radiation, i.e. photons emitted at the big bang. It's probably not possible to study it in the context of, say, ordinary matter, because gravitational effects will cause clumps of matter to get larger over time, forming very dense regions (stars, galaxies) separated by regions which are mostly empty. So gravity actually magnifies anisotropies over time. Keep in mind, though, that galaxies are actually very small compared to the size of the observable universe, and so anisotropies on the galactic scale shouldn't be too surprising.

People do study CMB anisotropies, and it is a very active area of research. In fact, these anisotropies are actually very small in magnitude. There's still a lot of work to be done here, but the precision measurements that have been done are consistent with what one expects from a quantum mechanical treatment of thermal fluctuations at the big bang (that is to say, quantum mechanical random fluctuations from uniformity at the big bang).

Also, there's no reason to believe that matter was created in all directions equally at the big bang. It's entirely possible to come up with models consistent with general relativity and the big bang which don't have this property. Finally, I question your statement that the galactic-scale structure we see today is chaotic. It exhibits a large number of patterns and has a great deal of structure and uniformity.

On a side note, about randomness, there is a more interesting question in the same vein, which is still open. Why there all matter and essentially no antimatter in the universe, despite the fact that they were created in almost equal quantities at the big bang? The hypothetical answer is CP violation, but all the known sources of CP violation aren't strong enough for the matter density to be what we observe.