Why are my network connections being rejected?

Well, I figured it out. And it's a doozy.

CentOS 8 uses nftables, which by itself isn't surprising. It ships with the nft version of the iptables commands, which means when you use the iptables command it actually maintains a set of compatibility tables in nftables.

However...

Firewalld -- which is installed by default -- has native support for nftables, so it doesn't make use of the iptables compatibility layer.

So while iptables -S INPUT shows you:

# iptables -S INPUT
-P INPUT ACCEPT

What you actually have is:

        chain filter_INPUT {
                type filter hook input priority 10; policy accept;
                ct state established,related accept
                iifname "lo" accept
                jump filter_INPUT_ZONES_SOURCE
                jump filter_INPUT_ZONES
                ct state invalid drop
                reject with icmpx type admin-prohibited  <-- HEY LOOK AT THAT!
        }

The solution here (and honestly probably good advice in general) is:

systemctl disable --now firewalld

With firewalld out of the way, the iptables rules visible with iptables -S will behave as expected.