CentOS 6 router/firewall seems to be throttling throughput
So, the problem here turned out to be a hardware issue. Things were working fine a month ago, and one does not expect failed hardware to still "work" in a degraded mode, but that's what was happening.
The troubleshooting step that revealed the issue was to actually look at the ethernet port lights on the back of the cable modem. Instead of the green "1Gbps" light it was orange, signifying "100Mbps". In that mode, it appears the modem supports throughput only up to 30Mbps or so.
I know the modem (Arris TG-852G) has GBEthernet ports, so something was preventing the Centos from talking to the modem at 1Gbps. Using
ethtool I saw this:
Settings for eth1: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Supported pause frame use: No Supports auto-negotiation: Yes Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full Link partner advertised pause frame use: Symmetric Receive-only Link partner advertised auto-negotiation: Yes Speed: 100Mb/s Duplex: Full Port: MII PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000033 (51) drv probe ifdown ifup Link detected: yes
which essentially said (from the Centos adapter's viewpoint) "I can support GBEthernet, and am advertising GBEthernet, but the peer doesn't support GBEthernet -- so I'm connected at 100Mbps instead".
I tried various fixes suggested in several online fora (including here) such as using a different cable, turning off auto-negotiation, advertising only 1GB speed, or setting the speed to 1GB manually. Turning off auto-neg and trying several different Cat6 cables had no effect, and the other two prevented a connection from being established at all.
I concluded it had to be the adapter itself and ordered a new adapter. When it was installed it immediately connected at 1Gbps. Problem solved.
The moral of the story is, of course, that even though hardware failures in devices without moving parts are rare these days, they're still possible and should be eliminated before blaming the software.
What I would do here is revert the changes individually and run a speed test after each or revert all the changes. Benchmark an unmodified version of CentOS (baseline) and then apply each change individually and run the speed test after each change.