UDP traffic not forwarded from Docker containers -> Docker host
It seems you have a modprobe
install directive that cannot work. Possibly it's a result of incomplete update to RHEL 7.2 or some manual fixes.
grep -r bridge /etc/modprobe.d /lib/modprobe.d for starters, or otherwise dig around
/lib/modprobe.d and try to find where does it define the
install rule that calls
sysctl -q -w net.bridge.bridge-nf-call-arptables=0 net.bridge.bridge-nf-call-iptables=0 net.bridge.bridge-nf-call-ip6tables=0
sysctl is clearly in wrong place. It is either superfluous or should appear after
br_netfilter, not before. Why? Recently the
/proc/sys/net/bridge handling has been moved from the
bridge module to the
br_netfilter module. This happens with some version of
kernel*.rpm, while the contents of
modprobe.d directories are distributed with other individual packages. I've verified on my RHEL 7.2:
# modprobe bridge # sysctl -q -w net.bridge.bridge-nf-call-iptables=0 sysctl: cannot stat /proc/sys/net/bridge/bridge-nf-call-iptables: No such file or directory # modprobe br_netfilter # sysctl -q -w net.bridge.bridge-nf-call-iptables=0 # ok now
I don't see these "broken" rules on my vanilla RHEL 7.1
and their origin is mysterious to me. I've tried:
# modprobe -n -vvv bridge modprobe: INFO: custom logging function 0x40a130 registered insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/llc/llc.ko insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/802/stp.ko insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/bridge/bridge.ko modprobe: INFO: context 0xf1c270 released # echo "install bridge echo example_of_a_modprobe_rule" > /etc/modprobe.d/zzz5.conf # modprobe -n -vvv bridge modprobe: INFO: custom logging function 0x40a130 registered insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/llc/llc.ko insmod /lib/modules/3.10.0-229.11.1.el7.x86_64/kernel/net/802/stp.ko install echo example_of_a_modprobe_rule modprobe: INFO: context 0xeaa270 released # rm /etc/modprobe.d/zzz5.conf
Update: Looks like xenserver uses a similar modprobe hack. It's a nasty bug to globally change kernel module behavior for everyone whether you actually run xenserver or not; and the bug has fired back at us.
Update 2: Now, you've found that
/etc/modprobe.d/dist.conf causes this problem and not docker. Whether you have docker or not,
modprobe bridge will always return 1 and print error. Normally dist.conf is a part of
module-init-tools package on RHEL6. This file is not supposed to be used on RHEL7. It's not on any of my RHEL7 systems and they run just fine. In RHEL7 the package is
kmod and it doesn't contain dist.conf. I would:
rpm -qf /etc/modprobe.d/dist.conf # what package owns this file?
If dist.conf is not owned by package, backup it and delete any lines that don't give you any obvious benefit (maybe even delete the file altogether).
If dist.conf is owned by a package, consider removing/updating that package, since it became clearly buggy in terms of RHEL 7.2 compatibility.
I figured it out.
We had a Trend Micro (anti-virus) agent running in the SOE which I didn't know about.
Fixing it was as simple as:
# systemctl stop ds_agent.service # pkill ds_agent
Not exactly sure at this point why it is blocking UDP from containers or how to stop it.