iptables --set-mark - Route diferent ports through different interfaces

Note: I only have considered the first script, ignoring the old one.

• You don't need to modprobe netfilter modules by hand with current iptables. This is only necessary for custom connection trackers.
• Don't mix up route and ip route. This is pure evil. Just use ip everywhere and forget about ifconfig and route

• /etc/iproute2/rt_tables is not reset across reboots. Appending the same entry over and over is not a good idea, you only need to do it once. Remember that rt_tables just define name aliases to numeric values, it does not change any configuration.

• Now for iptables: In your FORWARD chain, you drop packets coming from LAN to 4G. This is bad. the FORWARD hook is used after routing is done. At this point, all policy routing is done, and it is already known whether the packet should be sent to 4G or ADSL. There is no rerouting done in FORWARD or after FORWARD (well, technically, rerouting can be done after POSTROUTING in severe cases, but back to the point).

Now for your routing: Remember that Ubuntu enables reverse path filtering by default. Reverse path filtering works as follow: When the kernel receives a packet (may it be forwarded or not) from an interface A, it will invert the source address and the destination address, and check if the resulting packet should be routed through interface A. If it isn't, the packet is dropped as a address spoofing attempt.

For packets received from eth0, this is not a problem. For packets received from eth1, this is also not a problem, because when reversing the source IP address and the destination IP address, the kernel will it the default route in table main. For packets received from eth2, which you do not mark, this is a problem, because the kernel will hit the default route in table main, and consider that these packets should have been received from eth1. The easiest solution is to disable reverse path filtering on eth1:

sysctl -w net.ipv4.conf.eth1.rp_filter=0

BatchyX already give some very good explanation about iptables and routing, so I will exercise my laziness and go directly to script.

It should NAT all traffic to port 80,443,22,4070 through 192.168.0.91. All the rest will NAT through 192.168.1.254.

I re-do my testing and end up following this guide. What is missing in that guide is the last 3 lines in my script. Which I found out from another port, but I lost track of that link.

It is a tested working script.

Need Default Route

One thing I did not put in the script is setting up the default route. It should be

route add default gw 192.168.1.254

When you do route -n, it should be the only default route (Dest:0.0.0.0)

0.0.0.0    192.168.1.254    0.0.0.0    UG    0    0    0    eth1

fw-router.sh

# Reset/Flush iptables
iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT

#Reset/Flush/Setup IP Route (table 4)
ip route flush table 4
ip route show table main | grep -Ev ^default | while read ROUTE ; do ip route add table 4 $ROUTE ; done
ip route add table 4 default via 192.168.0.1

#Mark Packet with matching D.Port
iptables -t mangle -A PREROUTING -p tcp --dport 22   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 80   -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 443  -s 10.0.0.0/24 -j MARK --set-mark 4
iptables -t mangle -A PREROUTING -p tcp --dport 4070 -s 10.0.0.0/24 -j MARK --set-mark 4

#SNAT Rules
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 192.168.1.74
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 192.168.0.91

#IP Route
ip rule add fwmark 4 table 4
ip route flush cache

#IP Stack
#This is the missing part from the guide
echo 1 > /proc/sys/net/ipv4/ip_forward
for f in /proc/sys/net/ipv4/conf/*/rp_filter ; do echo 0 > $f ; done
echo 0 > /proc/sys/net/ipv4/route/flush

PS1: In short, MASQUERADE does not work (in most case, and definitely in your case) for NAT with multiple external IPs that need some kind of load balancing or need DNAT to handle incoming traffic. You need SNAT for direction control.

PS2: Pure iptables is not sufficient.