Is there any reason to have 2 NICs on a server?
Solution 1:
(2a). Load balancing.
(3). Separation of traffic (i.e. you could have a combo web/database server, same network, put all web traffic on one NIC, db traffic on the other, makes it easier to calculate loads for traffic types). This also makes it easier to split the two later on, nobody has to change connection strings.
Solution 2:
A separate network for backups.
A separate network for console access (these are typically now integrated on the system board for mainstream servers).
Increased throughput (multiple GB networks) bonded together.
Solution 3:
There's only one other major reason that I can think of, which is similar to redundancy, and that's Trunking, so you could configure traffic to flow over 2x1Gbit ports, giving you 2Gbit of total throughput. Some refer to this as LaCP, linux knows this as mode=4 as described below.
Linux interface bonding modes give a few other options:
mode=0 (balance-rr) Round-robin policy
mode=1 (active-backup) Active-backup policy
mode=2 (balance-xor) XOR policy: Transmit based on [(source MAC address XOR'd with destination MAC address) modulo slave count].
mode=3 (broadcast) Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.
mode=4 (802.3ad) IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings.
mode=5 (balance-tlb) Adaptive transmit load balancing: channel bonding that does not require any special switch support.
mode=6 (balance-alb) Adaptive load balancing: includes balance-tlb plus receive load balancing (rlb) for IPV4 traffic, and does not require any special switch support.
Solution 4:
Microsoft clusters require an additional NIC port for heartbeat purposes between nodes.