Linux: Bind UDP listening socket to specific interface (or find out the interface a datagram came in from)?

The solution that I found to work is as follows. First of all, we have to change ARP and RP settings. To /etc/sysctl.conf, add the following and reboot (there's also a command to set this dynamically):

net.ipv4.conf.default.arp_filter = 1
net.ipv4.conf.default.rp_filter = 2
net.ipv4.conf.all.arp_filter = 1
net.ipv4.conf.all.rp_filter = 2

The arp filter was necessary to allow responses from eth0 to route over a WAN. The rp filter option was necessary to strictly associate in-coming packets with the NIC they came in on (as opposed to the weak model that associates them with any NIC that matches the subnet). A comment from EJP led me to this critical step.

After that, SO_BINDTODEVICE started working. Each of two sockets was bound to its own NIC, and I could therefore tell which NIC a message came from based on the socket it came from.

s=socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, IF_NAMESIZE);
memset((char *) &si_me, 0, sizeof(si_me));
si_me.sin_family = AF_INET;
si_me.sin_port = htons(LISTEN_PORT);
si_me.sin_addr.s_addr = htonl(INADDR_ANY);
rc=bind(s, (struct sockaddr *)&si_me, sizeof(si_me))

Next, I wanted to respond to in-coming datagrams with datagrams whose source address is that of the NIC the original request came from. The answer there is to just look up that NIC's address and bind the out-going socket to that address (using bind).

s=socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP)
get_nic_addr(nics, (struct sockaddr *)&sa)
sa.sin_port = 0;
rc = bind(s, (struct sockaddr *)&sa, sizeof(struct sockaddr));
sendto(s, ...);

int get_nic_addr(const char *nic, struct sockaddr *sa)
{
    struct ifreq ifr;
    int fd, r;
    fd = socket(AF_INET, SOCK_DGRAM, 0);
    if (fd < 0) return -1;
    ifr.ifr_addr.sa_family = AF_INET;
    strncpy(ifr.ifr_name, nic, IFNAMSIZ);
    r = ioctl(fd, SIOCGIFADDR, &ifr);
    if (r < 0) { ... }
    close(fd);
    *sa = *(struct sockaddr *)&ifr.ifr_addr;
    return 0;
}

(Maybe looking up the NIC's address every time seems like a waste, but it's way more code to get informed when an address changes, and these transactions occur only once every few seconds on a system that doesn't run on battery.)

You can get the destination address used by the sender via the IP_RECVDSTADDR option if your platform supports it, by using recvmsg(). It's rather complicated, described in Unix Network Programming, volume I, 3rd edition, #22.2, and in the man page.

Re your edit, you are up against what is known as the 'weak end system model' of TCP/IP. Basically once a packet arrives the system can choose to deliver it via any appropriate interface listening to the correct port. It's discussed in the TCP/IP RFCs somewhere.

You're passing an illegal value to setsockopt.

rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, strlen(nic));

The man page says of SO_BIND_TO_DEVICE:

The passed option is a variable-length null-terminated interface name string with the maximum size of IFNAMSIZ

strlen doesn't include the terminating null. You can try:

rc=setsockopt(s, SOL_SOCKET, SO_BINDTODEVICE, nic, 1 + strlen(nic));

dnsmasq has this working correctly, and uses

setsockopt(fd, SOL_SOCKET, SO_BINDTODEVICE, intname, IF_NAMESIZE)