Resolving hostname takes 5 seconds

Short answer:

A workaround is forcing glibc to reuse a socket for look up of the AAAAand A records, by adding a line to /etc/resolv.conf:

options single-request-reopen

The real cause of this issue might be:

  • malconfigured firewall or a router (e.g. a Juniper firewall configuration described here) which causes dropping AAAA DNS packets
  • bug in DNS server

Long answer:

Programs like curl or wget use glibc's function getaddrinfo(), which tries to be compatible with both IPv4 and IPv6 by looking up both DNS records in parallel. It doesn't return result until both records are received (there are several issues related to such behaviour) - this explains the strace above. When IPv4 is forced, like curl -4 internally gethostbyname() which queries for A record only.

From tcpdump we can see that:

  • -> A? two requests are send at the beginning
  • -> AAAA? (requesting IPv6 address)
  • <- AAAA reply
  • -> A? requesting again IPv4 address
  • <- A got reply
  • -> AAAA? requesting IPv6 again
  • <- AAAA reply

One A reply gets dropped for some reason, that's this error message:

error sending response: host unreachable

Yet it's unclear to me why there's a need for second AAAA query.

To verify that you're having the same issue you can update timeout in /etc/resolv.conf:

options timeout:3

as described here:

$ curl -w "@curl-format.txt" -o /dev/null -s

            time_namelookup:  3.511
               time_connect:  3.511
            time_appconnect:  3.528
           time_pretransfer:  3.528
              time_redirect:  0.000
         time_starttransfer:  3.531
                 time_total:  3.531

There are two other related options in man resolv.conf:

single-request (since glibc 2.10) sets RES_SNGLKUP in _res.options. By default, glibc performs IPv4 and IPv6 lookups in parallel since version 2.9. Some appliance DNS servers cannot handle these queries properly and make the requests time out. This option disables the behavior and makes glibc perform the IPv6 and IPv4 requests sequentially (at the cost of some slowdown of the resolving process).

single-request-reopen (since glibc 2.9) The resolver uses the same socket for the A and AAAA requests. Some hardware mistakenly sends back only one reply. When that happens the client system will sit and wait for the second reply. Turning this option on changes this behavior so that if two requests from the same port are not handled correctly it will close the socket and open a new one before sending the second request.

Related issues:

As @Tombart says, the delay is due to waiting for the IPv6 resolution timeout.

Another possible course of action is giving precedence to IPv4 in /etc/gai.conf

From comments in /etc/gai.conf

#   For sites which prefer IPv4 connections change the last line to
precedence ::ffff:0:0/96  100

After changing gai.conf, you need to restart any app using the DNS resolver library for the change to take effect.

Mind you that if you are using a BIND server without IPv6 connectivity, I advise disabling IPv6 in named and taking from the root hints IPv6 addresses. Obviously it will still try to resolve AAAA addresses.

So for the BIND configuration,

In /etc/default/bind9, add -4 for IPv4 addresses:

OPTIONS="-4 -u bind"

and in /etc/bind/db.root, delete all lines with AAAA DNS roots.

I had a similar issue while using BIND9. To fix this I needed to add:

filter-aaaa-on-v4 yes;

option to my named.conf.

(More information)