How do I set hostname in docker-compose?

This seems to work correctly. If I put your config into a file:

$ cat > compose.yml <<EOF
dns:
  image: phensley/docker-dns
  hostname: affy
  domainname: affy.com
  volumes:
    - /var/run/docker.sock:/docker.sock
EOF

And then bring things up:

$ docker-compose -f compose.yml up
Creating tmp_dns_1...
Attaching to tmp_dns_1
dns_1 | 2015-04-28T17:47:45.423387 [dockerdns] table.add tmp_dns_1.docker -> 172.17.0.5

And then check the hostname inside the container, everything seems to be fine:

$ docker exec -it stack_dns_1 hostname
affy.affy.com

I found that the hostname was not visible to other containers when using docker run. This turns out to be a known issue (perhaps more a known feature), with part of the discussion being:

We should probably add a warning to the docs about using hostname. I think it is rarely useful.

The correct way of assigning a hostname - in terms of container networking - is to define an alias like so:

services:
  some-service:
    networks:
      some-network:
        aliases:
          - alias1
          - alias2

Unfortunately this still doesn't work with docker run. The workaround is to assign the container a name:

docker-compose run --name alias1 some-service

And alias1 can then be pinged from the other containers.

UPDATE: As @grilix points out, you should use docker-compose run --use-aliases to make the defined aliases available.

As of docker-compose version 3.0 and later, you can just use the hostname key:

version: "3.0"
services:
  yourservicename:
    hostname: your-name