How does DHT in torrents work?

The general theory can be found in wikipedia's article on Kademlia. The specific protocol specification used in bittorrent is here: http://wiki.theory.org/BitTorrentDraftDHTProtocol

What happens with bittorrent and a DHT is that at the beginning bittorrent uses information embedded in the torrent file to go to either a tracker or one of a set of nodes from the DHT. Then once it finds one node, it can continue to find others and persist using the DHT without needing a centralized tracker to maintain it.

The original information bootstraps the later use of the DHT.

With trackerless/DHT torrents, peer IP addresses are stored in the DHT using the BitTorrent infohash as the key. Since all a tracker does, basically, is respond to put/get requests, this functionality corresponds exactly to the interface that a DHT (distributed hash table) provides: it allows you to look up and store IP addresses in the DHT by infohash.

So a "get" request would look up a BT infohash and return a set of IP addresses. A "put" stores an IP address for a given infohash. This corresponds to the "announce" request you would otherwise make to the tracker to receive a dictionary of peer IP addresses.

In a DHT, peers are randomly assigned to store values belonging to a small fraction of the key space; the hashing ensures that keys are distributed randomly across participating peers. The DHT protocol (Kademlia for BitTorrent) ensures that put/get requests are routed efficiently to the peers responsible for maintaining a given key's IP address lists.